Lightening Tech Evidence, not opinion

UI/UX & Design

An A/B testing programme that produces answers you can act on

Evidence-backed hypotheses, tests that run long enough to mean something, and a written record of what your customers actually do.

From
$800–$5,000/mo
Typical time
ongoing
Usually bought by
High-traffic sites

What this actually solves

Two kinds of company arrive at this page, and only one of them should buy.

The first has real traffic and a conversion rate that has not moved in a year. Every acquisition channel costs more than it did, so the cheapest growth left is getting more out of the visitors already arriving. At that scale a fractional improvement is not a design opinion — it is a permanent monthly gain that keeps paying, because next month's traffic arrives whether or not anyone tests anything.

The second arrives with an argument. Someone senior wants the hero section rewritten, someone in product wants the form shortened, and the same meeting has now happened three times. A test settles that in a way no amount of seniority can, and it settles it in public. That alone is why programmes survive a change of management — the decisions stop being attached to whoever argued hardest.

What both are really buying is a monthly supply of facts about their own customers that no competitor has, including the uncomfortable ones. No measurable difference is the single most common outcome of a well-run test, and it is an answer rather than a wasted month. It stops a change that was about to ship on confidence alone, and it takes that idea off the backlog permanently. A programme that never reports a flat result is not testing carefully; it is reporting selectively.

This is ongoing work, and the rate is monthly because that is where the value lives. One test tells you about one change. A year of tests tells you how your customers behave, and that compounds into a body of knowledge that makes every subsequent design decision cheaper.

How a month of the programme runs

The order is not negotiable, and the first step is the one almost every programme skips.

  1. Verify the measurement before running a single test. Conversion events that fire on page load rather than on purchase, events firing twice, an experiment tool and an analytics tool that quietly disagree about who converted. If the number being optimised is wrong, every result that follows is confidently wrong, and the programme spends months producing precise nonsense. This is done once, properly, at the start. First two weeks of the engagement.
  2. Build the hypothesis backlog from evidence, not from opinion. Sources are session recordings, the funnel steps where people actually leave, recurring support complaints, an existing conversion audit, and talking to customers. Every entry is written in one fixed form — because of this specific evidence, we believe this specific change will increase this specific outcome, and we will measure it with this metric. Anything that cannot be written in that shape is not a hypothesis yet, it is a preference, and it goes back until someone finds the evidence. Three to four days at the start, topped up every month.
  3. Prioritise, then build the month's tests. Ordered by plausible effect size against build effort, and filtered by one hard question — can your traffic actually resolve a difference this small? The required sample size is calculated up front at 95% significance, and tests that would need half a year to reach it are removed from the plan rather than run hopefully. Two to four days per test.
  4. Run to a stopping rule agreed before launch. The end condition — sample size reached, or a fixed number of days — is written down before the test goes live, and the result is not read before then. The floor is two full weeks so that every day of the week appears in both variants, because the people who buy on a Sunday are not the people who buy on a Tuesday. Minimum two full weeks per test, longer where the calculated sample demands it.
  5. Read it, ship or revert, and write down what was learned. Winners get implemented properly rather than left running as a flag forever. Losers and flat results get logged with the evidence, so the same idea is not proposed again in six months by someone new. Half a day per test, plus one monthly session walking through results and the next month's plan.

Tools, so you know what you are buying rather than a black box — PostHog for most programmes, where the free tier covers both feature flags and experiments, VWO or Optimizely where an existing stack requires them, and sample-size and significance calculators for the arithmetic that decides when a test is allowed to stop.

What moves the price

The range is $800–$5,000/mo and it is set by three things.

Who builds the variants. If we build them — copy, layout, ordering, visibility changes, anything achievable in the testing tool — the cost is predictable. If your developers build them because the tests touch pricing logic, checkout steps or server-rendered pages, the programme becomes a coordination job as much as a testing one, and the cadence is limited by their sprint rather than by us.

How many surfaces are in scope. Two tests a month on one template of one site is a different engagement from a programme covering desktop, mobile web, an app and more than one market, where every variant has to be built and validated more than once and the traffic gets split further each time.

How much of the first month is repair. Where analytics are clean and events are trustworthy, testing starts almost immediately. Where the measurement layer has to be rebuilt before anything can be believed, the first month is largely that work — and it is worth saying plainly that this is the most common way a programme starts.

What usually goes wrong

A programme is sold to a site that cannot resolve a test. This is the most common and the most damaging failure, and it is usually a selling failure rather than a testing one. On modest traffic, proving a small improvement takes so many months that seasonality, ad spend and pricing all change underneath the experiment, and the result is unreadable. That client does not need tests, they need better design — and telling them so does not cost you the work, because they buy the design instead and they trust everything you say afterwards.

The test gets called early. On day three variant A is ahead, you are delighted, and a win is declared. Two weeks later B is ahead, and by then the winner has already been shipped and the team has moved on. Early-peeking at a running experiment is the fastest way to generate confident, permanent, wrong decisions. The defence is procedural rather than statistical — the end condition is fixed before launch and the dashboard is not opened before it is met.

Nobody checks the measurement layer, so the whole programme is precise about the wrong number. A conversion event bound to a thank-you page that also loads on refresh, or an experiment tool bucketing users on one identifier while analytics attributes revenue on another. Everything downstream looks rigorous — clean charts, real significance figures — and every conclusion is invalid. It is uniquely expensive here because unlike a redesign, a testing programme's only product is the numbers.

See where the site stands before you brief anyone.

If the site is already live, start from what it scores rather than from a guess. The audit reports what it checked, what it found, and — the part most tools leave out — what it could not check, and why.

See the website audit Talk about this work

Questions people ask before they commit.

How much traffic do I need before this is worth buying?
Enough that a realistic improvement can be separated from noise in a few weeks rather than a few quarters. We work that out from your current conversion rate and monthly volume before you commit, and if the answer is that your traffic cannot resolve the tests you want to run, we say so and recommend design work instead. That conversation happens before the first invoice, not after three months of inconclusive results.
How long does a single test run?
Until it reaches the sample size calculated for it, and never less than two full weeks regardless. The two-week floor exists because a weekday-only test compares a Tuesday audience against a Tuesday audience and tells you nothing about the people who buy on a Sunday. Some tests take longer than a month, and when that is true we say it before building the variant.
What happens when a test shows no difference?
It gets written up as a result, because it is one. No measurable difference is the most common outcome of a properly run test, and it is worth money — it stops a change that was about to ship on confidence alone, and it removes that idea from the backlog permanently. A month where two tests both come back flat has still bought you two decisions you would otherwise have argued about forever.
Do you need my developer?
For simple variants — copy, layout, ordering, showing or hiding an element — no. For anything touching pricing logic, checkout steps, or server-rendered pages, yes, and that is the main thing that changes the monthly rate. We scope which side builds what in the first week so nobody is waiting on the other.
Which testing tool do you use, and does it cost extra?
PostHog for most programmes — its free tier covers feature flags and experiments, so tooling can genuinely be a zero-cost line. VWO or Optimizely where an existing stack or a procurement process requires it, and those are licensed by you directly. Worth knowing that Google Optimize was shut down in 2023, so a lot of the A/B advice still ranking well describes a tool that no longer exists.
Can you guarantee a lift in conversion rate?
No. Anyone who does is either planning to call tests early or planning to report the ones that won and quietly drop the rest. What we commit to is a fixed number of tests per month, each with a stopping rule agreed in advance, each read honestly, and a written log of everything learned that stays yours if the engagement ends.

Often taken together.

Start with the brief, not the invoice.

Send what you are trying to achieve. You get a written scope, a price and a date — or an honest “this is not the right service for that”.

Talk about A/B testing programme All UI/UX & Design services