Run and read A/B tests correctly

A/B testing is the gold standard of marketing analytics: randomly show half your visitors version A and half version B, and any difference in conversion can be attributed to the change. It is simple in principle, but most real A/B tests are planned or read incorrectly. This lesson covers the full workflow.

Step 1: plan the sample size before you start

Decide the baseline conversion rate, the minimum detectable effect (MDE, the smallest uplift worth detecting), the significance level and the power. Then calculate how many visitors you need.

With a baseline of 4.00% and a 10% relative MDE (to 4.40%), 80% power and α = .05, you need 39,475 visitors per variant, 78,950 in total. At 3,000 test visitors a day, that takes 27 days; run for four full weeks so every weekday is covered equally. The table shows why small effects are so expensive to detect: a 5% uplift needs 154,304 visitors per variant, a 2% uplift almost a million.

Step 2: do not peek

Decide the duration in advance and evaluate only at the end. If you check the p-value every day and stop as soon as it drops below .05, the real false-positive rate can exceed 20%, because random fluctuations will eventually cross the threshold. If you need to monitor, use a sequential testing method designed for it, or the Bayesian approach with a pre-set decision rule.

Step 3: check the traffic split

Before reading the result, check that the randomisation worked. A sample ratio mismatch (SRM), where the observed split differs from the planned split by more than chance, signals a technical problem: bots filtered unevenly, a redirect that loses users in one variant, or tracking that fires differently.

In the example, a planned 50/50 test has 50,421 users in control and 49,283 in the variant. The difference looks small (50.57% versus 49.43%), but with almost 100,000 users it is highly significant: χ²(1) = 12.99, p < .001. This is an SRM. Do not trust the conversion results until the cause is found; SRM checks use a strict threshold (p < .001) because they are run on every test.

Step 4: evaluate the result (frequentist)

Conversion is 8.01% in A and 9.42% in B, a relative uplift of 17.6%. The difference of 1.41 percentage points has a 95% CI of [0.28, 2.53] and p = .014: significant at 95% confidence. Report the CI, not just the p-value: the true uplift could be as small as 0.28 points, which matters for the business case.

Step 4, alternative: the Bayesian view

Managers often ask “what is the probability that B is better?”, which a p-value does not answer. A Bayesian analysis does. With 96 of 1,200 conversions in A and 121 of 1,180 in B, P(B > A) = 97.2%, the expected relative uplift is 29.0% (95% credible interval −0.8% to 65.1%), and the expected loss of choosing B, the conversion you give up on average if B is actually worse, is only 0.013 percentage points. A common decision rule is to ship when the expected loss falls below a small threshold you set in advance.

More than two variants

With a control and three variants, there are three comparisons with the control, and the chance of a false positive rises. In the example, the overall χ²(3) = 8.17 (p = .043) suggests some difference, and Variant B beats the control with p = .042. But after the Holm correction for three comparisons, p = .126: not significant. This is exactly the false-positive risk the correction guards against. Plan the number of variants in the sample size calculation.

Good practice

• One primary metric, decided in advance; guardrail metrics (revenue per visitor, bounce rate) to catch harm.
• Randomise at the user level, not the session, so users see a consistent experience.
• Beware of novelty effects: a new design can perform well for a week because it is new.
• Segment results only for pre-planned segments; slicing by many segments finds false positives.
• Record every test, including the failed ones; they are your learning base.

Reporting

In a randomised A/B test (planned 50/50, four weeks), the sample ratio was within expectation. Variant B converted at 9.42% versus 8.01% for control, a relative uplift of 17.6% (difference 1.41 percentage points, 95% CI [0.28, 2.53], p = .014).