A/B Test Sample Size and Duration Calculator

Why plan the sample size?

An A/B test that is too small cannot detect realistic effects: it will usually end “not significant” even when the variant is genuinely better, wasting traffic and time, and the few significant results it produces will overstate the true effect. A test that is much larger than needed delays decisions. Planning the sample size before launch, and sticking to it, is the single most effective way to make experiments trustworthy. It also forces a useful conversation: what is the smallest improvement that would be worth acting on?

The ingredients

• Baseline: the current conversion rate of the control (for example 4% of visitors buy), or for continuous metrics such as revenue per visitor or time on site, the current mean and standard deviation. Take these from recent data for the same pages and audience.
• Minimum detectable effect (MDE): the smallest relative change you want the test to detect reliably, for example a 10% lift from 4.0% to 4.4%. Smaller MDEs need dramatically larger samples: halving the MDE roughly quadruples the sample size.
• Significance level (α): the false-positive rate you accept, usually .05 two-sided.
• Power (1 − β): the probability of detecting the MDE if it is real, usually 80% (90% for important decisions).
• Number of variants: each additional variant needs its own sample, and with several comparisons against the control the significance level is adjusted (Bonferroni) to keep the overall false-positive rate at α.
• Traffic: eligible visitors per day and the share of them included in the test, which turn a sample size into a duration.

The formulas

For conversion rates, the tool uses the standard two-proportion z-test formula for the sample size per variant:

n = [z_α√(2 p̄(1 − p̄)) + z_β√(p₁(1 − p₁) + p₂(1 − p₂))]² / (p₂ − p₁)²

where p₁ is the baseline, p₂ = p₁(1 + MDE), p̄ their average, z_α the critical value for α (α/2 for two-sided tests) and z_β the quantile for the chosen power. For continuous metrics it uses n = 2(z_α + z_β)² σ² / δ², where δ is the absolute difference to detect. Duration is the total sample (n × number of variants) divided by daily test traffic, rounded up to whole days, and the tool recommends rounding up to full weeks so that every day of the week is represented equally.

In the second mode, you enter a planned duration and the tool solves for the MDE your traffic can detect, which is often the more honest question for low-traffic sites.

Worked example

With a 4% baseline conversion rate, a 10% relative MDE (4.0% → 4.4%), α = .05 two-sided and 80% power, each variant needs 39,475 visitors, 78,950 in total. With 3,000 visitors a day split between two variants, that takes 27 days, so the recommendation is to run four full weeks. The table of other effect sizes shows how quickly requirements grow for smaller effects: detecting a 5% lift needs about 154,000 visitors per variant (103 days), while a 20% lift needs about 10,300 (7 days). The detectable-effect chart shows that after two weeks only lifts of about 14% or more can be detected reliably with this traffic.

Choosing a sensible MDE

The MDE is a business decision, not a statistical one. Ask what lift would justify the cost of building and maintaining the change, and what lift is plausible: most successful website changes improve conversion by a few percent, not by 30%. If the sample size for a realistic MDE is unachievable, consider testing bigger changes, using a metric closer to the change (click-through rather than purchases), increasing the traffic share, running fewer variants, or using variance-reduction methods such as CUPED.

Common mistakes

• Peeking: checking results daily and stopping as soon as p < .05 inflates the false-positive rate far above 5%. Decide the sample size in advance, or use a sequential or Bayesian method designed for monitoring.
• Stopping too early: even if the sample size is reached in three days, run at least one full week to cover weekday and weekend behaviour, and beware of novelty effects.
• Using the wrong baseline: site-wide conversion rates are often different from the rate on the tested page or segment.
• Ignoring multiple variants: testing five variants at α = .05 each without correction gives about a 19% chance of at least one false winner.
• Underpowered tests reported as “no effect”: a non-significant result from a small test means “not enough evidence”, not “no difference”.

After the test

Before analysing results, run the sample ratio mismatch checker to confirm the traffic split worked. Then analyse two variants with the A/B test significance calculator or the Bayesian A/B test calculator, and three or more with the A/B/n test calculator with multiple-comparison correction.

Reporting a test plan

A test plan should state the primary metric, baseline, MDE, α, power, number of variants, sample size per variant, expected duration and the decision rule. Example: “With a baseline conversion rate of 4.0%, detecting a 10% relative lift with 80% power at α = .05 (two-sided) requires 39,475 users per variant; at 3,000 eligible users per day the test will run for four weeks.”

Glossary of experiment-planning terms

• Baseline rate: the conversion rate the control is expected to have during the test, estimated from recent traffic to the same page or flow.
• Absolute vs relative MDE: a 10% relative MDE on a 4% baseline is an absolute change of 0.4 percentage points; always state which one you mean.
• Type I error (α): declaring a difference when none exists. With α = .05, one in twenty tests of a useless change will look significant.
• Type II error (β): missing a real effect of the size of the MDE. With 80% power, one in five such effects goes undetected.
• Allocation: the share of test traffic sent to each variant. Equal splits are the most efficient; unequal splits need more total traffic for the same power.
• Unit of randomisation: users, sessions or devices. Plan and analyse with the same unit; counting sessions when users were randomised understates uncertainty.
• Novelty effect: a temporary change in behaviour because something is new. Running for full weeks and checking the effect over time helps detect it.

Key references: Kohavi, Tang and Xu (2020), Trustworthy Online Controlled Experiments; Fleiss, Levin and Paik (2003), Statistical Methods for Rates and Proportions; Deng, Xu, Kohavi and Walker (2013) on CUPED.