A/B/n Test Calculator with Multiple-Comparison Correction

Testing more than two variants

An A/B/n test compares a control with two or more variants at the same time, for example four headlines, three prices or five button designs. It is efficient, because all variants share one control and run simultaneously under the same conditions, but it creates a statistical trap: the more comparisons you make, the more likely at least one of them is “significant” by chance. With four variants each compared with the control at α = .05, the probability of at least one false positive is about 1 − 0.95⁴ ≈ 19%, not 5%. With ten variants it is 40%. Without correction, A/B/n tests regularly crown winners that are just noise.

What this calculator does

1. Conversion rates for each variant with 95% Wilson confidence intervals, which behave well even for low rates.
2. An overall chi-square test of whether all variants have the same conversion rate (a k × 2 test of homogeneity). A significant result says that at least one variant differs, but not which.
3. Pairwise comparisons, either each variant against the control (the usual design) or all pairs, using the two-proportion z-test with a pooled standard error. For each comparison it reports the difference in percentage points with a 95% confidence interval, the relative uplift with a confidence interval (computed on the log of the rate ratio), the z statistic and the raw p-value.
4. Adjusted p-values using your chosen multiple-comparison correction, and a verdict (better, worse or no difference) based on the adjusted p-value.

Choosing a correction

• Holm (recommended). A step-down version of Bonferroni: the smallest p-value is multiplied by m (the number of comparisons), the next smallest by m − 1, and so on, with monotonicity enforced. It controls the family-wise error rate (the probability of any false positive) exactly like Bonferroni but is uniformly more powerful, so there is no reason to prefer plain Bonferroni.
• Bonferroni. Multiply every p-value by m. Simple and conservative.
• Šidák. 1 − (1 − p)^m, slightly less conservative than Bonferroni when comparisons are independent.
• Benjamini-Hochberg. Controls the false discovery rate, the expected share of false positives among the comparisons declared significant, rather than the chance of any false positive. It is more powerful and suits exploratory screening of many variants, where a few false leads are acceptable.
• None. Shows unadjusted results, for comparison only.

Choose the correction before you look at the results, and comparing each variant with the control rather than all pairs reduces the number of comparisons and so increases power.

Worked example

The default example has four arms with about 10,000 visitors each: Control 8.20%, Variant B 9.00%, Variant C 8.63% and Variant D 7.97%. The overall chi-square test is significant (χ²(3) = 8.17, p = .043). Compared with the control, Variant B shows a 9.8% relative uplift with a raw p-value of .042, which would be declared a winner in a naive analysis. After Holm’s correction for three comparisons, its adjusted p-value is .126, so it is not a statistically reliable winner. This is exactly the situation the correction is designed for. A sensible next step is a follow-up A/B test of Variant B against the control, with a sample size planned for a lift of about 10%.

Entering the data

Enter one line per variant, control first, with the variant name, the number of visitors and the number of conversions, separated by commas, semicolons or tabs, for example Variant B, 10050, 905. You can paste directly from a spreadsheet. Use the same unit of analysis for all variants (users, not sessions in one and users in another), and count each user once.

Interpreting the results

• A variant is a confirmed winner only if its adjusted p-value is below α and the uplift is positive.
• Look at the confidence intervals of the uplift: a significant result with an interval of +0.5% to +20% is uncertain about the size of the benefit.
• When no variant wins, compare the observed differences with the minimum detectable effect the test was designed for; if the test was underpowered, “no difference” is inconclusive.
• If the overall test is significant but no pairwise comparison survives correction, the evidence suggests some difference exists but is not strong enough to pick a specific winner; consider a follow-up test of the most promising variants.

Good practice for multi-variant tests

• Plan the sample size per variant in advance with the A/B test sample size calculator, including the Bonferroni adjustment for the number of variants.
• Check the traffic split with the sample ratio mismatch checker before reading the results.
• Limit the number of variants to those you would actually ship; every extra variant dilutes traffic and power.
• Avoid stopping early when one variant looks ahead: the winner’s curse means the best-looking variant in a multi-arm test is likely to have an overestimated effect.
• For a probabilistic interpretation of two-variant tests, the Bayesian A/B test calculator gives the probability that one variant beats another.

Reporting

Report each variant’s rate with its CI, the overall test, the correction used, and for each comparison the uplift with its CI and the adjusted p-value. Example: “Conversion rates differed across the four arms, χ²(3) = 8.17, p = .043. Compared with the control (8.20%), Variant B converted at 9.00% (+9.8%, 95% CI [+0.3%, +20.2%]); after Holm correction for three comparisons the difference was not significant (adjusted p = .126).”

Glossary of multiple-testing terms

• Family-wise error rate (FWER): the probability of at least one false positive among all comparisons in the test; Holm, Bonferroni and Šidák keep it at α.
• False discovery rate (FDR): the expected proportion of false positives among the comparisons declared significant; Benjamini-Hochberg controls it.
• Adjusted p-value: a p-value transformed so that it can be compared directly with α while respecting the chosen error rate across all comparisons.
• Pooled vs unpooled standard error: the z-test pools the two groups under the null hypothesis of no difference; confidence intervals use separate (unpooled) variances.
• Winner’s curse: the tendency for the best-looking of several variants to have an inflated estimated effect, because it partly won by luck.
• Omnibus test: a single test of whether any difference exists among all variants, here the chi-square test of homogeneity.
• Practical significance: whether an uplift is large enough to matter commercially, judged with the confidence interval rather than the p-value alone.

Key references: Holm (1979), Scandinavian Journal of Statistics; Benjamini and Hochberg (1995), Journal of the Royal Statistical Society B; Kohavi, Tang and Xu (2020), Trustworthy Online Controlled Experiments.