Sample Ratio Mismatch (SRM) Checker

What a sample ratio mismatch is

Every A/B test has a planned allocation: 50% of users to the control and 50% to the variant, or 34/33/33 for a three-arm test. Because assignment is random, the observed split is never exactly the planned one, but with thousands of users it should be very close. A sample ratio mismatch (SRM) occurs when the observed split differs from the plan by more than chance allows. For example, 50,421 users in the control and 49,283 in the variant looks almost balanced (50.6% vs 49.4%), yet with nearly 100,000 users the probability of such a difference arising by chance is about 0.03%.

SRM matters because it is a symptom: something is removing or adding users unevenly between the variants, and whatever causes it is very likely related to the outcome as well. Microsoft’s experimentation platform team found SRM in about 6% of their experiments, and teams at Booking.com, LinkedIn and others report similar rates. A test with SRM cannot be trusted, however large and significant its effect looks; the right response is to find and fix the cause, not to interpret the result.

How the check works

The tool compares the observed counts with the counts expected from the planned allocation using a chi-square goodness-of-fit test:

χ² = Σ (observed − expected)² / expected, with k − 1 degrees of freedom

where k is the number of variants. For two variants it also gives the exact binomial test, which does not rely on the large-sample approximation; with the sample sizes typical of online tests, the two tests agree closely. The table shows each variant’s observed and planned share, the difference in users and the standardised residual, which points to the variant that deviates most in tests with three or more arms.

Why the threshold is strict

SRM checks are run on every experiment, often every day, so a conventional threshold of p < .05 would raise many false alarms. Industry practice is to use a much stricter threshold: p < .001 is a good default, and several large platforms use p < .0005. With large samples, real mismatches usually produce p-values far below these thresholds, so the strict cut-off costs little sensitivity.

Worked example

The default example enters 50,421 and 49,283 users for a planned 50/50 split. The χ²(1) statistic is 12.99 and the exact binomial p-value is about .0003, below the .001 threshold: this test has a sample ratio mismatch. The control has 569 more users than expected (standardised residual 2.55). Before reading the conversion results, the team should investigate why roughly 1,100 users are missing from the variant, for example a slower page that loses users before tracking fires.

Common causes of SRM

• Redirects and page speed: if the variant redirects users or loads more slowly, some users leave before they are counted, and those users differ from the rest.
• Bots and filtering: bot-detection or data-cleaning rules that interact with the variant (for example a change that makes bots easier or harder to detect).
• Tracking differences: the analytics event that counts exposure fires differently in the variants, or is blocked by ad blockers in one variant.
• Assignment bugs: faulty hashing, users switching between variants, caching that serves one variant more often, or a variant that crashes on some browsers.
• Triggering conditions: users are counted only when they reach a feature that behaves differently between variants.
• Ramp-ups: changing the allocation during the test (for example from 10/90 to 50/50) while analysing all data with the final split.
• Interference from other experiments running at the same time on the same users.

How to investigate an SRM

1. Check the split by day: a mismatch that starts on a particular date points to a deployment or tracking change.
2. Break the split down by browser, device, operating system, country and new vs returning users: mismatches concentrated in one segment point to the cause.
3. Compare the counts at each step of the pipeline: assignment logs, exposure events, analytics sessions. The step at which the ratio breaks is where users are lost.
4. Look for differences in page performance and error rates between variants.
5. Fix the issue and rerun the experiment; do not try to rescue the data by reweighting, because the lost users are not missing at random.

What to enter

Enter the number of units of randomisation in each variant (users, devices or sessions, whichever your platform assigns), separated by commas, and the planned allocation as percentages, for example 50, 50 or 20, 40, 40. Variant names are optional. Do not enter conversions: SRM is about who was assigned, not about outcomes. If the allocation changed during the test, check each period with its own planned split.

SRM and other experiment checks

An SRM check belongs at the start of every analysis, together with a check that pre-experiment metrics are balanced (an A/A comparison of behaviour before the test started) and that the test ran for its planned duration. Once the split passes, analyse the result with the A/B test significance calculator, the Bayesian A/B test calculator or, for several variants, the A/B/n test calculator, all on this site. Plan future tests with the sample size and duration calculator.

Reporting

Report the observed and planned split, the test and p-value, the threshold and the decision. Example: “A sample ratio mismatch check showed that the observed split (50.57% / 49.43%, N = 99,704) deviated from the planned 50/50 allocation, χ²(1) = 12.99, p < .001; the experiment was paused to investigate.”

Glossary of SRM terms

• Planned allocation: the intended share of users in each variant, for example 50/50 or 10/90 during a ramp-up.
• Goodness-of-fit test: a chi-square test of whether observed counts match the expected counts implied by the planned allocation.
• Exact binomial test: for two variants, the exact probability of a split at least as extreme as the one observed under the planned ratio.
• Standardised residual: (observed − expected) / √expected for one variant; values beyond about ±3 flag the variant that deviates most.
• Triggered analysis: an analysis restricted to users who reached the changed feature; triggering conditions are a common source of SRM.
• A/A test: an experiment in which both arms receive the same experience; it checks the platform and should show neither SRM nor metric differences.
• Ramp-up: gradually increasing a variant’s share of traffic; analyse each allocation period separately.

Key references: Fabijan et al. (2019), “Diagnosing sample ratio mismatch in online controlled experiments”, KDD; Kohavi, Tang and Xu (2020), Trustworthy Online Controlled Experiments.