All lessons

Module 7: A/B testing · Lesson 7.6 · about 15 min

Sample ratio mismatch

When a 50/50 test ends with 50.5 % of users in one group, something in the setup is broken and the result cannot be trusted. How to test the split with a chi-square, and the usual causes.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 7 · about 15 minutes · uses Lesson 6.5. You’ll be able to test whether a split matches the plan and list the common causes when it doesn’t.

If a 50/50 test does not end close to 50/50, something in the setup is broken, and no result from it can be trusted until you find out what.

In everyday terms: you deal a deck of cards to two players and one ends up with 30 cards. You do not argue about who won the hand; you check how the cards were dealt.

1. Why the split matters

Randomisation makes the groups alike in everything except the change (Lesson 7.1). If users go missing from one group, or extra ones arrive, the groups stop being alike, and the result measures who is in each group as well as the change. That is a sample ratio mismatch (SRM).

It’s surprisingly common. Large experimentation teams report it in a few percent of all tests, and it usually reveals a real bug.

Try it: is the split broken?
visitors A / B
χ² p-value for 50/50
verdict (threshold p < 0.001)
The shaded band is the 99.9 % range for B's share if visitors really were split 50/50. With 1,000 visitors a 50.5 % share is nothing; with about 200,000 visitors (101,000 vs 99,000) it is far outside the band, so something in the setup is losing or adding visitors on one side. Because tests have so many visitors, the check uses a strict threshold (p < 0.001) to avoid false alarms.

Move the visitors slider: the band of “normal” shares narrows as the test grows. A 50.5 % share is nothing with 1,000 users and a red flag with 200,000.

2. The check

Compare the observed counts with the planned split using a chi-square goodness-of-fit test (Lesson 6.5) with one degree of freedom. Run it on every test, before reading any metric, and use p < 0.001 as the alarm.

3. Common causes

  • Redirects or extra page loads in one variant: users drop off (or get counted twice) before logging.
  • Tracking that fails more in one variant: a heavier page, a script that loads late, an ad blocker that blocks one design.
  • Bot filtering that treats one variant differently.
  • Assignment bugs: the hash isn’t stable, users switch groups, or a test started on one platform earlier than the other.
  • Triggering on something the change affects: “users who clicked the new button” exists only in B.
  • A ramp (10 % → 50 %) analysed as if it were one 50/50 test.

What to do: don’t “fix” it by dropping users. Find the cause, fix it, and rerun. Sometimes a mismatch limited to one browser or one day can be cut out and the rest analysed, but say so plainly.

Expected counts
Total × planned share for each group.
χ² = Σ (O − E)² ÷ E
How far the actual split is from the plan; 1 df for two groups.
SRM
Sample ratio mismatch: a split so uneven that chance is not a believable explanation (p below about 0.001).
Physical meaning
A random split cannot go far wrong on large numbers. When it does, some users are being lost or added in one group: a crash, a redirect, a bot filter. Those missing users are rarely typical, so they bias the comparison.

“The split is only off by 1.5 %, so it is close enough.” With 100,000 users, a 51.5/48.5 split is wildly unlikely by chance (p far below 0.0001). Small percentage gaps on big samples are red flags.

Watch it happen, one step at a time:

The split. A 52,140, B 50,610; planned 50/50.
Expected. 102,750 ÷ 2 = 51,375 each.
χ². 2 × 765² ÷ 51,375 ≈ 22.8 with 1 df: p ≈ 0.000002. That is an SRM.
Hunt. Split by browser: the whole gap is in one old mobile browser, where B's new gallery crashed before tracking fired.
Decide. No conclusion. Those users spend less, so B's "+4 % revenue" is partly an artefact. Fix and rerun.
Maths and principle

Principle: the split itself is a random outcome with a known distribution, so it can be tested like any other.

Equation (observed OA,OB; expected EA=nqA, EB=nqB for planned shares q):

χ2=(OA−EA)2EA+(OB−EB)2EB,df=1

Check: 101,000 vs 99,000 against 100,000 each: χ2=2×10002/100,000=20, p ≈ 0.000008. The same 50.5 % share with 1,000 users (505 vs 495) gives χ2=0.1, p = 0.75.

Rule of thumb: for a 50/50 split, B’s share has SD 0.25/n; the 99.9 % band is about 0.5±3.290.25/n.

4. Worked example

Made-up story. A test of a new product page shows +4 % revenue per user. The split: A 52,140, B 50,610 (planned 50/50).

  • Expected 51,375 each. χ2=2×7652/51,375≈22.8, p ≈ 0.000002. SRM.
  • Breaking it down by browser: the gap is all in one older mobile browser, where B’s new image gallery crashed the page before the tracking fired.
  • Those users (likely on older, cheaper phones, who spend less) are missing from B, which explains much of B’s “higher revenue per user”.
  • Decision: no conclusion. Fix the gallery, rerun.

5. Practice

  1. 50/50 test: 5,060 vs 4,940. SRM?
  2. 90/10 test (90 % control): 90,500 vs 9,500. SRM?
  3. Why not just reweight the groups to 50/50?
Answers
  1. Expected 5,000 each: χ2=2×602/5,000=1.44, p = 0.23. Fine.
  2. Expected 90,000 and 10,000: χ2=5002/90,000+5002/10,000≈2.8+25=27.8, p < 0.000001. SRM.
  3. Reweighting fixes the counts, not the mix: the missing users are a particular kind of user, and weighting the rest can’t bring them back.

Try it on your own data: the cross-tab and chi-square tool runs goodness-of-fit tests on counts; the readout in Lesson 7.5 checks the split automatically.

New words: sample ratio mismatch (SRM), goodness of fit, ramp-up, triggering bias.

  • Always test the split before reading results.
  • χ² on the group counts against the planned shares; p < 0.001 means SRM.
  • Look for the cause by browser, device, day and entry point.
  • An SRM test gives no trustworthy result; fix the cause and rerun.

Check yourself

1A 50/50 test has 101,000 users in A and 99,000 in B. What should you do?
2Why do SRM checks use a strict threshold such as p < 0.001?
3B has a heavier page that sometimes fails to load the tracking script. What happens?

Sources

  • Fabijan, A. et al. (2019), Diagnosing Sample Ratio Mismatch in Online Controlled Experiments, KDD 2019 https://doi.org/10.1145/3292500.3330722
  • Kohavi, R., Tang, D. and Xu, Y. (2020), Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, Cambridge University Press, Chapter 21

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.