Module 7: A/B testing · Lesson 7.8 · about 20 min
A/B testing pitfalls
Novelty effects, the winner's curse, too many metrics and segments, users who interact with each other, and the habit of trusting results that look too good. The checks that catch them.
Module 7 · about 20 minutes · no new maths. You’ll be able to spot the common ways a “winning” test misleads, and know the check for each.
In everyday terms: a new café on the high street is packed in its first week because it is new. Judging its long-term business by that week overestimates it badly. A redesign gets the same first-week bump.
1. Novelty and learning effects
People react to change itself. A shiny new button gets clicks because it’s new (novelty effect); a rearranged menu annoys regulars at first (learning effect, or change aversion). Either way, the first days mislead.
With the starting settings, stopping after one week estimates about +5 %; the true long-run lift is +1 %. Run to six weeks and the running estimate falls to about +2 %, much closer; the last weeks alone sit right on +1 %. Set the launch value negative to see a change that looks bad at first and comes good.
Checks: plot the lift by day; compare new users (who have nothing to be surprised by) with returning ones; run at least two full weeks; for big changes, keep a small holdout group on the old version for months.
2. The winner’s curse
If a test has low power, it only reaches significance when noise pushes the estimate up. So significant results from small tests overstate the effect, sometimes by double. Over a portfolio of tests, the sum of all “wins” is bigger than the real gain.
Checks: plan for power (Lesson 7.3); treat a surprising win as a hypothesis and replicate it; shrink estimates towards zero (a Bayesian prior, Lesson 7.7, does this).
3. Too many metrics and segments
Twenty metrics or twenty segments (“it worked for Android users in Ontario on weekends!”) give about one false win at 5 % each (Lesson 6.8).
Checks: one primary metric decided in advance (Lesson 7.2); treat segment findings as ideas for the next test; if you must look at many, correct for it (Holm or Benjamini-Hochberg).
4. Interference between users
Standard tests assume one user’s group doesn’t affect another’s outcome. That fails when users share something:
- Marketplaces: a discount for buyers in B uses up stock that buyers in A wanted.
- Social features: a sharing tool in B sends messages to users in A.
- Shared budgets: an ad test where both groups bid from the same budget.
Checks: randomise larger units (cities, time windows, clusters of friends) and analyse at that level.
5. Twyman’s law
“Any figure that looks interesting or different is usually wrong.” A +30 % lift from a button colour is almost always a bug: a tracking change, SRM (Lesson 7.6), bots, or a metric that double counts. Check the plumbing before celebrating.
- Novelty effect
- Users react to anything new; the effect fades.
- Winner's curse
- Tests that pass with low power overstate the true effect.
- Interference
- Users in A and B affect each other (marketplaces, social features), so the groups are not separate.
- Non-additivity
- Two +3 % wins on the same page rarely give +6 %.
- Physical meaning
- A test measures a short window in an artificial setting. The long-run effect is usually smaller, so trust repeated, well-powered results and holdout groups over single exciting numbers.
“12 winning tests at +1.5 % each mean +18 % this quarter.” Inflated winners, novelty and overlap mean the real total is usually far smaller. A long-running holdout group shows what all the changes really added.
Watch it happen, one step at a time:
Maths and principle
Principle: selecting results on significance biases their size upward.
How big is the curse? If the true effect is and the test’s SE is , a significant estimate must exceed about . When is well below that (low power), the average significant estimate is close to or more, however small is.
Example: true lift 0.5 points, SE 0.5 points (power about 17 %). Significant positive estimates average about 1.25 points, two and a half times the truth.
Novelty: if the lift on day is , the average over the first days is biased upwards by about once is several times , so longer tests shrink the bias.
6. Worked example
Made-up story. A team’s quarterly report lists 12 winning tests adding up to +18 % conversion. Actual conversion over the quarter rose 3 %.
- Several tests had power around 30 %: their significant lifts were inflated (winner’s curse).
- Three wins were measured in the first week of eye-catching redesigns (novelty).
- Two “wins” were in segments picked after the fact.
- Lifts from different tests on the same page don’t simply add up.
The fix: plan power, run full weeks, one primary metric, replicate surprising wins, and keep a long-term holdout to measure the combined effect.
7. Practice
- How would you tell a novelty effect from a real lift?
- A test with 15 % power comes out significant with +8 %. What do you expect the true effect to be?
- Suggest a randomisation unit for testing a new delivery-fee rule in a food delivery app.
Answers
- Plot lift by day (novelty fades), compare new and returning users (new users show no novelty), and use a long-term holdout.
- Probably much smaller than 8 %, possibly near zero; replicate before trusting it.
- City (or city × week), because riders, restaurants and couriers are shared within a city.
Try it on your own data: the bootstrap test tool lets you see how much an estimate wobbles by chance; sample size tells you whether a test had the power to be trusted.
New words: novelty effect, learning effect, holdout group, winner’s curse, interference, Twyman’s law.
- Watch for novelty: compare early and late weeks of the test.
- Low-powered wins exaggerate effects (the winner’s curse).
- Interference between users breaks the A/B separation; randomise by cluster if needed.
- Lifts from separate tests do not simply add; use a holdout to measure the total.
Check yourself
Sources
- Kohavi, R., Tang, D. and Xu, Y. (2020), Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, Cambridge University Press, Chapters 3 and 23
- Gelman, A. and Carlin, J. (2014), Beyond Power Calculations: Assessing Type S and Type M Errors, Perspectives on Psychological Science 9(6), 641-651 https://doi.org/10.1177/1745691614551642
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.