Module 7: A/B testing · Lesson 7.1 · about 25 min
Designing an A/B test
The parts of a good online experiment: a clear hypothesis, the unit you randomise, a fair random split, one primary metric, a planned sample size and duration, and a written decision rule, all fixed before the test starts.
Module 7 · about 25 minutes · no new maths. You’ll be able to design an A/B test: state a hypothesis, choose the randomisation unit and metric, and write a plan that holds up when the results come in.
In everyday terms: a fair taste test. You decide in advance which two recipes, who tastes (picked at random, not friends), what counts as “better”, how many tasters, and what you will do with the answer, before anyone takes a bite.
1. Why experiments
Observational data (Lesson 4.2) is full of self-selection: customers who use a feature differ from those who don’t. An A/B test (randomised controlled experiment) removes that problem by letting chance decide who gets the change.
2. The parts of a design
1. Hypothesis. “Showing delivery dates on product pages will raise purchase conversion, because uncertainty about delivery stops people buying.” A because gives you something to learn even if the test loses.
2. Randomisation unit. Who gets assigned: user, account, session, store, region. Choose the unit that experiences the change and that you analyse. Usually the user: assigned by a stable ID so they see the same version on every visit. If you randomise users, analyse per user (not per page view), or the standard errors are wrong.
3. Population and trigger. Who is eligible (all visitors? logged-in? Canada only?), and when does someone count as “in” the test? Only include people who could see the difference, such as visitors who reached checkout for a checkout change. Including others adds noise.
4. Split. Usually 50/50 for the most power. A smaller treatment share (10 %) limits risk but needs a longer test.
5. Metrics. One primary metric, a few secondary, and guardrails (next lesson).
6. Sample size and duration. From the baseline, the smallest worthwhile effect and power (Lesson 7.3). Run whole weeks (weekday and weekend behaviour differ) and at least one full business cycle.
7. Decision rule. Written before launch: “Ship if the 95 % CI for the conversion lift is above 0 and no guardrail worsens significantly; otherwise keep the current page.”
3. Before launch
- A/A test (both groups get the same version) to check the pipeline: you should see no difference and a correct split.
- Check the assignment is random and stable: the split should match the plan (Lesson 7.6).
- Freeze the plan in a document with a date.
- Hypothesis
- What will change, for whom, and why.
- Unit of randomisation
- What gets assigned to A or B: a user, a session, a store. Analyse at the same level.
- Primary metric
- The one number that decides.
- Decision rule
- Written before launch: "ship if the 95 % CI for the lift is above 0 and no guardrail is breached".
- Physical meaning
- Randomisation is what lets you say "caused". Because chance alone decides who sees B, the two groups are alike in everything except the change, so a difference in results can only come from the change or from luck, and statistics measures the luck.
“We can decide what success means after we see the data.” Choosing the metric, segment or end date after looking lets luck pick your winner. Write the plan first and keep extra findings as ideas for the next test.
Watch it happen, one step at a time:
Maths and principle
Principle: with random assignment, the expected difference between groups in anything other than the treatment is zero, so the measured difference estimates the treatment’s causal effect.
Equation: average treatment effect
estimated by with standard error (Lesson 4.4).
Hashing: a common way to assign users stably is hash(user_id + test_name) mod 100 < 50 → B. Including the test name keeps different tests’ assignments independent.
4. Worked example
Made-up plan for “show delivery dates”:
| Item | Plan |
|---|---|
| Hypothesis | Delivery dates raise purchase conversion by reducing uncertainty |
| Unit | Logged-in user, hashed ID, 50/50 |
| Trigger | User views a product page during the test |
| Primary metric | Share of triggered users who purchase within 7 days |
| Secondary | Add-to-cart rate; average order value |
| Guardrails | Cancellations; “where is my order” contacts; page load time |
| Baseline / MDE | 4.0 %; smallest worthwhile lift +10 % relative (to 4.4 %) |
| Size / duration | About 39,500 users per group; at 6,000 eligible users a day, 14 days (2 full weeks) |
| Decision rule | Ship if 95 % CI for lift > 0 and no guardrail worse at p < 0.05 |
5. Practice
- A store tests a new shelf layout. What is a sensible randomisation unit, and why not the customer?
- Why should a checkout test only include users who reach checkout?
- Write a decision rule for a test whose goal is “don’t hurt revenue while simplifying the menu”.
Answers
- The store (or store-week): customers can’t be shown different shelves in the same store. With store-level randomisation, analyse per store, and you’ll need many stores.
- Users who never reach checkout can’t be affected; including them adds noise and dilutes the effect, so you need more traffic for the same power.
- Example: “Ship the simpler menu if the 95 % CI for the change in revenue per user is above −1 % (a non-inferiority test) and support contacts don’t rise.”
Try it on your own data: the sample size calculator plans the test size; the bayesian A/B test tool reads finished tests.
New words: A/B test, randomised controlled experiment, randomisation unit, trigger, A/A test, decision rule, average treatment effect, non-inferiority.
- Write the hypothesis, unit, metrics, size, duration and decision rule before launch.
- Randomise the same unit you analyse (usually the user).
- Run whole weeks to cover weekday patterns.
- Randomisation is what turns a difference into evidence of cause.
Check yourself
Sources
- Kohavi, R., Tang, D. and Xu, Y. (2020), Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, Cambridge University Press, Chapters 2-3
- OpenIntro Statistics, 4th edition (Diez, Cetinkaya-Rundel, Barr), Section 1.4 Experiments https://www.openintro.org/book/os/
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.