Module 9: Capstone and interviews · Lesson 9.1 · about 90 min
Capstone: an experiment readout
Analyse a realistic 24,000-user A/B test from raw data, check the split and guardrails, estimate the lift with intervals, look at segments without cherry-picking, and write a one-page experiment readout memo for your portfolio.
Module 9 · about 90 minutes · uses all of Module 7. You’ll analyse a full test from raw data and write the kind of readout memo analysts send to product teams. It becomes a portfolio piece that shows an employer you can run an experiment end to end.
In everyday terms: a doctor’s letter: diagnosis and treatment first, test results and caveats below. Nobody wants to read the lab method before learning what is wrong.
1. The brief
Made-up company, made-up data. An online store tested a one-page checkout (B) against the current three-step checkout (A) for 14 days. Here is the plan the team froze before launch (Lesson 7.1):
| Item | Plan |
|---|---|
| Hypothesis | One-page checkout raises purchase conversion because fewer steps means fewer drop-offs |
| Unit, split | User, hashed ID, 50/50 |
| Primary metric | Share of users who purchase (converted) |
| Secondary | Revenue per user; conversion by device and new/returning |
| Guardrails | Refund rate among buyers must not rise significantly; page load must not rise by more than 100 ms |
| Size | 10,300 users per group detect 4.0 % → 4.8 % with 80 % power; planned about 12,000 for a margin |
| Decision rule | Ship if the 95 % CI for the conversion lift is above 0 and no guardrail is breached |
Download the data: checkout-test.csv (24,000 rows, 870 KB). Columns: user_id, group (A/B), device (phone/computer), visitor (new/returning), day (1 to 14), converted (0/1), revenue ($, 0 if no purchase), refunded (0/1), page_load_ms.
2. Your tasks
- Check the split (Lesson 7.6): counts per group and the chi-square p-value.
- Primary metric (Lesson 7.5): conversion in each group with 95 % intervals, absolute and relative lift with a 95 % CI, z and p. Enter your counts below to check them.
- Guardrails: refund rate among buyers in each group (two-proportion test); mean page load in each group and the difference with a 95 % CI.
- Secondary: revenue per user with a Welch t-test or a bootstrap (Lesson 4.5), since revenue is mostly zeros.
- Segments: conversion by device and by new/returning, as a consistency check only.
- Novelty check (Lesson 7.8): lift by day or by week. Is it fading?
- Decision: apply the rule and write the memo (section 4).
| Visitors | Conversions | |
|---|---|---|
| A (control) | ||
| B (new) |
Tools that do each step in your browser: CSV profiler to look at the file first; grouped data for counts and means by group; hypothesis tests for the proportion and Welch tests; bootstrap test for revenue; bayesian A/B test for the chance B is better. Or use Python: pandas Lesson 8.6 does this whole readout in code.
3. What a strong analysis shows
- Numbers with units and intervals, never a bare p-value.
- Checks before results: split, data quality (missing values, impossible values, duplicates).
- Guardrails judged by size against the limit in the plan, not by significance alone.
- Segments as context, clearly labelled as not decisive.
- One clear recommendation, with what it’s expected to be worth (extra orders or revenue per month) and the main risk.
- Decision
- Ship, do not ship, or keep testing, by the rule written in the plan.
- Effect
- Absolute and relative lift with 95 % CIs, in business units.
- Checks
- Split, data quality, guardrails, novelty.
- Segments
- Context only: consistent or not, never the basis of the decision.
- Physical meaning
- The memo turns numbers into an action someone can take on Monday, with the risk stated plainly. Its value is the decision it supports, not the number of tests it reports.
“More tables make a stronger memo.” One clear decision, the key interval and the checks beat ten tables. Put the rest in an appendix.
Watch it happen, one step at a time:
Maths and principle
Principle: the readout answers three questions in order: can we trust the data (split, quality), what happened (estimate and interval), and what should we do (decision rule, value, risk).
Formulas you’ll use: split check (Lesson 7.6); difference in proportions with its interval and pooled z (Lesson 7.5); Welch’s t for means (Lesson 6.3):
Value: extra orders per month ≈ absolute lift × monthly eligible users; give the range using the CI’s ends.
4. The memo (one page)
Write it in this order. Aim for about 400 words plus one table and one chart.
- Title and one-line recommendation. “Ship the one-page checkout.”
- Headline result. The lift with its interval, in plain words, and what it’s worth.
- Was the test sound? Split, data checks, duration.
- Guardrails. Each with its result against its limit.
- Supporting evidence. Revenue per user, segments, novelty check.
- Risks and next steps. What could make this wrong; what to monitor after launch.
- Appendix. The frozen plan, the method, a link to your analysis (notebook or spreadsheet).
For your portfolio: publish the memo as a project page (on nihalsulaiman.com, under Projects, or your own site), with the chart, the table and a link to the code on GitHub. In interviews, walk through it in two minutes: the question, the checks, the result, the decision.
Answer key (check your numbers after you've done the work)
- Split: A 11,968, B 12,032; χ² p = 0.68. Fine.
- Conversion: A 493 / 11,968 = 4.12 %; B 586 / 12,032 = 4.87 %. Lift +0.75 points (95 % CI +0.23 to +1.28), +18 % relative; z = 2.81, p = 0.005.
- Refunds among buyers: A 6.7 %, B 6.0 %; no significant rise. Pass.
- Page load: B is 46 ms slower (95 % CI about 37 to 56 ms), p < 0.0001: real, but below the 100 ms limit. Pass, and mention it.
- Revenue per user: A $2.42, B $2.87; Welch t = 2.37, p = 0.018.
- Segments: B higher on phones (2.7 % → 3.3 %) and computers (6.4 % → 7.4 %), and for new (3.8 % → 4.4 %) and returning visitors (4.5 % → 5.5 %). Consistent.
- Novelty: week 1 lift +0.64 points, week 2 +0.86 points. No sign of fading.
- Decision: ship. Every condition of the frozen rule is met.
New words: readout, pre-registration, decision memo, guardrail breach.
- Lead with the decision and the effect size with its interval.
- Show checks (split, guardrails, novelty) before secondary results.
- Segments are context, not the basis of the decision.
- Translate the lift into orders or money per month.
Check yourself
Sources
- Kohavi, R., Tang, D. and Xu, Y. (2020), Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, Cambridge University Press, Chapters 3 and 17
- OpenIntro Statistics, 4th edition (Diez, Cetinkaya-Rundel, Barr), Section 6.2 Difference of two proportions https://www.openintro.org/book/os/
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.