All lessons

Module 9: Capstone and interviews · Lesson 9.1 · about 90 min

Capstone: an experiment readout

Analyse a realistic 24,000-user A/B test from raw data, check the split and guardrails, estimate the lift with intervals, look at segments without cherry-picking, and write a one-page experiment readout memo for your portfolio.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 9 · about 90 minutes · uses all of Module 7. You’ll analyse a full test from raw data and write the kind of readout memo analysts send to product teams. It becomes a portfolio piece that shows an employer you can run an experiment end to end.

A readout memo leads with the decision and the size of the effect, then shows the checks that make it trustworthy.

In everyday terms: a doctor’s letter: diagnosis and treatment first, test results and caveats below. Nobody wants to read the lab method before learning what is wrong.

1. The brief

Made-up company, made-up data. An online store tested a one-page checkout (B) against the current three-step checkout (A) for 14 days. Here is the plan the team froze before launch (Lesson 7.1):

Item Plan
Hypothesis One-page checkout raises purchase conversion because fewer steps means fewer drop-offs
Unit, split User, hashed ID, 50/50
Primary metric Share of users who purchase (converted)
Secondary Revenue per user; conversion by device and new/returning
Guardrails Refund rate among buyers must not rise significantly; page load must not rise by more than 100 ms
Size 10,300 users per group detect 4.0 % → 4.8 % with 80 % power; planned about 12,000 for a margin
Decision rule Ship if the 95 % CI for the conversion lift is above 0 and no guardrail is breached

Download the data: checkout-test.csv (24,000 rows, 870 KB). Columns: user_id, group (A/B), device (phone/computer), visitor (new/returning), day (1 to 14), converted (0/1), revenue ($, 0 if no purchase), refunded (0/1), page_load_ms.

2. Your tasks

  1. Check the split (Lesson 7.6): counts per group and the chi-square p-value.
  2. Primary metric (Lesson 7.5): conversion in each group with 95 % intervals, absolute and relative lift with a 95 % CI, z and p. Enter your counts below to check them.
  3. Guardrails: refund rate among buyers in each group (two-proportion test); mean page load in each group and the difference with a 95 % CI.
  4. Secondary: revenue per user with a Welch t-test or a bootstrap (Lesson 4.5), since revenue is mostly zeros.
  5. Segments: conversion by device and by new/returning, as a consistency check only.
  6. Novelty check (Lesson 7.8): lift by day or by week. Is it fading?
  7. Decision: apply the rule and write the memo (section 4).
Try it: read an A/B test
VisitorsConversions
A (control)
B (new)
rates A / B
lift (points)
relative lift
z, p (two-sided)
sample ratio check

Top: each group's rate with its 95 % Wilson interval. Bottom: the difference B − A with its 95 % interval, against zero (dashed). The verdict reads the interval first, then the p-value, and refuses to read anything if the split between groups looks broken (Lesson 7.6). Starting numbers are made up.

Tools that do each step in your browser: CSV profiler to look at the file first; grouped data for counts and means by group; hypothesis tests for the proportion and Welch tests; bootstrap test for revenue; bayesian A/B test for the chance B is better. Or use Python: pandas Lesson 8.6 does this whole readout in code.

3. What a strong analysis shows

  • Numbers with units and intervals, never a bare p-value.
  • Checks before results: split, data quality (missing values, impossible values, duplicates).
  • Guardrails judged by size against the limit in the plan, not by significance alone.
  • Segments as context, clearly labelled as not decisive.
  • One clear recommendation, with what it’s expected to be worth (extra orders or revenue per month) and the main risk.
Decision
Ship, do not ship, or keep testing, by the rule written in the plan.
Effect
Absolute and relative lift with 95 % CIs, in business units.
Checks
Split, data quality, guardrails, novelty.
Segments
Context only: consistent or not, never the basis of the decision.
Physical meaning
The memo turns numbers into an action someone can take on Monday, with the risk stated plainly. Its value is the decision it supports, not the number of tests it reports.

“More tables make a stronger memo.” One clear decision, the key interval and the checks beat ten tables. Put the rest in an appendix.

Watch it happen, one step at a time:

Check the split. 11,968 against 12,032: χ² p ≈ 0.68, fine.
Primary. Conversion 4.12 % → 4.87 %: +0.75 points (CI +0.23 to +1.28), p = 0.005.
Guardrails. Load +46 ms (CI 37 to 56), inside the 100 ms limit. Refunds unchanged.
Segments. B wins on phones and computers, new and returning: consistent.
Decide. Ship. About 7.5 extra orders per 1,000 visitors.
Maths and principle

Principle: the readout answers three questions in order: can we trust the data (split, quality), what happened (estimate and interval), and what should we do (decision rule, value, risk).

Formulas you’ll use: split check χ2=∑(O−E)2/E (Lesson 7.6); difference in proportions with its interval and pooled z (Lesson 7.5); Welch’s t for means (Lesson 6.3):

t=y‾B−y‾AsA2/nA+sB2/nB

Value: extra orders per month ≈ absolute lift × monthly eligible users; give the range using the CI’s ends.

4. The memo (one page)

Write it in this order. Aim for about 400 words plus one table and one chart.

  1. Title and one-line recommendation. “Ship the one-page checkout.”
  2. Headline result. The lift with its interval, in plain words, and what it’s worth.
  3. Was the test sound? Split, data checks, duration.
  4. Guardrails. Each with its result against its limit.
  5. Supporting evidence. Revenue per user, segments, novelty check.
  6. Risks and next steps. What could make this wrong; what to monitor after launch.
  7. Appendix. The frozen plan, the method, a link to your analysis (notebook or spreadsheet).

For your portfolio: publish the memo as a project page (on nihalsulaiman.com, under Projects, or your own site), with the chart, the table and a link to the code on GitHub. In interviews, walk through it in two minutes: the question, the checks, the result, the decision.

Answer key (check your numbers after you've done the work)
  • Split: A 11,968, B 12,032; χ² p = 0.68. Fine.
  • Conversion: A 493 / 11,968 = 4.12 %; B 586 / 12,032 = 4.87 %. Lift +0.75 points (95 % CI +0.23 to +1.28), +18 % relative; z = 2.81, p = 0.005.
  • Refunds among buyers: A 6.7 %, B 6.0 %; no significant rise. Pass.
  • Page load: B is 46 ms slower (95 % CI about 37 to 56 ms), p < 0.0001: real, but below the 100 ms limit. Pass, and mention it.
  • Revenue per user: A $2.42, B $2.87; Welch t = 2.37, p = 0.018.
  • Segments: B higher on phones (2.7 % → 3.3 %) and computers (6.4 % → 7.4 %), and for new (3.8 % → 4.4 %) and returning visitors (4.5 % → 5.5 %). Consistent.
  • Novelty: week 1 lift +0.64 points, week 2 +0.86 points. No sign of fading.
  • Decision: ship. Every condition of the frozen rule is met.

New words: readout, pre-registration, decision memo, guardrail breach.

  • Lead with the decision and the effect size with its interval.
  • Show checks (split, guardrails, novelty) before secondary results.
  • Segments are context, not the basis of the decision.
  • Translate the lift into orders or money per month.

Check yourself

1In what order should the readout memo present things?
2Page load rose by 46 ms (p < 0.0001). The plan's guardrail was "must not rise by more than 100 ms". Is the guardrail breached?
3B wins in every device and visitor segment. How should segments appear in the memo?

Sources

  • Kohavi, R., Tang, D. and Xu, Y. (2020), Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, Cambridge University Press, Chapters 3 and 17
  • OpenIntro Statistics, 4th edition (Diez, Cetinkaya-Rundel, Barr), Section 6.2 Difference of two proportions https://www.openintro.org/book/os/

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.