All lessons

Module 8: Analyses employers ask for · Lesson 8.6 · about 25 min

Analysing an A/B test in pandas

Read a real-shaped A/B test file, check the split, compute conversion and revenue per visitor for each group, the lift with a 95 % confidence interval and a p-value using only pandas and NumPy, and look at segments without fooling yourself.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 8 · about 25 minutes · uses Statistics Module 7. You’ll be able to go from a raw experiment file to a readout, in code.

An A/B readout in pandas is four steps: check the split, compute each group's rate, measure the difference with its uncertainty, then check guardrails before you call it.

In everyday terms: a fair taste test: first check both groups really had the same number of tasters, then count who preferred each cup, then ask whether the gap could be chance.

This lesson uses the experiment file from the Statistics course: 24,000 visitors split between the old checkout (A) and a new one (B). The ideas behind each number are in Statistics Lesson 7.5; here we do it in pandas.

1. Look and check the split

Pythoncheckout-test.csv

A p-value well above 0.01 means the split looks healthy. Each row is one visitor: converted is 1 if they bought.

2. The headline numbers

Pythoncheckout-test.csv

These match the Statistics capstone: about +0.75 points, CI from about +0.23 to +1.28, p ≈ 0.005. Everything came from one groupby; the rest is arithmetic on two rows.

3. Guardrails and segments

Pythoncheckout-test.csv
  • Guardrails: B is about 46 ms slower to load. Is that acceptable? A decision for the team, written in the readout.
  • Segments: B wins on both phones and computers, which makes the overall result more believable. If one segment flipped, treat it as a question for a new test, not a finding.
Sample ratio check
Users per group should match the planned split (Statistics Lesson 7.5).
groupby('group').agg(users=..., conv=...)
One row per group.
rate = conv ÷ users
Conversion per group.
Lift
Difference in rates, in percentage points, with a confidence interval.
Physical meaning
The lift is a difference between two proportions, each with sampling noise. Its standard error shrinks with more users, so the same 0.5-point lift can be convincing with 50,000 users and pure noise with 2,000.

“B won, it has the higher rate.” Every test has a winner by chance. Report the lift with its interval and p-value, and check the split and guardrails first.

Watch it happen, one step at a time:

Raw file: one row per user.
usergroupconverted
1A0
2B1
3A1
.........
Split check. value_counts() → A 5,012, B 4,988: close to 50/50.
Rates. groupby('group')['converted'].agg(['size', 'sum', 'mean'])
groupusersconvrate
A501250110.0 %
B498854911.0 %
Lift. 11.0 − 10.0 = +1.0 point; SE ≈ 0.61 points; 95 % CI about −0.2 to +2.2.
Readout. Promising but not conclusive at this size.

4. Your turn

Your turncheckout-test.csv
How it works

Principle: an A/B readout is a comparison of two means with an honest error bar. For conversion:

Δˆ=pˆB−pˆA,SE=pˆA(1−pˆA)nA+pˆB(1−pˆB)nB,95% CI=Δˆ±1.96SE

The p-value uses the pooled rate, because under “no difference” both groups share one rate. erf turns a z-score into a normal tail area, so no extra library is needed.

Revenue per visitor is very skewed (most visitors spend 0). The mean is still the right number to compare, but use a bootstrap or Welch’s t-test for its interval (Statistics Lessons 4.5 and 6.3).

5. Practice

  1. Conversion by day of the test: is B ahead every day?
  2. Why compare revenue per visitor, not revenue per buyer?
  3. Add a 95 % CI for the conversion of group A alone.
Answers
  1. ab.pivot_table(index='day', columns='group', values='converted', aggfunc='mean'); expect it to wobble day to day.
  2. Per buyer ignores how many people bought; B could make fewer, bigger orders and look better while earning less.
  3. p ± 1.96 * sqrt(p * (1 - p) / n) with A’s p and n.

Try it on your own data: bayesian A/B test and sample size check a result or plan the next test.

New words: readout, sample ratio, lift, pooled rate, guardrail metric, segment.

  • Check the sample ratio before any result.
  • One groupby gives users, conversions and rate per group.
  • Report the lift in points with a confidence interval.
  • Check guardrail metrics and pre-planned segments only.

Check yourself

1What should you check before looking at any result?
2Why is groupby('group')['converted'].mean() the conversion rate?
3You split by 6 segments and one shows p = 0.03. What do you conclude?

Sources

  • Kohavi, R., Tang, D. and Xu, Y. (2020), Trustworthy Online Controlled Experiments, Cambridge University Press
  • McKinney, W. (2022), Python for Data Analysis, 3rd edition, O'Reilly (open access) https://wesmckinney.com/book/

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. All datasets are made up. Python runs in your browser (the first run downloads it once, about 20 MB); nothing you type is sent anywhere.