All lessons

Module 6: Hypothesis tests · Lesson 6.9 · about 20 min

Effect size: significant is not the same as important

A p-value says whether an effect is distinguishable from noise; an effect size says how big it is. Raw differences, relative lift, Cohen's d, and the practical question every readout must answer: is it big enough to matter?

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 6 · about 20 minutes · school maths. You’ll be able to report effect sizes (absolute, relative and standardised), tell statistical from practical significance, and write readouts that lead with size.

A p-value says whether an effect stands out from noise; the effect size says how big it is, and only the size tells you whether it matters.

In everyday terms: a scale accurate to a gram can prove you gained 3 grams this week. It is certainly real and certainly unimportant. With enough data, every tiny difference becomes “significant”.

1. Two separate questions

  1. Is there an effect? (Could the result be noise?) → p-value, confidence interval.
  2. How big is it, and does it matter? → effect size, its interval, and its value to the business.

With enough data, any non-zero effect becomes “significant”. With too little, important effects don’t. So significance alone can’t tell you what to do.

Try it: more data, same tiny effect
customers per group
expected p-value
effect size d (difference ÷ SD)
95 % interval for the difference
Basket values have SD $25 (made up). The curve shows the p-value you would typically get for this true difference at each sample size (log scale, 100 to 1,000,000 per group). With a million customers, a 50-cent difference is "p < 0.0001" yet is 0.02 standard deviations: real, but probably not worth a redesign. Always report the size of the effect and its interval, not just the p-value.

Fix a 50-cent difference and slide the sample size. The p-value falls through 0.05 and keeps falling; the effect size stays at 0.02 SD the whole time.

2. Ways to state size

  • Absolute difference in natural units: +$1.80 per order, +0.6 conversion points, −2.3 minutes. Best for business decisions.
  • Relative lift: (B − A) ÷ A. +0.6 points on a 4 % base is +15 %. Popular, but say which: “+0.6 points (+15 %)”. Relative numbers on small bases sound dramatic.
  • Standardised (Cohen’s d): difference ÷ pooled SD. Unit-free, useful for comparing across metrics and for power planning. Conventions: 0.2 small, 0.5 medium, 0.8 large, though in product analytics, d = 0.05 can be worth millions.
  • For categories: risk ratio, odds ratio or Cramér’s V (Lesson 6.5). For relationships: r or R² (Module 8).
  • Number needed to treat: 1 ÷ (difference in rates). A retention email lifting renewal from 70 % to 72 % needs 50 emails per extra renewal.

3. Practical significance

Before the test, agree on the smallest effect worth acting on (from cost, risk and value). Then read the interval against it:

Interval Reading
Entirely above the minimum Real and worth it
Above zero, but includes values below the minimum Real, maybe not worth it; more data or a judgement call
Includes zero and the minimum Inconclusive; underpowered
Entirely below the minimum, around zero Probably no worthwhile effect, a useful result in itself
Raw difference
In the metric's own units: +$0.35 a week, +0.8 points.
Relative lift
Difference ÷ baseline: +0.67 %.
Cohen's d
Difference ÷ SD: size compared with how much individuals vary (0.2 small, 0.5 medium, 0.8 large).
Physical meaning
Size times reach is value. A d of 0.009 is invisible for one customer, yet across 1.5 million customers a week it can be worth hundreds of thousands. Judge importance in money or time, not in p-values.

“p = 0.001 means a big effect.” A tiny p can come from a tiny effect measured on a huge sample. Always put the estimate and its interval next to the p-value, and translate it into business value.

Watch it happen, one step at a time:

The test. 200,000 users per group. Weekly spend: A $52.40, B $52.75, SD $40.
Size. +$0.35 (+0.67 %); Cohen's d = 0.35 ÷ 40 ≈ 0.009: tiny per person.
Certainty. SE ≈ 0.126, z ≈ 2.77, p ≈ 0.006; CI +$0.10 to +$0.60.
Scale up. 1.5 million users × $0.10 to $0.60 = $150k to $900k a week.
Decide. Tiny per user, large in total: ship if the new search is cheap to run.
Maths and principle

Principle: effect sizes describe the data’s message; p-values describe the evidence’s strength. Report both, but lead with size.

Equations: Cohen’s d=(x‾B−x‾A)/sp, with pooled sp=(nA−1)sA2+(nB−1)sB2nA+nB−2. Relative lift (x‾B−x‾A)/x‾A. Number needed to treat 1/(pB−pA).

Why big n makes everything significant: the test statistic is roughly dn/2, which grows without limit as n grows for any d≠0.

4. Worked example

Made-up numbers. A grocery app tests a new search. 200,000 users per group. Average weekly spend: A $52.40, B $52.75; SD $40 in both.

  • Difference +$0.35 (+0.67 %); SE = 40 × √(2/200,000) ≈ 0.126; z ≈ 2.77; p ≈ 0.006. 95 % CI: +$0.10 to +$0.60.
  • Cohen’s d = 0.35 ÷ 40 ≈ 0.009: tiny.
  • Business value: 1.5 million weekly users × $0.10 to $0.60 = $150k to $900k a week in extra spend (before margin).

Tiny per user, large in total. If the new search costs little to run, ship it. The point isn’t that small effects are unimportant; it’s that you judge importance from the size, not from the p-value.

5. Practice

  1. Conversion 5.0 % → 5.5 %. State the absolute and relative lift.
  2. Means 64 and 70, pooled SD 12. Cohen’s d?
  3. A test is “not significant” with CI −1 % to +6 %. The minimum worthwhile lift is 2 %. What do you tell the team?
Answers
  1. +0.5 points; +10 %.
  2. 6 ÷ 12 = 0.5 (medium).
  3. Inconclusive: the data fits anything from a small loss to a large gain, including the 2 % you care about. The test needs more data before deciding.

Try it on your own data: the hypothesis tests tool reports Cohen’s d and confidence intervals with every test; the bayesian A/B test tool shows the chance that a lift beats your minimum worthwhile size.

New words: effect size, absolute difference, relative lift, Cohen’s d, practical significance, minimum worthwhile effect, number needed to treat.

  • Report the effect in real units, with its interval, every time.
  • Cohen’s d = difference ÷ SD compares effects across metrics.
  • Large samples make tiny effects significant; small samples hide large ones.
  • Importance = size × reach × value per unit, compared with cost.

Check yourself

1With 2 million users per group, a test finds a 0.02-point conversion lift with p < 0.001. What is the right reading?
2Two groups differ by $4 and the pooled SD is $20. What is Cohen's d?
3Which is the most useful single line in an A/B readout?

Sources

  • Sullivan, G. M. and Feinn, R. (2012), Using Effect Size - or Why the P Value Is Not Enough, Journal of Graduate Medical Education 4(3), 279-282 https://doi.org/10.4300/JGME-D-12-00156.1
  • Cohen, J. (1988), Statistical Power Analysis for the Behavioral Sciences, 2nd edition, Routledge

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.