Module 6: Hypothesis tests · Lesson 6.2 · about 25 min
P-values: what they are and what they are not
A p-value is the chance of data at least this extreme if nothing were going on. What it measures, five things it does not mean, why it jumps around between repeats, and how to report it with the effect size.
Module 6 · about 25 minutes · school maths. You’ll be able to define a p-value correctly, avoid the five common misreadings, and report it alongside an effect size and interval.
In everyday terms: you find your friend has won the lottery. “How likely is a win if they played fairly?” is tiny. That does not make cheating likely, because millions play fairly and someone always wins. The p-value only answers the first question.
1. The definition
The p-value is the probability, if the null hypothesis were true, of getting a result at least as extreme as the one you observed.
Small p: the data would be surprising if nothing were going on. Large p: the data is the kind of thing chance produces often.
In practice: compute the test statistic (Lesson 6.1), then the area beyond it in the null distribution, in both tails for a two-sided test.
2. Five things a p-value is not
- Not the probability that the null is true. p = 0.03 doesn’t mean “3 % chance of no effect”. That needs Bayes (Lesson 2.2) and a prior.
- Not the probability the result is due to chance. Same mistake, different words.
- Not the size or importance of the effect. A tiny effect with huge data gives a tiny p (Lesson 6.9).
- Not proof of no effect when large. p = 0.4 means “can’t tell”, often because the sample was too small.
- Not a cliff at 0.05. 0.049 and 0.051 carry the same evidence. The American Statistical Association’s 2016 statement on p-values makes these same points.
3. P-values dance
Run the same experiment 100 times with a real effect of half a standard deviation and 30 people per group. About half the runs reach p < 0.05; the rest don’t, though the effect is identical. A single p-value is a noisy measurement. Set the effect to 0: p-values spread evenly between 0 and 1, and about 5 % fall below 0.05. That’s the false alarm rate the threshold promises, not a bug.
4. How to report
Report the effect, its interval and the p-value, in that order of importance:
“The new page raised conversion from 4.0 % to 4.8 % (difference 0.8 points, 95 % CI 0.0 to 1.6; p = 0.051, n = 5,000 per group).”
That sentence lets a reader judge both how big and how certain. Compare “the result was not significant”, which throws away nearly everything.
Write small p-values as “p < 0.001” rather than “p = 0.000”.
- p
- P(data this extreme or more | H₀ true).
- Small p
- The data sit poorly with H₀.
- p ≥ α
- The data are compatible with H₀; not proof of it.
- Physical meaning
- The p-value is a measure of how surprised a believer in "no effect" should be. It says nothing on its own about how big or how important the effect is; that is the interval's job.
“p = 0.03 means a 3 % chance the result is a fluke.” It means: if there were no effect, data this extreme would turn up 3 % of the time. The chance that the effect is real also depends on how plausible it was beforehand (Lesson 2.2).
Watch it happen, one step at a time:
Maths and principle
Principle: the p-value measures how compatible the data is with the null model. It is a tail area of the null distribution of the test statistic.
Equations: two-sided z-test ; t-test .
| z | 1.645 | 1.96 | 2.576 | 3.29 |
|---|---|---|---|---|
| two-sided p | 0.10 | 0.05 | 0.01 | 0.001 |
Under the null, p is uniform: any value between 0 and 1 is equally likely, which is why exactly 5 % of null results fall below 0.05.
5. Worked example
Made-up numbers (from Lesson 4.4). A: 200 of 5,000 converted (4.0 %). B: 240 of 5,000 (4.8 %).
- Pooled rate under : 440 ÷ 10,000 = 4.4 %.
- SE under : √(0.044 × 0.956 × (1/5,000 + 1/5,000)) ≈ 0.0041.
- z = 0.008 ÷ 0.0041 ≈ 1.95. Two-sided p ≈ 0.051.
So: “just short of 0.05”. The honest reading: there’s suggestive evidence B is better, by somewhere between nothing and 1.6 points. If B is cheap to ship and has no downside, ship it and keep measuring; if it’s costly, run the test longer. Neither reading depends on whether p fell on one side of 0.05.
6. Practice
- Rewrite correctly: “p = 0.02, so there’s a 98 % chance the campaign worked.”
- A test with 40 customers gives p = 0.35. A colleague says, “So the change has no effect.” What do you reply?
- Two-sided z = 2.576. What’s p?
Answers
- “If the campaign had no effect, results this extreme would occur about 2 % of the time.” Report the size of the effect and its interval too.
- “We couldn’t detect one with 40 customers. The interval is wide and includes effects that would matter. We need more data to say there’s no meaningful effect.”
- 0.01.
Try it on your own data: the hypothesis tests tool shows the null distribution with your statistic and the p-value area shaded.
New words: p-value, tail area, statistical significance, null distribution.
- p = chance of data this extreme if H₀ were true.
- It is not the chance H₀ is true, nor the chance the result will repeat.
- p says nothing about size; always report the estimate and its interval.
- p near 0.05 is weak evidence; p-values bounce around between repeats of the same study.
Check yourself
Sources
- Wasserstein, R. L. and Lazar, N. A. (2016), The ASA Statement on p-Values: Context, Process, and Purpose, The American Statistician 70(2), 129-133 https://doi.org/10.1080/00031305.2016.1154108
- OpenIntro Statistics, 4th edition (Diez, Cetinkaya-Rundel, Barr), Section 5.3 https://www.openintro.org/book/os/
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.