Module 6: Hypothesis tests · Lesson 6.6 · about 25 min
ANOVA: comparing three or more groups
One-way analysis of variance compares several group means at once by splitting total variation into between-group and within-group parts. The F ratio, the ANOVA table, follow-up comparisons and effect size η².
Module 6 · about 25 minutes · school maths. You’ll be able to run and read a one-way ANOVA, explain the F ratio, and follow up with corrected pairwise comparisons.
In everyday terms: three classes take the same test. If the class averages differ by a few marks but pupils within each class range from 20 to 90, the class makes little difference. If every class is tightly bunched and the averages are far apart, the class matters.
1. One question for many groups
Three store layouts, four ad creatives, five regions: are their averages really different? Pairwise t-tests multiply the chances of a false alarm (Lesson 6.8). Analysis of variance (ANOVA) asks one question first: is the spread of the group means bigger than the spread within groups would explain?
Pull the layouts apart and F grows; add noise within groups and F shrinks. The same gap between means can be convincing or meaningless depending on the noise.
2. Splitting the variation
Every value’s distance from the grand mean splits into two parts: how far its group’s mean is from the grand mean (between), and how far it is from its own group’s mean (within).
Dividing each by its degrees of freedom gives mean squares, and their ratio is F:
with groups and values in total. If the null (all means equal) is true, F is around 1. Large F → small p.
3. The ANOVA table
| Source | SS | df | MS | F | p |
|---|---|---|---|---|---|
| Between groups | k − 1 | from F(k − 1, N − k) | |||
| Within groups | N − k | ||||
| Total | N − 1 |
Effect size : the share of all variation explained by the groups.
4. After a significant F
- Post-hoc tests: Tukey’s HSD compares every pair with the right correction; or pairwise Welch t-tests with a Holm correction (Lesson 6.8).
- Planned contrasts: if you decided in advance to compare each new layout with the current one, test only those.
Assumptions: independent observations; roughly normal within groups (less important with larger groups); similar spreads. With very different spreads use Welch’s ANOVA; with skewed small samples, the Kruskal-Wallis rank test (Lesson 6.7).
- SS between
- How far each group mean is from the grand mean, weighted by group size.
- SS within
- How far values are from their own group mean.
- F = MS between ÷ MS within
- Signal (differences between groups) over noise (scatter inside groups), each divided by its df.
- Physical meaning
- All the variation in the data is split into two piles: the part explained by which group a store is in, and the part left over. F asks whether the first pile is bigger than chance would make it.
“A significant F tells you which groups differ.” It says at least one mean differs. Follow up with planned comparisons or Tukey’s test to find which, and correct for the number of comparisons (Lesson 6.8).
Watch it happen, one step at a time:
Maths and principle
Principle: if all groups share one mean, the group means differ only by sampling noise, so the between-group mean square estimates the same variance as the within-group one, and F ≈ 1.
In code: Excel Data Analysis ToolPak → “ANOVA: Single Factor”; Python scipy.stats.f_oneway(a, b, c); statsmodels pairwise_tukeyhsd for Tukey.
Two groups: ANOVA with k = 2 gives F = t² from the pooled t-test.
5. Worked example
Made-up numbers (also checked in tests/stats.mjs). Weekly units sold under three shelf placements, 6 stores each:
| Eye level | 6, 8, 4, 5, 3, 4 | mean 5 |
|---|---|---|
| Middle | 8, 12, 9, 11, 6, 8 | mean 9 |
| End cap | 13, 9, 11, 8, 7, 12 | mean 10 |
- Grand mean = (30 + 54 + 60) ÷ 18 = 8.
- SS between = 6(5 − 8)² + 6(9 − 8)² + 6(10 − 8)² = 54 + 6 + 24 = 84.
- SS within = 16 + 24 + 28 = 68 (squared gaps from each group’s own mean).
- MS between = 84 ÷ 2 = 42; MS within = 68 ÷ 15 ≈ 4.53.
- F = 42 ÷ 4.53 ≈ 9.26 with (2, 15) df; p ≈ 0.0024. η² = 84 ÷ 152 ≈ 0.55.
At least one placement differs, and placement explains about half the variation. Follow-ups would show “eye level” is the odd one out (lower, surprisingly; worth checking which products were on test).
6. Practice
- 4 groups, 40 values in total. What are the two df for F?
- SS between = 30, SS within = 270. Find η².
- F = 0.6. What would you conclude?
Answers
- 3 and 36.
- 30 ÷ 300 = 0.10.
- No evidence the group means differ; the means are even closer than noise would typically make them.
Try it on your own data: the yield trial ANOVA tool runs one- and two-way ANOVA with post-hoc comparisons, and the regression workbench handles groups as categorical predictors.
New words: ANOVA, sum of squares, mean square, F ratio, η² (eta squared), post-hoc test, Tukey’s HSD, Kruskal-Wallis.
- ANOVA compares 3 or more means in one test, avoiding many t-tests.
- F = (between-group variance) ÷ (within-group variance).
- A large F means group means differ more than scatter explains.
- Follow a significant F with corrected pairwise comparisons.
Check yourself
Sources
- OpenIntro Statistics, 4th edition (Diez, Cetinkaya-Rundel, Barr), Section 7.5 Comparing many means with ANOVA https://www.openintro.org/book/os/
- NIST/SEMATECH e-Handbook of Statistical Methods, 7.4.3 Are the means equal? https://www.itl.nist.gov/div898/handbook/prc/section4/prc43.htm
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.