All lessons

Module 6: Hypothesis tests · Lesson 6.3 · about 30 min

t-tests: one sample, two samples and paired

The workhorse tests for means: one-sample (against a target), two-sample (two groups, Welch by default) and paired (the same units before and after). Formulas, assumptions, worked examples and how to run each in Excel and Python.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 6 · about 30 minutes · school maths. You’ll be able to choose between one-sample, two-sample and paired t-tests, compute each by hand, and run them in a spreadsheet or Python.

A t-test asks whether a difference in means is big compared with how much means wobble: difference ÷ standard error.

In everyday terms: two runners’ average lap times differ by 2 seconds. If their laps usually vary by 10 seconds, 2 seconds could be luck; if they vary by 0.5 seconds, it is a real gap. The t-test measures the gap in units of that wobble.

1. Three questions, three tests

Question Test Example
Is the mean different from a target? One-sample t Is average handling time different from the 6-minute target?
Do two separate groups differ? Two-sample t (Welch) Do mobile and desktop users spend differently?
Did the same units change? Paired t Did the same 20 stores sell more after the refit?

All three are signal ÷ noise: a difference in means divided by its standard error.

Try it: signal against noise
observed difference
standard error
t = difference ÷ SE
p-value (Welch)
Group A gets the old page, group B the new one (made-up basket values). The t statistic is the gap between the sample means divided by its standard error, so it grows with a bigger gap, less noise or more people. The bars are each group's 95 % interval for its mean. Overlapping bars do not by themselves mean "no difference"; the test on the difference is what counts.

2. One-sample t

t=x‾−μ0s/n,df=n−1

Lesson 6.1’s pack weights were an example.

3. Two-sample t (Welch)

t=x‾B−x‾AsA2nA+sB2nB

The degrees of freedom come from the Welch-Satterthwaite formula (software does it; it lands between the smaller n−1 and nA+nB−2). Use Welch by default. The older pooled t-test assumes equal variances; it’s fine when they really are equal and sample sizes match, risky otherwise.

4. Paired t

When each unit is measured twice (before/after, two treatments on the same customer), compute each unit’s difference di and run a one-sample t on the differences against 0:

t=d‾sd/n,df=n−1

Pairing is powerful because big stores and small stores both cancel out: only the change is left.

5. Assumptions

  • Independent units (customers, stores), not repeated rows of the same customer.
  • Roughly normal means: by the CLT this holds for moderate n; for small n, the data shouldn’t be wildly skewed. With heavy skew and small n, use a rank test (Lesson 6.7) or the bootstrap.
  • For paired tests, the differences are what need to behave.
One-sample t
(x̄ − target) ÷ (s/√n): is the mean different from a fixed number?
Welch two-sample t
(x̄₂ − x̄₁) ÷ √(s₁²/n₁ + s₂²/n₂): do two separate groups differ?
Paired t
Work out each unit's own difference, then a one-sample t on those differences.
Physical meaning
Pairing removes the differences between units (big stores, small stores) so only the change is left. That is why a before-and-after on the same stores can find effects that two separate groups would hide in the noise.

“Use the equal-variance t-test by default.” Groups rarely have the same spread. Welch’s version does not assume it and loses almost nothing when spreads are equal, so it is the safe default.

Watch it happen, one step at a time:

Two groups. Mobile baskets $38.50 (SD $22, n 400); desktop $42.10 (SD $25, n 350).
Each wobble squared. 22²/400 = 1.21; 25²/350 ≈ 1.786.
SE of the difference. √(1.21 + 1.786) ≈ 1.73.
t. (42.10 − 38.50) ÷ 1.73 ≈ 2.08.
Read it. p ≈ 0.038; 95 % CI $0.21 to $6.99. Desktop baskets are higher, by somewhere between a little and a lot.
Maths and principle

Principle: every t-test divides an observed difference by its standard error and compares that ratio with the t distribution.

In code:

  • Excel: T.TEST(range1, range2, 2, 3) gives the two-sided p for Welch (type 3), type 2 for pooled, type 1 for paired.
  • Python: scipy.stats.ttest_1samp(x, 6), ttest_ind(b, a, equal_var=False), ttest_rel(after, before).

Check: groups A = 6, 8, 4, 5, 3, 4 and B = 8, 12, 9, 11, 6, 8 give Welch t = 3.46 (B − A), df ≈ 9.6, p ≈ 0.006. tests/stats.mjs checks this against scipy.

6. Worked examples

Made-up numbers.

Two-sample. Average basket on mobile: x̄ = $38.50, s = $22, n = 400. Desktop: x̄ = $42.10, s = $25, n = 350.

  • SE = √(22²/400 + 25²/350) = √(1.21 + 1.786) = √2.996 ≈ 1.73.
  • t = (42.10 − 38.50) ÷ 1.73 ≈ 2.08; df ≈ 702; two-sided p ≈ 0.038.
  • 95 % CI for the difference: 3.60 ± 1.96 × 1.73 = $0.21 to $6.99.

Desktop baskets are higher by somewhere between almost nothing and $7.

Paired. 8 stores’ weekly sales ($k) before and after a refit:

Store 1 2 3 4 5 6 7 8
Before 52 87 31 120 64 45 98 73
After 55 90 33 124 63 49 101 76
Change 3 3 2 4 −1 4 3 3
  • Mean change d‾ = 21 ÷ 8 = 2.625; s_d ≈ 1.60; SE = 1.60 ÷ √8 ≈ 0.57.
  • t = 2.625 ÷ 0.57 ≈ 4.65, df = 7, p ≈ 0.002.

A two-sample test on the same numbers would find nothing (store sizes range from 31 to 124, swamping a 3k change). Pairing is what reveals it.

7. Practice

  1. Which test: (a) same customers rate two menus; (b) customers in Ontario vs Quebec; (c) is the average call longer than the 5-minute target?
  2. One-sample: n = 16, x̄ = 52, s = 8, target 50. Find t.
  3. Why might a paired analysis be wrong if the “after” period includes a holiday?
Answers
  1. (a) Paired. (b) Two-sample (Welch). (c) One-sample, one-sided if only “longer” matters and that was decided in advance.
  2. SE = 8 ÷ 4 = 2; t = 2 ÷ 2 = 1.0 (df 15, two-sided p ≈ 0.33).
  3. The holiday changes every store at once, so the “change” mixes the refit with the season. Compare with stores that didn’t get the refit over the same weeks (a control group).

Try it on your own data: the hypothesis tests tool runs all three t-tests from pasted numbers or a CSV, and the regression workbench handles more than two groups or extra variables.

New words: one-sample t-test, two-sample t-test, Welch’s t-test, pooled t-test, paired t-test, degrees of freedom.

  • t = difference ÷ SE of the difference.
  • One-sample: against a target. Welch: two independent groups. Paired: same units twice.
  • Use Welch by default; pair whenever the same units are measured twice.
  • Large samples make t close to z; small samples need the t table.

Check yourself

120 stores are measured before and after a new layout. Which test fits?
2Why is Welch's t-test a good default for two groups?
3In t = (x̄_B − x̄_A) ÷ SE, what makes t bigger?

Sources

  • OpenIntro Statistics, 4th edition (Diez, Cetinkaya-Rundel, Barr), Chapter 7 Inference for numerical data https://www.openintro.org/book/os/
  • Delacre, M., Lakens, D. and Leys, C. (2017), Why Psychologists Should by Default Use Welch's t-test Instead of Student's t-test, International Review of Social Psychology 30(1) https://doi.org/10.5334/irsp.82

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.