All lessons

Module 9: Capstone and interviews · Lesson 9.2 · about 45 min

Statistics interview questions, answered

40 statistics questions that come up in data analyst, BI and data scientist interviews, from mean vs median to A/B test design, each with a short answer you can say out loud and a link to the lesson that explains it.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 9 · about 45 minutes · revision. Read each question, answer it out loud in under a minute, then open the answer. The lesson number tells you where to go deeper.

Good interview answers give the one-line definition, a business example and the trap, in under a minute.

In everyday terms: like explaining a recipe to a friend: what it is, when you would make it, and the one mistake that ruins it.

How to answer: give the one-line definition, an example from business data, and the trap. “The median is the middle value. For order values it’s better than the mean because a few huge orders drag the mean up. I’d report both if they differ a lot.”

Describing data

1. Mean or median: when do you use which?

Median for skewed data (income, order value, time on page) or with outliers; mean when the data is roughly symmetric or when totals matter (mean × count = total). Report both if they differ a lot. (Lesson 1.2)

2. What does the standard deviation tell you?

The typical distance of values from the mean, in the data’s own units. For roughly normal data about 68 % fall within 1 SD and 95 % within 2. (Lesson 1.3)

3. How do you find outliers?

Plot first (box plot, histogram). Common rules: beyond 1.5 × IQR from the quartiles, or |z| > 3. Then investigate: error, or a real extreme? Never delete without a reason you can state. (Lesson 1.4)

4. What is a z-score?

How many SDs a value is from the mean: (x − mean) ÷ SD. It puts different scales on one footing. (Lesson 1.5)

5. What does a right-skewed distribution look like, and an example?

A long tail to the right; mean above median. Revenue per customer, delivery times, house prices. (Lesson 1.4)

Probability

6. A test for a rare condition is 99 % accurate. A positive result: how likely is the condition?

It depends on the base rate. If 1 in 1,000 have it, most positives are false: about 9 %. Use Bayes’ rule or a table of 100,000 people. (Lesson 2.2)

7. What is conditional probability?

The chance of A given that B happened: P(A and B) ÷ P(B). Example: chance a customer churns given they contacted support. (Lesson 2.2)

8. What is expected value, and where do businesses use it?

The long-run average outcome: sum of value × probability. Pricing warranties, valuing a lead, deciding whether a promotion pays. (Lesson 2.3)

9. Which distribution for the number of support tickets per hour?

Poisson, if tickets arrive independently at a steady rate; check whether the variance is close to the mean. (Lesson 3.3)

10. Why is the normal distribution so common?

The central limit theorem: sums and averages of many independent effects tend towards normal, whatever the shape of each effect. (Lessons 3.4 and 4.3)

Sampling and estimation

11. What is the central limit theorem?

The average of a large enough random sample is roughly normally distributed around the true mean, with spread σ/√n, even if the data itself isn’t normal. It’s why confidence intervals and tests work. (Lesson 4.3)

12. Standard deviation vs standard error?

SD describes the spread of individual values; SE describes the uncertainty of an estimate such as a mean: SD ÷ √n. More data shrinks the SE, not the SD. (Lesson 4.4)

13. Name three kinds of sampling bias.

Selection (only app users answer), non-response (unhappy customers skip the survey), survivorship (only products still on sale are analysed). (Lesson 4.2)

14. What is bootstrapping?

Resampling your data with replacement many times and recomputing the statistic each time to see how much it varies. Gives intervals for medians, ratios and anything without a formula. (Lesson 4.5)

15. What does a 95 % confidence interval mean?

If we repeated the study many times, 95 % of intervals built this way would contain the true value. Not “95 % chance the true value is in this interval” (that’s a Bayesian credible interval). (Lesson 5.3)

16. How many people to survey for ±3 points at 95 %?

About 1,070 for a proportion near 50 %: n = 1.96² × 0.25 ÷ 0.03². (Lesson 5.4)

Hypothesis tests

17. Explain a p-value to a manager.

“If there were really no difference, a gap this big would show up by chance only 2 % of the time. So chance alone is an unlikely explanation.” It isn’t the chance the result is wrong. (Lesson 6.2)

18. Type I and type II errors?

Type I: a false alarm (calling a difference real when it isn’t), rate α. Type II: a miss (failing to detect a real difference), rate β; power = 1 − β. (Lesson 6.4)

19. What is statistical power and what raises it?

The chance of detecting a real effect of a given size. Raise it with more data, a bigger effect, less noise (better metrics, CUPED) or a higher α. (Lesson 6.4)

20. When do you use a t-test, and which one?

Comparing means. Welch’s two-sample t-test by default (doesn’t assume equal variances); paired t-test for before/after on the same units; one-sample against a fixed value. (Lesson 6.3)

21. When do you use a chi-square test?

For counts in categories: whether two categorical variables are related (contingency table), or whether counts fit expected shares (goodness of fit, such as an A/B split). (Lesson 6.5)

22. What does ANOVA test?

Whether several group means are all equal. A significant F says at least one differs; follow up with corrected pairwise comparisons. (Lesson 6.6)

23. When would you use a non-parametric test?

Small samples from clearly non-normal data, ranks or ratings, or heavy outliers: Mann-Whitney instead of a two-sample t-test, Wilcoxon signed-rank instead of paired. (Lesson 6.7)

24. You test 20 metrics and one is significant. Excited?

No: at α = 0.05 you expect about one false positive in 20. Correct with Holm or Benjamini-Hochberg, or treat it as a lead for a new test. (Lesson 6.8)

25. Statistical vs practical significance?

Statistical: unlikely to be chance. Practical: big enough to matter. Huge samples make trivial effects significant; always report the effect size and interval. (Lesson 6.9)

A/B testing

26. Walk me through designing an A/B test.

Hypothesis with a because; randomisation unit (usually user); eligible population and trigger; one primary metric, secondary metrics and guardrails; sample size from baseline, MDE, α and power; whole weeks; a written decision rule. Freeze it before launch. (Lesson 7.1)

27. How do you choose the primary metric?

Tied to value, sensitive enough to move during the test, hard to game. Often conversion or revenue per user, with guardrails for what must not get worse. (Lesson 7.2)

28. How long should a test run?

Until it reaches the planned sample size, rounded up to whole weeks (at least one, ideally two) so every weekday counts equally. Not “until it’s significant”. (Lesson 7.3)

29. What's wrong with stopping when p drops below 0.05?

Peeking: each look is another chance for noise to cross the line, so false wins rise from 5 % to 20 % or more. Use a fixed horizon or a sequential design. (Lesson 7.4)

30. What is a sample ratio mismatch?

When the split between groups differs from the plan more than chance allows (chi-square p < 0.001). It signals a bug (tracking, redirects, bots) and invalidates the result until fixed. (Lesson 7.6)

31. Frequentist vs Bayesian A/B testing?

Frequentist: p-value and confidence interval, fixed sample. Bayesian: chance B is better and expected loss, with a prior. Bayesian answers read more naturally; neither fixes peeking or bad data. (Lesson 7.7)

32. The test won in week one, but the effect faded. Why?

Likely a novelty effect. Check lift by day and new vs returning users; run longer or keep a holdout. (Lesson 7.8)

33. You can't randomise users (a price change in stores). What do you do?

Randomise larger units (stores, regions, weeks), use a switchback design, or a quasi-experiment such as difference-in-differences with careful comparison groups. (Lessons 7.1 and 7.8)

Relationships and time

34. Correlation vs causation, with an example.

Ice cream sales and drownings correlate because hot weather drives both. To show cause you need an experiment or a design that rules out confounders. (Lesson 8.1)

35. Pearson vs Spearman?

Pearson measures straight-line association and is sensitive to outliers; Spearman uses ranks and measures any steady rising or falling relationship. Use Spearman for skewed data. (Lesson 8.1)

36. How do you interpret a regression coefficient?

The average change in Y for one more unit of X, holding other inputs fixed, with a CI. It’s an association unless the design supports a causal reading. (Lesson 8.2)

37. What is Simpson's paradox?

A trend in every subgroup reverses when the groups are combined, because the groups are mixed differently. Example: a version wins on phones and computers but loses overall because most of its traffic is phones. (Lesson 8.3)

38. Sales dropped 8 % this week. How do you tell if it's real?

Compare with the same week last year and the same weekday last week; check the 7-day average; put it on a control chart. Check data issues (tracking, late-arriving data) before looking for business causes. (Lessons 8.4 and 8.6)

39. How would you forecast next month's demand?

Start with a seasonal naive baseline, try exponential smoothing or regression with calendar features, compare on a held-out recent period with MAE, and give a range. (Lesson 8.5)

40. What is R², and is a low R² bad?

The share of the variation in Y explained by the model. A low R² is fine when you’re estimating an effect (a slope can be clearly real) but poor for prediction of individual values. (Lesson 8.2)

Definition
One sentence, plain words.
Example
A real analyst situation: orders, conversion, stores.
Trap
The common misreading, and how you avoid it.
Physical meaning
Interviewers test whether you can use statistics to make a decision safely, not whether you remember formulas. Showing the trap shows judgement.

“I must know every formula by heart.” Interviewers care more that you know which tool fits and what can go wrong. Saying “I would check the formula, but the idea is…” is fine.

Watch it happen, one step at a time:

The question. "What is a p-value?"
Definition. The chance of data at least this extreme if there were no real effect.
Example. Our checkout test had p = 0.005: a gap this big would be rare if the forms converted equally.
Trap. It is not the chance the effect is real, and it says nothing about size, so I report the interval too.
Close. "So we shipped, because the whole interval for the lift was above zero."
Maths and principle: the five formulas worth knowing by heart
SE=sn,x‾±1.96SE,z=pˆB−pˆASE,n≈16σ2δ2,P(A|B)=P(B|A)P(A)P(B)

Standard error, a 95 % interval, a test statistic, Lehr’s rule for sample size (80 % power, α = 0.05) and Bayes’ rule cover most whiteboard questions.

Worked example: a case question

“Our sign-up rate fell from 12 % to 10 % after the redesign. What do you do?”

  1. Clarify: over what period, how many visitors, same traffic sources? Was it an A/B test or a before/after?
  2. Check the data: tracking changes in the redesign, bot traffic, a campaign that changed the visitor mix (Simpson’s paradox).
  3. Check it’s beyond noise: with 5,000 visitors a week each side, 12 % vs 10 % gives z ≈ 3.2, p ≈ 0.001, so unlikely to be chance, if the comparison is fair.
  4. Before/after isn’t an experiment: seasonality or traffic mix may explain it. Propose an A/B test of old vs new.
  5. Recommend: revert or test, depending on cost; monitor the guardrails.

Practice

  1. Explain the central limit theorem in two sentences to a non-technical manager.
  2. Explain why you shouldn’t stop a test early when it looks good.
  3. Answer: “How would you test whether a new recommendation widget increases revenue?”
Answers
  1. “If you take lots of random samples and average each one, those averages form a bell curve around the true value, even if the raw data is lopsided. That’s why we can put a margin of error on an average.”
  2. Each look gives noise another chance to look like a win; stopping on a good look turns a 5 % false alarm rate into 20 % or more.
  3. Randomise users 50/50; primary metric revenue per user (capped, or with CUPED); guardrails page speed and returns; size from historical variance; two full weeks; decide with the 95 % CI for the lift; check SRM and novelty.

Try it on your own data: practise questions 17 to 25 with hypothesis tests, and 28 with sample size.

New words: none; this lesson revises the course.

  • Answer in three parts: definition, example, trap.
  • Use business examples: orders, conversion, stores.
  • Mention uncertainty (intervals) whenever you give a number.
  • It is fine to say what you would check if unsure.

Check yourself

1An interviewer asks "what is a p-value?". Which answer is right?
2Your test result is significant but the lift is tiny. What do you say?
3How do you answer a question you are not sure about?

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.