Module 3: Cleaning data · Lesson 3.6 · about 20 min
Checking data: ranges, rules and outliers
Write checks that prove a table is right before you analyse it: allowed ranges, allowed values, keys that must match another table, totals that must reconcile, and outliers worth a second look.
Module 3 · about 20 minutes · uses Lessons 2.3 and 3.1. You’ll be able to write the checks that catch bad data before it reaches a report.
In everyday terms: a pilot’s pre-flight checklist. Each item is quick and boring, and the one time a check fails it saves the flight.
1. Rules as masks
Every rule is a filter for the rows that break it:
A table like this, run every time a new file arrives, is a simple data quality report.
2. Assert the rules that must hold
assert condition, message stops with the message when the condition is False. Put them after cleaning:
The last rule fails on purpose: it shows what a failed check looks like. Change 50 to 100 and run again.
3. Outliers: flag, don’t delete
These are real sales of expensive items, not errors. An outlier rule finds rows to look at; whether to cap, keep or remove them is a decision you state (Statistics Lesson 1.4).
4. Reconcile totals
Compare your total with a number from the source system (a finance report, the till summary). If revenue from your cleaned table differs, find out why before going further: dropped rows, double-counted lines after a join, or a filter that was too strict.
- Mask rule
- df['qty'] <= 0 flags rows that break "quantity is positive".
- assert cond, msg
- Stop with a message if a rule that must hold does not.
- Outlier flag
- Mark unusual values for review rather than deleting them.
- Reconcile
- Check totals against a trusted source (the finance figure).
- Physical meaning
- A validation rule is a promise the data makes. When a promise breaks, everything built on it is suspect, so the code should shout, not carry on quietly.
“The data came from our own system, so it is correct.” Exports change, joins duplicate, test orders slip in. Rules catch the change the day it happens, not the day a manager spots a strange number.
Watch it happen, one step at a time:
bad = orders[(orders.qty <= 0) | ~orders.discount.between(0, 0.5)] → 0 rows.assert orders.store_id.isin(stores.store_id).all(), 'unknown store'5. Your turn
How it works
Principle: analysis can only be as right as its inputs. Validation turns what you believe about the data (“every order has a customer”) into checks that run every time. The cheapest time to find a problem is before anyone has seen a chart.
Kinds of rule: type (dates are dates), range (0 ≤ discount ≤ 0.5), allowed values (store in the store list), uniqueness (one row per key), reference (every key matches a lookup table), and reconciliation (totals match the source).
6. Practice
- Write a rule for “email must contain @”.
- What does
orders['customer_id'].isin(customers['customer_id']).all()check? - Why flag outliers rather than delete them?
Answers
df['email'].str.contains('@', na=False); rows where it’s False break the rule.- That every order’s customer exists in the customer table (no orphan orders).
- They may be real and important (big customers, expensive items); deleting them changes totals and hides information. Decide case by case and document it.
Try it on your own data: CSV profiler shows ranges, missing counts and odd values per column.
New words: validation, data quality report, assert, orphan key, outlier flag, reconciliation.
- Express rules as masks; count and inspect violations.
- assert stops the run when a must-hold rule breaks.
- Flag outliers for review; do not delete real values.
- Reconcile key totals with a trusted source.
Check yourself
Sources
- pandas documentation, User Guide: Indexing and selecting data (boolean indexing) https://pandas.pydata.org/docs/user_guide/indexing.html
- Breck, E. et al. (2019), Data Validation for Machine Learning, MLSys 2019 https://mlsys.org/Conferences/2019/doc/2019/167.pdf
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. All datasets are made up. Python runs in your browser (the first run downloads it once, about 20 MB); nothing you type is sent anywhere.