Module 3: Cleaning data · Lesson 3.2 · about 15 min
Duplicates
Find exact and partial duplicate rows, decide which copy to keep, and check that a key column really is unique before you join or count.
Module 3 · about 15 minutes · uses Lesson 2.3. You’ll be able to find duplicates, remove the right copies and prove a key is unique.
In everyday terms: a guest list where someone signed in twice. Count heads from the list and you order an extra meal; the fix is to know which column (the name, the ticket number) should be unique.
1. Exact duplicates
keep=False marks every copy, which is handy for looking at them side by side.
2. Duplicates on some columns
Rows can differ in an unimportant column but still be the same thing. Pass subset:
Sort first, so the copy you want (the latest, the largest) is in the position keep picks.
3. Is the key unique?
order_id repeats because each row is an order line. That’s correct for this table, but it means “number of orders” is orders['order_id'].nunique(), not len(orders).
- duplicated()
- True for each row that repeats an earlier one.
- duplicated(subset=[...])
- Repeats only on chosen columns.
- drop_duplicates(keep=...)
- Remove repeats, keeping the first or last.
- is_unique
- True if a column has no repeats: the test for a key.
- Physical meaning
- A duplicate row makes one real event count twice: revenue, customers and averages all drift. A repeated key in a lookup table also multiplies rows when you join (Lesson 5.3).
“drop_duplicates() is always safe.” Two genuine orders of the same product by the same customer on the same day look identical if order_id is missing. Decide which columns define “the same thing” first.
Watch it happen, one step at a time:
| order_id | product | qty |
|---|---|---|
| 50001 | P101 | 2 |
| 50002 | P105 | 1 |
| 50001 | P101 | 2 |
| 50003 | P101 | 2 |
df.duplicated().sum() → 1 (row 2 repeats row 0).df['order_id'].is_unique → False.df.drop_duplicates() → 3 rows.4. Your turn
How it works
Principle: duplicated compares each row with all earlier rows (by hashing their values), so it’s fast even on millions of rows. Two rows count as the same only if every compared value matches exactly: 'Maple Ridge' and ' maple ridge' are different until you clean the text (next lessons). Clean first, then de-duplicate.
5. Practice
- Mark every copy of a duplicated email, not just the later ones.
- Keep each product’s cheapest offer from a table of offers.
- Why might
drop_duplicates()find nothing in a messy file that clearly has repeats?
Answers
df.duplicated(subset=['email'], keep=False).offers.sort_values('price').drop_duplicates('product_id', keep='first').- The copies differ slightly (spaces, capitals, date formats, a timestamp column); clean those or use
subset.
New words: duplicate, drop_duplicates, subset, keep, is_unique, key.
- duplicated() flags repeats; .sum() counts them.
- Use subset= to check the columns that define “the same thing”.
- Check keys with is_unique before joining.
- Look at duplicates before deleting them.
Check yourself
Sources
- pandas documentation, User Guide: Duplicate labels and drop_duplicates https://pandas.pydata.org/docs/user_guide/duplicates.html
- McKinney, W. (2022), Python for Data Analysis, 3rd edition, O'Reilly (open access) https://wesmckinney.com/book/
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. All datasets are made up. Python runs in your browser (the first run downloads it once, about 20 MB); nothing you type is sent anywhere.