Module 8: What to learn next · Lesson 8.3 · about 20 min
Mini projects for your portfolio
Seven mini projects and a capstone on synthetic or public data and free tools, with a README checklist and how to plant faults and answers in your data.
Module 8 · about 20 minutes to read; the projects take 45 minutes to 10 hours each. You’ll be able to pick the right project for each module, write a README a hiring manager can read in two minutes, and generate synthetic data with planted faults and a planted answer you can check.
1. The projects
Each module ends with a mini project that uses what you just read. Together they build towards the capstone: a small but complete retail media data platform. All data is synthetic or public: never use an employer’s data, even masked.
| # | Project | Module | What it shows | Time |
|---|---|---|---|---|
| 1 | Platform one-pager | The big picture | You can map tools to jobs and justify them | 45 min |
| 2 | The nightly sales job | Spark | PySpark cleaning, joins, skew, partitioned Parquet, reading the plan | 2 to 3 h |
| 3 | The cloud nightly pipeline | Google Cloud | An Airflow DAG, sensors, safe backfills, BigQuery marts, a cost estimate | 3 to 4 h |
| 4 | Three stores, one dataset | Databases | Real-time OLAP, key-value design and search, compared honestly | 3 h |
| 5 | The live store dashboard | Kafka | Schemas, dead letters, windows, watermarks, restarts without double counting | 3 to 4 h |
| 6 | An ML-ready feature pipeline | AI and ML | Point-in-time features, a leakage test, an online lookup | 3 to 4 h |
| 7 | Capstone: retail media platform | Retail media | A design document, a streaming and batch slice, attribution against measured lift | 6 to 10 h |
| 8 | Decode a posting | What next | Your evidence mapped to a real job, honest experience statements | 1 h |
New to pipelines altogether? The seven “proof” projects in Pipeline to Dashboard are a gentler first portfolio, and each one feeds the projects here.
2. What every README needs
- The business question in one sentence, and who would ask it.
- A diagram of the flow, with each tool and why it was chosen.
- How to run it in under five commands.
- Tests and checks: row counts, duplicates, freshness, a reconciliation against the source.
- Results: the numbers or screenshots, and what they mean.
- Cost: what it would cost per month at a realistic scale, with your working (and “check the official price list” next to any price).
- What I would do next with a real team and real data.
3. Plant the faults, plant the answer
Real data is messy, and a project on perfectly clean data proves little. So plant the mess yourself, on purpose, where you know exactly how much there is:
- Faults: duplicates, late events, nulls in keys, one hot key, refunds as negative amounts, a schema change halfway through.
- The answer: for the capstone, set a true lift (say +15%) and check that your measurement recovers it within its confidence interval.
- A seed: fix the random seed so others can reproduce your exact data and results.
Think of a teacher writing an exam with an answer key. You plant the answer in the data first, then build the pipeline that has to find it. If the pipeline misses, you know there is a bug, and you know before an interviewer finds it.
fct_sales and the measurement table.A seeded generator in Python takes a few lines:
import numpy as np, pandas as pd
rng = np.random.default_rng(42) # fixed seed: same data every run
n, holdout, base, lift = 50_000, 0.10, 0.04, 0.15
users = pd.DataFrame({"user_id": np.arange(n)})
users["group"] = np.where(rng.random(n) < holdout, "holdout", "exposed")
p = np.where(users["group"] == "exposed", base * (1 + lift), base)
users["bought"] = rng.random(n) < p
print(users.groupby("group")["bought"].agg(["size", "mean"]))
Check: the exposed buying rate comes out near 4.6% and the holdout near 4.0%, and rerunning gives exactly the same table.
- Synthetic data
- Data you generate with code, so it contains no real person and you know its true patterns.
- Planted fault
- A known mess (duplicates, late events, a bot) added on purpose to test the pipeline.
- Planted answer
- A true effect you set, such as +15% lift, that the analysis must recover.
- Seed
- The starting number for the random generator, which makes the "random" data repeatable.
- Physical meaning
- When a hiring manager opens your repository, runs one command and sees a check that says "planted 270 extra buyers, measured 255, inside the 95% interval: pass", they know you test your own work. That single line says more than a page of claimed skills.
“A portfolio project should use real company data to be impressive.” Real company data is not yours to publish, even masked, and using it can end an application. Synthetic data with planted faults is more impressive, because it shows you know what can go wrong and can prove your pipeline catches it.
4. Try it
Pick the project for the module you finished most recently. Before writing any pipeline code, write the README’s first three sections (question, diagram, how to run) and a list of three faults you will plant.
Check: a friend who does not work in data can read the question and the diagram and say, in their own words, what the project does.
New words: synthetic data, planted fault, planted answer, random seed, reproducible.
- One project per module, building towards a retail media capstone; all data synthetic or public.
- Every README: the question, a diagram, how to run, tests and checks, results, cost, next steps.
- Plant faults (duplicates, late events, hot keys, schema changes) and a known answer, and check the pipeline finds both.
- Seed your generator so anyone can reproduce your data and results exactly.
Check yourself
Sources
- Reis, J. and Housley, M. (2022), Fundamentals of Data Engineering, O'Reilly
- NumPy documentation: Random Generator (numpy.org/doc)
- Kohavi, R., Tang, D. and Xu, Y. (2020), Trustworthy Online Controlled Experiments, Cambridge University Press
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. All company data in the examples is synthetic. Tool versions and prices change: check the official docs before you rely on them.