Module 8: What to learn next · Lesson 8.2 · about 25 min
Job postings, interviews and honest stories
Decode a big-data job posting, answer concept and design questions with a clear structure, size a system quickly, and describe your experience honestly.
Module 8 · about 25 minutes · read first: 8.1. You’ll be able to turn a posting’s requirements into a study and portfolio plan, answer common concept and design questions with a clear structure, do the back-of-envelope sizing interviewers ask for, and describe partial experience truthfully.
1. Decode the posting
Here is the kind of requirement list you will see for a big-data role at a large retailer’s media business (paraphrased and combined from several postings):
- Builds and tunes batch jobs in Spark or PySpark.
- Works with Google Cloud data services: Dataproc, BigQuery, Cloud Storage and Airflow.
- Knows analytical or NoSQL stores such as Druid, Bigtable or Elasticsearch.
- Has used streaming technology such as Kafka.
- Has built data platforms that support machine learning.
- Nice to have: retail media, advertising technology or measurement.
| Requirement | What they really want | Your evidence after this course |
|---|---|---|
| Spark / PySpark | Can write, debug and tune batch jobs | Module 2 mini project, with the plan and Spark UI screenshots |
| Google Cloud: Dataproc, BigQuery, Cloud Storage, Airflow | Can build a scheduled cloud pipeline | Module 3 mini project, with a cost estimate |
| Druid, Bigtable, Elasticsearch | Knows when to use which; has touched one | Module 4 mini project and its decision table |
| Kafka | Understands topics, partitions, consumer groups, guarantees | Module 5 mini project, with late data and restarts |
| Platforms for ML | Builds time-correct features and reliable data for models | Module 6 mini project |
| Retail media, adtech, measurement | Speaks the domain: attribution, incrementality, pacing | Module 7 and the capstone |
Read “experience with” as “has done it”, “familiarity” as “understands it and has tried it”, and “nice to have” as optional. Apply when you match most of the core and can show learning on the rest.
2. Concept questions: what, why, example
Answer every concept question in three parts: what it is, why it matters, and a short example.
- What is a shuffle and why is it expensive? Data moves across the network between stages for wide operations like group-by and join; it costs disk, network and memory. Example: a group-by on store, made cheaper by pre-aggregating or broadcasting a small table.
- How does Kafka keep order? Only within a partition; key by what must stay in order. Example: keying by order ID.
- Partitioning or clustering in BigQuery? Partitioning skips whole date or range slices; clustering sorts within partitions so selective filters read less. Example with bytes scanned before and after.
- When would you not stream? When nobody acts faster than daily; streaming costs money and on-call time.
- Attribution or incrementality? Credit against cause; a holdout design; iROAS against ROAS.
- What is point-in-time correctness? Features use only data known at prediction time. Example: a “spend this month” feature that leaked days after the coupon was sent.
3. Design questions: one structure, every time
A pilot does not improvise the take-off; they run the same checklist every time, which frees their attention for the unusual. A fixed answer structure does the same for you:
- Clarify: users, delay, volume, accuracy, privacy.
- Sketch: sources, ingest, store, process, serve, use.
- Choose: each store and engine with a reason, and an alternative you rejected.
- Break it: late data, duplicates, a failed job, a schema change, a hot key. How would you notice, and how would you recover?
- Cost and running it: what drives the bill, what you monitor, who is on call.
user#campaign#date, because lookups are by exact key in milliseconds. Rejected: BigQuery, because it is built for scans, not 20,000 single-row reads a second.Practice prompts: “design live campaign dashboards for 5,000 advertisers”, “design a nightly pipeline for 2 billion till lines”, “how would you measure whether in-store screens increase sales?”
Interviewers often ask for a quick size estimate. Doing it out loud, with round numbers, shows you understand how storage and pruning work:
- What, why, example
- A three-part answer to any concept question.
- Clarify, sketch, choose, break, cost
- A five-step answer to any design question.
- Bridge and proof
- Pair each gap with related work you did do and a link they can open.
- Back-of-envelope sizing
- A quick estimate with round numbers: rows, bytes, compression, how much a query reads.
- Physical meaning
- Two candidates with the same skills: one jumps to "I would use Kafka and Druid", the other asks three questions, sketches the flow, names what could break and estimates the cost. The second one gets the offer, because that is exactly how the job is done on a real Monday morning.
4. Telling the truth about experience
Interviewers forgive gaps; they do not forgive inflated claims that fall apart in a technical round. Use specific, scoped language:
- “In a personal project I built a PySpark job over 2 million synthetic till lines, found skew from one online store in the Spark UI, and fixed it with salting, which cut the stage from 6 minutes to 50 seconds on my laptop.”
- “I have not run Kafka in production. I built a local pipeline with three partitions, a consumer group, a dead-letter topic and a Structured Streaming job that survives restarts without double counting.”
- “My production experience is in SQL warehouses and BI. The measurement maths in retail media is the same discipline I used when reconciling reports with finance.”
Pair each gap with a bridge (related work you did do) and a proof (a repository or write-up they can open).
“Everyone exaggerates, so I have to as well.” A technical interviewer’s job is to ask one more question than you prepared for. A scoped, honest answer with a link to working code is stronger than a big claim, and it protects you in the job itself.
5. Portfolio checklist
- One pipeline repository per mini project, each with a README: the problem, a diagram, how to run it, tests, results, cost, and what you would do next (Lesson 8.3 has the full checklist).
- The capstone design document and working slice, pinned on your GitHub profile.
- A short blog post per project explaining one hard thing you learned (skew, watermarks, iROAS).
- All data synthetic or public, never an employer’s.
6. You have finished the course
You can now draw a modern data platform from memory; explain Spark, BigQuery, Dataproc, Cloud Storage, Airflow, Druid, Bigtable, Elasticsearch and Kafka in plain words; say who uses each and why; build small working versions of each; and explain how a retail media network proves its ads work. That is a solid foundation. The next step is depth: pick the two tools your target roles mention most and build one more serious project on each.
7. Final task (about 1 hour)
Take one real posting you would like to apply for. Fill in the “decode” table above with your evidence, linking to your repositories. Write three honest experience statements in the bridge-and-proof style. Then go to Mini projects for your portfolio and tick off what you have built.
Check: every row of your table links to something a stranger can open, and none of your statements claims production experience you do not have.
New words: requirement decoding, what-why-example, design framework, back-of-envelope sizing, bridge and proof.
- Read “experience with” as done it, “familiarity” as understood and tried, “nice to have” as optional; map each requirement to a piece of your own evidence.
- Answer concept questions with what, why, example; design questions with clarify, sketch, choose, break, cost.
- Size systems out loud with round numbers: rows, bytes, compression, partitions and columns read.
- Describe experience in specific, scoped terms, and pair each gap with a bridge and a proof.
Check yourself
Sources
- Kleppmann, M. (2017), Designing Data-Intensive Applications, O'Reilly
- Reis, J. and Housley, M. (2022), Fundamentals of Data Engineering, O'Reilly
- Xu, A. (2020), System Design Interview: An Insider's Guide, independently published
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. All company data in the examples is synthetic. Tool versions and prices change: check the official docs before you rely on them.