Home / Learn / Data & AI / Data at Scale / Lesson 2.4
All lessons

Module 2: Spark and distributed processing · Lesson 2.4 · about 25 min

Spark's relatives, and when not to use Spark

Hadoop, Databricks, Flink, Beam and Dataflow, Trino, Dask, Ray, Polars and DuckDB: where each fits, how to choose, and the Module 2 mini project.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 2 · about 25 minutes · read first: 2.1 to 2.3. You’ll be able to place Spark’s relatives on one map, pick an engine for a job by size, speed and team, and build a small end-to-end PySpark batch job for your portfolio.

Spark is one engine in a family; the right choice depends on four questions: how big the data is, how fresh the answer must be, where the data already lives, and what the team knows.

1. Why this matters

“Why Spark and not X?” is one of the most common interview questions for data roles, and one of the most common design mistakes at work is using a cluster for a job a laptop could do. Knowing the family lets you choose on purpose and explain the choice.

2. The family tree

Engine What it is Best at Where you’ll see it
Hadoop (HDFS, YARN, MapReduce) The original open-source big-data stack Mostly legacy now; HDFS and YARN still run under some Spark clusters Older banks and telecoms, own data centres
Hive SQL on Hadoop; its metastore still catalogues tables The table catalogue (Hive Metastore) Anywhere Spark reads “Hive tables”
Spark General distributed engine Big batch jobs, ML features, lakehouse tables Everywhere
Databricks Commercial platform built on Spark and Delta Lake Notebooks, jobs, SQL and ML in one place Retail, banks, media, on any cloud
Apache Flink Stream-first engine that keeps state Low-delay streaming: fraud, live metrics Payments, adtech, ride-hailing
Apache Beam / Dataflow One programming model for batch and streaming; Dataflow runs it on Google Cloud Serverless pipelines on Google Cloud Google Cloud-heavy companies
Trino / Presto Distributed SQL over data where it lives Ad hoc SQL on lake files; joining many sources Lakehouses; Amazon Athena is built on it
Dask Parallel pandas and NumPy Scaling Python code you already have Science, research
Ray Distributed Python for ML and AI Model training, tuning, large batch inference ML platforms
Polars / DuckDB Very fast single-machine engines Up to tens or hundreds of GB on one machine Analysts, small teams, tests, dbt-duckdb

Think of vehicles. DuckDB and Polars are a fast car: perfect for one person and most trips. Spark is a fleet of lorries: overkill for groceries, essential for moving a warehouse. Flink is a motorbike courier: carries small things constantly and never stops. Trino is a taxi that comes to you: it goes to where the data already is. Ray is a team of specialist movers for awkward heavy items such as model training.

3. Choosing an engine: four questions

  1. How big? Up to roughly 100 GB: DuckDB or Polars on one machine. Bigger, or growing fast: Spark, Dataflow, or the warehouse itself.
  2. How fresh must results be? Hours: batch Spark or SQL. Minutes: small batches (Spark Structured Streaming). Seconds or less: Flink, Dataflow streaming or Kafka Streams.
  3. Where does the data already live? All in BigQuery: often just SQL in BigQuery, no Spark at all. Files in a lake: Spark or Trino.
  4. What does the team know? A Python team that knows pandas becomes productive in PySpark quickly. A Java team may prefer Flink or Beam.

Here are the four questions applied to one real-looking request from a hotel group: “Every night, turn 30 GB of booking files into a table of nights sold per hotel.”

How big? 30 GB a night. That fits on one machine with room to spare. Point to DuckDB or Polars.
How fresh? Nightly is fine; nobody needs it before breakfast. Batch, no streaming.
Where does it live? Files land in Cloud Storage, and the result goes to BigQuery, where the analysts work. BigQuery could load the files and do it in SQL, too.
Who maintains it? Two analysts who know SQL and a little Python. A cluster would be one more thing to learn and pay for.
Decision. SQL in BigQuery (or DuckDB in a small scheduled job). Revisit Spark if the data grows past a few hundred GB a night or needs heavy custom Python. Write that threshold down: it is your rationale.

Cost is the other half of the decision. On the cloud you pay per machine-hour while a cluster runs.

Single-machine engine
DuckDB or Polars: all the data on one computer, no network, no start-up wait.
Stream-first engine
Flink: handles each event as it arrives and remembers state between events.
Query engine
Trino: reads data in place, in the lake or other databases, without loading it first.
Managed runner
Dataflow: Google runs your Beam pipeline and decides the machines.
Physical meaning
For a restaurant chain with 30 GB of orders a night, a small scheduled DuckDB or BigQuery job may cost a few dollars a month; a Spark cluster for the same job costs more and needs someone who knows Spark. Choosing the smaller engine on purpose, and writing down when you would switch, is a sign of experience.

“Real data engineers always use Spark.” Many teams moved their transformations out of Spark and into the warehouse (BigQuery SQL plus dbt), because SQL is easier to hire for and the warehouse scales itself. Spark stays for what SQL does badly: very heavy joins on lake data, complex Python logic, ML feature engineering and unstructured data. Saying this in an interview shows you have seen real platforms. Pipeline to Dashboard teaches the single-machine side with Polars, DuckDB and dbt.

4. Mini project for Module 2: the nightly sales job

Brief (2 to 3 hours). Build sales_nightly.py, a PySpark batch job you can show on GitHub.

  1. Input: generate 2 million rows of synthetic till data with the Lesson 2.2 code. Add customer_id with 10 % nulls, and one “online” store with 40 % of the rows, to create skew.
  2. Clean: typed dates, refunds flagged, duplicate txn_ids removed (keep the latest), null customer_id set to "guest".
  3. Model: a daily store-by-product fact table and a store dimension. Use a broadcast join for stores.
  4. Write: Parquet partitioned by sale_date, with files of roughly 64 to 256 MB.
  5. Check: rows in = rows out + duplicates removed (log the duplicates); no store-day with negative net revenue unless it has refunds.
  6. Explain: a README with the explain() plan, a Spark UI screenshot of the slowest stage, and what you changed to make it faster.

Checklist for “done”:

  • The job runs end to end with one command: python sales_nightly.py --date 2026-10-15.
  • Running it twice for the same date gives the same output (it overwrites that date, never appends).
  • The row-count check prints numbers and fails the job if they don’t add up.
  • The README shows one before-and-after timing with the reason for the change.

Acceptance checks: row counts balance; the plan shows BroadcastHashJoin for stores; the slowest task is no more than about three times the median after your skew fix. Keep the project: in Module 3 you will run it on Dataproc against Cloud Storage, scheduled by Airflow.

New words: Hadoop, Hive Metastore, Databricks, Flink, Beam, Dataflow, Trino, Dask, Ray, Polars, DuckDB.

  • Spark is the general engine; Flink is for low-delay streams; Beam/Dataflow is serverless on Google Cloud; Trino queries data in place; DuckDB and Polars are fast on one machine.
  • Choose by four questions: size, freshness, where the data lives, and what the team knows.
  • Many teams now do SQL transformations in the warehouse and keep Spark for heavy, non-SQL work.
  • Cluster cost is machines × hours × rate: more machines for less time costs about the same.
  • Write down the threshold at which you would switch engines; that is your design rationale.

Check yourself

1Which engine is built first for true event-at-a-time streaming with low delay?
2Trino is best described as...
3Your team writes Apache Beam pipelines. On Google Cloud, which service runs them fully managed?

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. All company data in the examples is synthetic. Tool versions and prices change: check the official docs before you rely on them.