Home / Learn / Data & AI / Data at Scale / Lesson 2.1
All lessons

Module 2: Spark and distributed processing · Lesson 2.1 · about 25 min

How distributed processing works

Drivers, executors, partitions, lazy plans and shuffles: what really happens when Spark runs your code on a cluster, shown with six rows on three workers.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 2 · about 25 minutes · read first: 1.1. You’ll be able to explain the driver, executors, tasks and partitions, tell transformations from actions, and spot the narrow and wide operations that decide how fast a job runs.

Spark splits your data into partitions, sends one small task per partition to many machines, and only moves data across the network when rows with the same key must meet; that move, the shuffle, is where jobs get slow.

1. Spark in one paragraph

Apache Spark is an engine that runs data processing across many machines. You write what you want (in Python with PySpark, in SQL, Scala, Java or R) and Spark works out how to split it into small tasks, runs them in parallel and puts the results together. It started at UC Berkeley in 2009 as a faster successor to Hadoop MapReduce, mainly because it keeps data in memory between steps instead of writing it to disk after every one.

Picture marking 100,000 exam papers. The head examiner (the driver) marks nothing. They split the papers into bundles (partitions), hand bundles to markers (executors, each with a few pairs of hands, the cores) and collect the totals. “How many passed?” is easy: each marker counts their bundle and reports one number. But “rank every student within their school” means papers from the same school must end up on one desk, so everyone stands up and swaps papers. That swapping is the shuffle, and it is always the slow part.

2. The parts of a Spark application

  • Driver: one process that runs your program, builds the plan and schedules tasks. If the driver dies, the job dies.
  • Executors: processes on worker machines that run tasks and hold data in memory. Each has some cores (slots that run one task each) and some memory.
  • Cluster manager: hands out machines. On Google Cloud that is YARN on Dataproc, or Kubernetes; Databricks has its own.
  • Partition: one chunk of the data. One task processes one partition. 100 GB in 128 MB pieces is about 800 partitions, so about 800 tasks per step.

Put together: a job (started by one action) is cut into stages (a new stage begins at each shuffle), and each stage runs one task per partition. Tasks run in waves: with 50 cores, 800 tasks go 50 at a time, 16 waves.

3. Lazy evaluation: Spark plans before it runs

Spark operations come in two kinds:

  • Transformations describe a new dataset: filter, select, withColumn, join, groupBy. They return at once and run nothing.
  • Actions need a result: count(), show(), collect(), write. Only now does Spark run.

Waiting lets Spark’s optimiser (called Catalyst) look at the whole recipe and improve it: push filters down so less data is read, read only the columns you use, pick the best join. It is like a chef who reads the whole order before cooking, instead of starting each dish the moment a word is spoken.

If a partition is lost because an executor crashed, Spark recomputes just that piece from the recipe, called the lineage. That is how it survives failures without copying all the data everywhere.

4. Narrow and wide operations

  • Narrow: each output partition needs only one input partition. filter, select, withColumn. Fast, because no data moves. Each marker works on their own bundle.
  • Wide: an output partition needs rows from many input partitions. groupBy, most joins, distinct, orderBy, repartition. These trigger a shuffle: executors write rows to local disk, send them over the network to the executor that owns each key, and the next stage reads them.

Here is a shuffle you can follow by hand: “total sales per store” on six rows spread over three workers. Each row is a store letter and an amount.

Start. Rows sit wherever they were read: worker 1 has A 5 and A 1, worker 2 has B 3 and A 2, worker 3 has C 4 and B 1. Store A is spread over two workers.
Add up locally (narrow). Each worker first adds rows with the same store inside its own partition: worker 1 turns A 5 and A 1 into A 6. No network used, and there is less to send later.
Shuffle (wide). Every store is assigned to one worker by hashing its key: A to worker 1, B to 2, C to 3. A 2 travels from worker 2 to worker 1; B 1 travels from 3 to 2. Two rows crossed the network.
Finish. Each worker adds up the store it now owns: A 8, B 4, C 4. This was one job with two stages, split at the shuffle.

With six rows the shuffle is instant. With six billion rows, the “two rows crossed the network” becomes hundreds of gigabytes written to disk and sent between machines. Shuffles are where jobs get slow, run out of memory or cost money, so half of Spark tuning (Lesson 2.3) is about making them smaller or avoiding them.

Driver
The planner: turns your code into stages and tasks and hands them out.
Executor
A worker process that runs tasks on partitions, with a few cores each.
Task
One partition processed by one core, in one stage.
Transformation / action
A step in the recipe / the request that makes Spark actually cook.
Shuffle
Rows sent between machines so that equal keys meet. Starts a new stage.
Physical meaning
When a retailer's nightly job "totals sales per store", every receipt for store 214 must end up on one machine. If receipts were stored by date, that means moving almost all of them across the network: the time and cost you see on the bill is mostly that move.

“Spark is fast because it uses lots of machines.” Machines help only with work that splits cleanly. A job full of shuffles can be slower on 50 machines than DuckDB on one laptop, because the network becomes the bottleneck. Spark is fast when most steps are narrow and the shuffles are few and small.

5. What Spark is made of

Piece What it is for
Spark SQL and DataFrames Tables with columns, written as SQL or DataFrame code. What you will use 90 % of the time.
Structured Streaming The same DataFrame code run on a stream, in small batches (Module 5).
MLlib Machine learning spread over a cluster (regression, trees, recommendations).
RDDs The original low-level API. You will read about it and rarely write it.

You will meet Spark as open-source Spark on Dataproc (Google Cloud), EMR (AWS) and Microsoft Fabric, and most of all on Databricks, the company founded by Spark’s creators. Retailers use it for nightly processing of billions of receipts and for building features for demand models; ad platforms to join impressions to sales for attribution; banks for risk reports over years of transactions.

6. Try it (no code)

For each step, write N (narrow) or W (wide):

  1. Keep only rows from 2026.
  2. Add a column price_with_tax = price * 1.13.
  3. Total sales per store.
  4. Join sales to a 300-row store table.
  5. Sort all rows by date.
Answers
  1. N. 2. N. 3. W (shuffle by store). 4. Usually W, but Spark will copy a tiny 300-row table to every executor (a broadcast join), which makes it effectively narrow: Lesson 2.3. 5. W (a global sort shuffles everything).

What to check: every W has a key that rows must be grouped by (store, join key, sort order).

New words: driver, executor, core, task, stage, wave, transformation, action, lazy evaluation, Catalyst, lineage, narrow, wide, shuffle.

  • The driver plans; executors run one task per partition per stage, in waves as wide as the number of cores.
  • Transformations build a plan; actions run it. Being lazy lets Spark optimise the whole recipe first.
  • Narrow operations (filter, select, withColumn) stay inside a partition; wide ones (groupBy, join, orderBy) shuffle.
  • Each shuffle starts a new stage and is usually the slowest, costliest step.
  • Lost partitions are recomputed from their lineage, so jobs survive failed machines.

Check yourself

1In Spark, which part plans the work and hands out tasks?
2You write df.filter(...).select(...) and nothing runs. Why?
3Which operation causes a shuffle?

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. All company data in the examples is synthetic. Tool versions and prices change: check the official docs before you rely on them.