Home / Learn / Data & AI / Data at Scale
All lessons

Data at Scale: big data, cloud and streaming, explained simply

How companies process data too big for one computer: Spark and PySpark, the Google Cloud data stack (Cloud Storage, BigQuery, Dataproc, Airflow), Druid, Bigtable and Elasticsearch, Kafka and stream processing, data platforms for AI, and retail media measurement, with animations and live problems.

For analysts and BI developers who know some SQL and Python and keep seeing these tools in job postings, and anyone who wants the big picture first. Free, no sign-up. All 32 lessons, about 13.3 hours.

What you will be able to do

  • Explain when data is "big" and which tool fits which job, from one laptop to a cluster.
  • Work with Spark: how it splits work, PySpark basics, and why shuffles and skew make jobs slow.
  • Use the cloud data stack: a data lake on object storage, BigQuery, Dataproc and Airflow, and what each one costs.
  • Pick the right database: Druid, Bigtable, Elasticsearch and the rest of the database zoo, and stream events with Kafka.
  • Feed machine learning and AI, and understand the data behind retail media and advertising measurement.

Every lesson has a step-by-step animation, a plain-words meaning box, live problems with fresh numbers, a short quiz and a summary sheet to download. The course ends with mini projects and free sandboxes so you can practise without paying. Take Pipeline to Dashboard first if pipelines and warehouses are new to you.

0 of 32 lessons done

Without the tick, your progress lasts until you close this tab. Either way it stays in this browser: there is no sign-up and nothing is sent to us.

All modules

Part A: Processing at scale

1 The big picture 3 lessons

  1. 1.1 Why big data needs different tools
  2. 1.2 The map: how the pieces connect
  3. 1.3 Who uses what, and why, by industry

2 Spark and distributed processing 4 lessons

  1. 2.1 How distributed processing works
  2. 2.2 PySpark DataFrames, hands on
  3. 2.3 Making Spark fast and cheap
  4. 2.4 Spark's relatives, and when not to use Spark

3 The Google Cloud data stack 5 lessons

  1. 3.1 A tour of the Google Cloud data stack
  2. 3.2 Cloud Storage and the data lake
  3. 3.3 BigQuery: the warehouse with no servers
  4. 3.4 Dataproc: Spark on Google Cloud
  5. 3.5 Airflow and Cloud Composer

Part B: Storing and streaming

4 Analytical and NoSQL databases 4 lessons

  1. 4.1 The database zoo
  2. 4.2 Apache Druid and real-time OLAP
  3. 4.3 Bigtable and wide-column stores
  4. 4.4 Elasticsearch, and choosing a store

5 Streaming with Kafka 4 lessons

  1. 5.1 Thinking in events and streams
  2. 5.2 Kafka: the log at the centre
  3. 5.3 Reliable streams: guarantees, schemas, Connect and CDC
  4. 5.4 Stream processing: windows, watermarks and state

Part C: Platforms in practice

6 Data platforms for AI and ML 4 lessons

  1. 6.1 What machine learning needs from a data platform
  2. 6.2 Feature stores, pipelines and MLOps
  3. 6.3 Data for LLMs and generative AI
  4. 6.4 Platform architecture: putting it together

7 Retail media, adtech and measurement 4 lessons

  1. 7.1 Retail media 101
  2. 7.2 Adtech plumbing: how an ad gets served
  3. 7.3 Measurement: did the ads work?
  4. 7.4 Capstone: design a retail media data platform

8 What to learn next 4 lessons

  1. 8.1 What to learn, in what order
  2. 8.2 Job postings, interviews and honest stories
  3. 8.3 Mini projects for your portfolio
  4. 8.4 Free sandboxes to practise in