Home / Learn / Data & AI / Data at Scale / Lesson 6.2
All lessons

Module 6: Data platforms for AI and ML · Lesson 6.2 · about 25 min

Feature stores, pipelines and MLOps

Offline and online feature stores, materialisation, ML pipelines, experiment tracking, model registries and serving, with Vertex AI, Feast and MLflow.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 6 · about 25 minutes · read first: 6.1. You’ll be able to explain offline and online feature stores and materialisation, work out how stale an online feature can be, sketch an ML pipeline from data to deployed model, and name the MLOps tools on Google Cloud and in open source.

A feature store defines each feature once, keeps its full history for training and its latest value for live models, and a job called materialisation keeps the two in step.

1. The feature store

A feature store is a catalogue plus two storage layers that make features reusable and consistent:

  • Feature definitions: named, owned and documented, such as shopper.buys_30d, computed from fct_sales.
  • Offline store: the full history, used to build training sets with point-in-time joins (Lesson 6.1). Usually BigQuery or lake tables.
  • Online store: only the latest value per entity, for fast lookups by live models. Usually Bigtable, Redis or DynamoDB.
  • Materialisation: jobs (batch or streaming) that compute feature values and copy the newest ones into the online store.

Think of a pharmacy. The offline store is the warehouse that keeps every batch ever made, with dates, for audits and research. The online store is the counter: only what is current, ready to hand over in seconds. Materialisation is the van that restocks the counter from the warehouse. The list of feature definitions is the formulary: what each medicine is and who is responsible for it.

Options include Vertex AI Feature Store (built on BigQuery, with online serving), Feast (open source), Databricks feature engineering, Tecton and SageMaker Feature Store. Many teams simply use well-modelled BigQuery tables plus a Bigtable or Redis copy: a feature store in all but name.

10:00. A sale lands. Shopper s77 buys coffee at store 214. The sale reaches fct_sales in BigQuery at 10:12.
10:15. Materialisation starts. The job recomputes buys_7d for shoppers with new sales. For s77 it goes from 3 to 4.
Offline store: append. A new row (s77, 10:15, buys_7d = 4) is added. The old row (s77, yesterday, 3) stays, so training sets can still ask "what was it yesterday?"
Online store: overwrite. The key s77 now holds buys_7d = 4. Only the latest value is kept.
10:31. A live request. s77 opens the app. The offer model reads s77 from the online store in a few milliseconds and sees 4: the same number, computed by the same code, that training rows use.
Feature definition
The one agreed recipe for a feature: its source, its formula, its owner.
Offline store
Every value over time, for building training sets.
Online store
The latest value per entity, for live predictions in milliseconds.
Materialisation
The scheduled or streaming job that refreshes the online store.
Physical meaning
When a pharmacy app shows a member a vitamin offer the moment they log in, a model read that member's features from a fast online store, filled minutes earlier by a materialisation job, using the same feature recipe the model was trained with months before.

The online store holds just one row per entity, but for millions of shoppers that still adds up:

Freshness is the other number to know. A batch materialisation job can only be as fresh as its schedule allows:

2. ML pipelines

An ML pipeline is a DAG like the ones you built with Airflow, with machine learning steps:

  1. Extract a training snapshot (point-in-time features plus labels) from BigQuery.
  2. Validate the data: schema, ranges, and drift compared with last time.
  3. Train (scikit-learn, XGBoost, PyTorch or BigQuery ML) and tune.
  4. Evaluate against the current live model, including fairness checks.
  5. Register the model with its metrics and lineage.
  6. Deploy (a batch scoring job or an online endpoint) after approval.
  7. Monitor, and trigger retraining when needed.

Tools: Vertex AI Pipelines (runs Kubeflow Pipelines definitions), Kubeflow, Airflow (common for batch retraining), Metaflow, ZenML and Databricks workflows.

3. Tracking, registry and serving

Need Google Cloud Open source and others
Experiment tracking Vertex AI Experiments MLflow, Weights & Biases
Model registry Vertex AI Model Registry MLflow Model Registry
Batch prediction Vertex AI batch prediction, BigQuery ML ML.PREDICT Spark or Airflow jobs
Online serving Vertex AI endpoints KServe, BentoML, Ray Serve
Monitoring Vertex AI Model Monitoring Evidently, custom checks
Distributed training Vertex AI training on GPUs and TPUs Ray, PyTorch distributed, Spark MLlib

MLOps is the name for running all of this reliably: the machine learning version of the testing, deployment and monitoring habits from software engineering.

“We need to buy a feature store before we can do machine learning.” Many teams run well for years on tested, documented BigQuery tables for training and a small Redis or Bigtable copy for serving. Buy or adopt a feature store when several teams reuse the same features and the hand-made copies start to disagree.

4. What “a platform supporting ML” looks like

  • Clean, documented, time-stamped tables in the warehouse that data scientists trust.
  • Raw history kept in the lake (Parquet or Iceberg), so features can be rebuilt with new definitions.
  • Reusable feature pipelines (Spark, BigQuery SQL or dbt, Flink for streaming) with tests.
  • An online store fed by those pipelines, with freshness monitoring.
  • Reproducibility: each model linked to its code version, data snapshot and features.
  • Cost controls on GPU and training jobs, and access controls on personal data.

If you have built tested star schemas and reconciled numbers with finance (as in Pipeline to Dashboard), you already have the most valuable skill here: trustworthy, well-modelled data. Add point-in-time correctness, feature freshness and model lineage to your vocabulary and you can talk with machine learning teams as an equal.

5. Try it (optional code, about 30 minutes)

Install Feast and follow its quickstart, which builds a small demo repository with sample data:

pip install feast
feast init my_project
cd my_project/feature_repo
feast apply                                   # register the feature definitions
feast materialize-incremental $(date -u +"%Y-%m-%dT%H:%M:%S")   # fill the online store

Then run the quickstart’s script that fetches online features. Finally, write three sentences that map each Feast piece (feature view, offline store, online store, materialize) to its Vertex AI equivalent.

Check: the online lookup returns values for the demo’s driver IDs, and running materialize-incremental again with a later time changes them only for rows with newer data.

New words: feature store, feature definition, offline store, online store, materialisation, ML pipeline, experiment tracking, model registry, endpoint, MLOps.

  • A feature store = feature definitions + an offline store (full history, for training) + an online store (latest values, for serving).
  • Materialisation refreshes the online store; worst-case staleness = data landing delay + time between runs + run time.
  • An ML pipeline is a DAG: snapshot, validate, train, evaluate, register, deploy, monitor.
  • Experiment tracking and a model registry make every model reproducible and every deployment reversible.

Check yourself

1What does the online store of a feature store hold?
2What is materialisation?
3Why keep a model registry?

Sources

  • Feast documentation: Quickstart and concepts https://docs.feast.dev/
  • MLflow documentation: Tracking and Model Registry https://mlflow.org/docs/latest/index.html
  • Google Cloud documentation: Vertex AI Feature Store and Vertex AI Pipelines (cloud.google.com/vertex-ai/docs)
  • Sculley, D. et al. (2015), Hidden Technical Debt in Machine Learning Systems, Advances in Neural Information Processing Systems 28

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. All company data in the examples is synthetic. Tool versions and prices change: check the official docs before you rely on them.