Module 7: Run it in production · Lesson 7.1 · about 35 min
Observability and incident response
Turn reports into measurable promises: freshness and completeness SLOs, error budgets, a health board, an evidence ladder for triage and blameless reviews.
Module 7 · about 35 minutes · read first: 3.5 and 6.6. You’ll be able to write freshness and completeness promises for each report, measure them, work out the error budget, build a pipeline health board from tables you already have, and handle an incident from first alert to a written review.
1. Why this matters
Up to now you built things. From here, half the job is keeping promises like “the restaurant report is in inboxes by 08:00, with every restaurant included, on 98% of days”. A bus company works the same way. It does not promise every bus is on time. It promises 95% are, measures it every day, and fixes the routes that drag the number down.
Observability means you can see, from data the pipeline leaves behind, whether it is healthy. Incident response is the calm routine you follow when it is not.
2. Promises you can measure: SLOs
An SLO (service level objective) is a written, measurable promise about a report. For data, two kinds matter most:
- Freshness: the data is up to date by a time. “Sent by 08:00 on 98% of days.”
- Completeness: all the expected parts are in. “At least 17 of 18 restaurants, or a banner says who is missing.”
| Report | Freshness SLO | Completeness SLO | Measured from |
|---|---|---|---|
| Restaurant daily email | Sent by 08:00 on 98% of days | 17 of 18 restaurants, or a banner | ops.report_sent, manifest |
| Hotel revenue page | Data as of 07:30 or earlier on 95% of days | All 5 hotels | ops.load_audit |
| Ads dashboard | Updated by 12:00 daily | Every campaign with spend | data quality results |
The error budget is the number of misses the promise allows. 98% over 30 days allows 2% × 30 = 0.6 late days. That sounds odd, but it is the point: one late day already spends more than the budget. When the budget is gone, reliability work comes before new features.
- SLO
- A measurable promise about a report, such as "sent by 08:00 on 98% of days".
- Freshness
- How up to date the data is when someone looks at it.
- Completeness
- Whether every expected part (store, hotel, campaign) is included.
- Error budget
- The misses the promise allows: (100% − SLO) × days. Spend it, and reliability work comes first.
- Physical meaning
- A regional manager reads the restaurant email at 08:05 before a call with the owner. If it arrives at 08:12 twice a month, she stops trusting it and goes back to phoning each restaurant. The SLO is how you notice before she does.
3. The health board
You do not need a new tool. Build a pipeline health dashboard from tables you already have:
- Dagster run history (from its GraphQL API or its Postgres storage): runs by status per day.
ops.load_auditfrom Lesson 4.3: rows loaded per table per day.dq_resultsfrom Lesson 2.6: failed checks by family.- The manifest from Lesson 1.3: which files arrived and when.
ops.report_sent(new today): one row per report sent, with its time.
Panels worth having: runs by status per day; duration per asset as p50 and p95 (the middle and the slow tail); freshness per source against its SLO; quality failures by family; error budget left. Metabase or Grafana both work.
-- Freshness SLO compliance for the restaurant email, last 30 days
select
count(*) as days,
sum(case when sent_at::time <= '08:00' then 1 else 0 end) as on_time,
round(100.0 * sum(case when sent_at::time <= '08:00' then 1 else 0 end) / count(*), 1) as pct
from ops.report_sent
where report = 'qsr_daily' and business_date >= current_date - 30;
Maths and principle
Principle: watch the tail, not the average. A pipeline whose average finish is 07:04 can still miss 08:00 often.
Equation: with finish times and a deadline,
is 1 when the condition is true and 0 when it is false, so the sum just counts on-time days.
Analogy: a fire alarm that works 99% of the time but was off on the night of the fire failed at its only job.
Where it breaks: small samples. Over 30 days one miss is 3.3 percentage points, so also judge over 90 days before you panic or relax.
Try it: SLO 98% over 30 days, 1 miss so far. Budget left? (0.6 − 1 = −0.4: already over.)
4. Triage: the evidence ladder
When an alert fires or someone says “the numbers look wrong”, climb from the cheapest evidence to the most expensive. Stop as soon as the evidence settles it. A doctor works the same way: temperature and questions first, scans only if needed.
- Is it real? Look at the alert’s own data. Maybe the check was unavailable, not failed.
- Where in the chain? Manifest (did the file arrive?) → run log (did the asset run?) → load audit (were rows loaded?) → data quality (were rows quarantined?) → mart (query it) → report (when did it refresh?).
- What changed? The source file’s shape, a code deploy (
git log), config, volumes. - Verdict, exactly one of: source issue (contact the owner), pipeline bug (fix plus a test), expected behaviour (explain, maybe label better), transient (a retry worked; keep watching), not yet determined (say what evidence would settle it).
You can ask Claude Code to run the ladder with you, in read-only mode, saving each piece of evidence to triage/<date>-<slug>/report.md. It is quick at reading logs; you stay responsible for the verdict.
“Unavailable means failed.” If the freshness check could not reach the manifest database, it measured nothing. Reporting “stale data” sends people chasing a problem that may not exist; reporting “fresh” hides one that might. The honest verdict is not yet determined, followed by checking the manifest another way.
5. Blameless post-incident reviews
Write a short review for anything that broke a promise: a timeline, the impact (who saw what wrong number, for how long), the root cause, why it was not caught earlier, and actions with owners and dates. Blameless means “the process let this happen”, never “Sam broke it”. Teams that write these get steadily more reliable; teams that skip them have the same incident every quarter. Download the incident review template to start.
6. Try it
Start the capstone repository daily-data-platform today; Lessons 7.1 to 7.6 all add to it.
- Write SLOs for your three reports in
docs/slo.md. - Create
ops.report_sentand make the email asset write one row each time it sends. - Run the pipeline over 30 simulated days with a backfill (
python generate_data.py --days 30 --out data --seed 7, from the course data generator). Inject two late hotel files and one broken restaurant file. Compute compliance with the query above. - Build the health board in Metabase or Grafana.
- Triage each injected incident with the evidence ladder and save the reports. Write one blameless review for the worst one.
What to check: compliance from your query matches a hand count of the late days; each triage report ends in exactly one verdict with evidence linked; the review names a process change, not a person. Commit as docs+feat(ops): SLOs, health board, triage reports, incident review.
- An SLO is a measurable promise; data needs freshness and completeness SLOs per report.
- Error budget = (100% − SLO) × days − misses; when it runs out, reliability work comes first.
- Watch p95 finish times, not the average: deadlines are missed in the tail.
- Triage cheapest evidence first, along the chain, and end with one clear verdict (“not yet determined” is allowed).
- Blameless reviews fix the process, so the same incident does not come back.
Check yourself
Sources
- Beyer, B., Jones, C., Petoff, J. and Murphy, N. R. (2016), Site Reliability Engineering, O'Reilly, Chapter 4 (service level objectives) and Chapter 15 (postmortem culture) https://sre.google/sre-book/table-of-contents/
- Reis, J. and Housley, M. (2022), Fundamentals of Data Engineering, O'Reilly, Chapter 2 (DataOps: observability and incident response)
- Dagster documentation: Run history, asset health and alerts https://docs.dagster.io/
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. All company data in the examples is synthetic. Tool versions and prices change: check the official docs before you rely on them.