Module 7: Run it in production · Lesson 7.3 · about 35 min
Choosing tools with a written rationale
Compare orchestrators, ingestion and transformation tools, port a pipeline to a second orchestrator, then decide with a weighted matrix and an ADR.
Module 7 · about 35 minutes · read first: 3.1 and 3.2. You’ll be able to compare orchestrators, ingestion tools and transformation tools on the criteria that matter, port a pipeline to a second orchestrator, and turn a tool choice into a weighted matrix, a sensitivity check and an ADR anyone can argue with.
1. Why this matters
Job ads name different tools: Airflow here, Azure Data Factory there, Prefect somewhere else. Nobody masters all of them. What you can master is the set of shared ideas from Modules 1 to 6: idempotent landing, business dates, partitions, retries, checks, upserts. Every orchestrator expresses the same ideas in its own words, the way every car has a steering wheel and brakes even if the buttons move. Someone who knows the ideas can switch tools in a week.
2. Orchestrators side by side
| Dagster | Airflow | Prefect | Mage | Kestra | Azure Data Factory / Fabric | |
|---|---|---|---|---|---|---|
| Thinks in | assets (data first) | tasks in DAGs; assets added in Airflow 3 | Python flows and tasks | blocks in a notebook-like UI | YAML flows, event driven | low-code pipelines |
| Local dev and tests | strong | heavier | very easy | easy | easy (Docker) | in the portal |
| Partitions and backfills | built in | data intervals | your own patterns | basic | good | tumbling windows |
| Lineage | built in, with dbt | through OpenLineage | limited | basic | basic | through Purview |
| Hosting | open source or paid cloud | open source or several managed services | open source or paid cloud | open source or paid | open source or paid | pay per use in Azure |
This is a starting point, not a verdict: features move every release, so check each tool’s docs for what is current. In job ads, Airflow is still the most requested, Dagster keeps growing in newer stacks, and Data Factory is everywhere in Microsoft-based companies.
The same hotel pipeline in Airflow
from airflow.sdk import dag, task # Airflow 3; Airflow 2 used: from airflow.decorators import dag, task
import pendulum
@dag(schedule="15 6 * * *", start_date=pendulum.datetime(2026, 9, 1, tz="America/Toronto"), catchup=True)
def hotel_daily():
@task(retries=3)
def land(ds=None): # ds = the run's logical date, YYYY-MM-DD: our business date
return pms.land_day(ds)
@task
def stage(path: str):
return pms.stage(path)
stage(land())
hotel_daily()
And in Prefect
from prefect import flow, task
@task(retries=3, retry_delay_seconds=30)
def land(bdate: str) -> str:
return pms.land_day(bdate)
@flow
def hotel_daily(bdate: str):
pms.stage(land(bdate))
Look at what did not change: the pms module, where the real work lives. That is the payoff of the rule from Lesson 3.2: keep business logic in plain functions and let the orchestrator only call them.
3. Ingestion and transformation tools
- dlt: a Python library, code first, runs inside any orchestrator, cheap. Good for APIs and files.
- Airbyte: hundreds of ready connectors with a UI; self-host (it needs some infrastructure) or use its cloud.
- Meltano: Singer taps and targets driven by a command line and config files; open source.
- Managed connectors (Fivetran, Data Factory copy, Fabric): the fastest start and the highest bill at scale, priced by volume or activity.
For transformation, dbt is the default (Lesson 5.4): dbt Core is open source, and dbt Cloud adds a hosted editor and scheduler. SQLMesh is the strongest alternative, with virtual environments and a plan-then-apply workflow. Stored procedures are still common in SQL Server shops.
4. Deciding on paper: the weighted matrix
A weighted decision matrix lists the criteria, gives each a weight (all weights add to 1), scores each option from 1 to 5, and adds up weight × score. It is like a judge’s scorecard: the rules are written before the contest, so anyone can check the result.
- Weighted matrix
- Criteria with weights that add to 1, a 1-to-5 score per option, total = sum of weight × score.
- Sensitivity check
- Move each weight by about 0.1 and see whether the winner changes.
- ADR
- Architecture decision record: a one-page note of the context, options, decision and consequences, kept in the repo.
- Thin orchestration
- The orchestrator only calls plain functions, so switching tools rewrites wiring, not logic.
- Physical meaning
- Two years later a new head of data asks "why are we on Dagster?" Instead of a shrug, there is ADR-002 with the matrix, the hinge and what would make you revisit it. She can agree, or change one weight and see the new answer.
Maths and principle
Principle: a weighted matrix makes trade-offs explicit, so people argue about the weights (their real disagreement) instead of the answer.
Equation: for option with scores from 1 to 5 and weights ,
Analogy: a school report where maths counts double: the weights say what matters, written down before the marks come in.
Where it breaks: criteria that measure the same thing (“easy to learn” and “fast onboarding”) count it twice. Merge them first.
Try it: weights cost 0.4, skills 0.4, features 0.2. Tool A scores 3, 5, 4; Tool B scores 5, 2, 5. Winner? (A: 4.0; B: 3.8. A, narrowly.)
“The tool with the most features wins.” Features you will not use are free on the scorecard but cost money and training in real life. For most teams, migration cost and the skills already in the room outweigh a longer feature list. A company with 300 Airflow DAGs and an Airflow-skilled team should usually stay on Airflow, adopt its asset features and OpenLineage, and keep its logic in plain Python and dbt.
In interviews, “why Dagster and not Airflow?” tests whether you can argue both sides. A good answer: “Dagster’s assets, partitions and local testing suited a small team building something new. In a company with hundreds of Airflow DAGs and an Airflow team, I would stay on Airflow and keep the logic portable.”
5. Try it
- Port the hotel pipeline (land → stage → on-the-books) to Airflow (the official Docker Compose setup, or the Astro CLI) or to Prefect (
pip install prefect, thenprefect server start). Keep the core module unchanged. - Run a backfill in that tool and note how its intervals or parameters compare with Dagster partitions.
- Re-build the ads ingest with dlt. If your machine allows, try Airbyte open source with a Postgres destination.
- Fill in a decision matrix for three company profiles: a 40-store retail chain on Microsoft 365; a 300-restaurant chain on Google Workspace; a 5-hotel group with no IT team. Do the sensitivity check for each. Start from the decision matrix CSV.
- Write ADR-002 (orchestrator) and ADR-003 (ingestion) for the capstone, using the ADR template.
What to check: the diff of your port touches only orchestration files; each matrix states whether the winner flips under a 0.1 weight change; each ADR says what would make you revisit it. Commit as docs+feat: Airflow or Prefect port, dlt trial, decision matrices, ADR-002 and ADR-003.
- Every orchestrator expresses the same ideas (business dates, partitions, retries, checks) in its own words.
- Keep business logic in plain functions; then a port changes only the wiring.
- Score options with a weighted matrix written before you look at the answer.
- Always run a sensitivity check and report the criterion the decision hinges on.
- Record the choice in an ADR: context, options, decision, consequences and when to revisit.
Check yourself
Sources
- Apache Airflow documentation: TaskFlow API and assets https://airflow.apache.org/docs/
- Prefect documentation: Flows and tasks https://docs.prefect.io/
- Dagster documentation: Assets and partitions https://docs.dagster.io/
- Nygard, M. (2011), Documenting Architecture Decisions (the original ADR article) https://cognitect.com/blog/2011/11/15/documenting-architecture-decisions
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. All company data in the examples is synthetic. Tool versions and prices change: check the official docs before you rely on them.