Module · Data Engineer
Orchestration & Scheduling
Dependencies, retries and backfills - the difference between a job and a pipeline.
TOPIC 01
DAG Design
Declare what depends on what, and let the scheduler work out the order.
What you'll practice
- Declare Task Dependencies — guided · beginner
- Plan Slots and Backfills — practice · intermediate
A command you'll meet
cat ~/orchestration/dags/daily_load.pyA DAG is a promise about order. Open the task below for the full walkthrough.
Hands-On Tasks in This Topic (2)
Declare Task Dependencies
The scheduler reads the dependencies, not the file order. Declare the tasks of the daily load and what waits for what.
Plan Slots and Backfills
Catchup runs, concurrency limits and backfills - the operational side of a schedule. Write the configuration that keeps a backlog from melting the cluster.
TOPIC 02
Reliability & Recovery
Retries, deadlines and the alert that tells you before your stakeholders do.
What you'll practice
- Set Retry and Backoff Policies — challenge · advanced
- Alert on a Late Pipeline — practice · intermediate
A command you'll meet
cat ~/orchestration/policies.yamlA pipeline that fails quietly is worse than one that fails loudly. Open the task below for the full walkthrough.
Hands-On Tasks in This Topic (2)
Set Retry and Backoff Policies
Transient failures are normal - a network hiccup is not a broken pipeline. Write the policy that says when to try again and when to give up.
Alert on a Late Pipeline
Consumers need the data by a certain time, so the pipeline needs a deadline of its own. Write the alert that fires when the daily load misses it.