Paths/Data Engineer/Orchestration & Scheduling / DAG Design
Plan Slots and Backfills
Catchup runs, concurrency limits and backfills - the operational side of a schedule. Write the configuration that keeps a backlog from melting the cluster.
Validator
⏱ 0:00
IDLE — checks run live as you type commands; submit when done
01Requirements
live checks- File ~/orchestration/dags/schedule.yaml exists○
- The config states the schedule and whether catchup is allowed○
- The config limits how many runs may be active○
- The config describes how a backfill is run○
02Walkthrough
1 of 4 revealed- 01
Write the schedule configuration
catchup decides what happens after a holiday - false means the missed days are skipped, true means every one of them runs.
echo 'dag_id: daily_load\nschedule_interval: "@daily"\ncatchup: false\nmax_active_runs: 3\nstart_date: "2026-01-01"' > ~/orchestration/dags/schedule.yaml - 3 locked · each failed submit reveals one
03Stuck?
0 of 3 revealedTake a hint without losing the lab — each one points at the thing to look at, not at the answer.
04AI Tutor
guides, never solvesTell me what you're stuck on — I'll nudge, not solve. Which requirement are you looking at?
04Notebook
saved locally in this browserlearner@lab — ubuntu:24.04
Session is disposable
↑/↓ recalls commands · Ctrl+Enter submits · type help for the command list