Paths/Data Engineer/Orchestration & Scheduling / DAG Design
Declare Task Dependencies
The scheduler reads the dependencies, not the file order. Declare the tasks of the daily load and what waits for what.
Validator
IDLE — checks run live as you type commands; submit when done
01Requirements
live checks- Directory ~/orchestration/dags exists○
- File ~/orchestration/dags/daily_load.py exists○
- The file declares the tasks○
- The file states what each task depends on○
02Walkthrough
5 steps- 01
Create the dags directory
One file per DAG - the name of the file is the name of the schedule.
mkdir -p ~/orchestration/dags - 02
Declare the first task
extract, transform and load - the three verbs of every batch pipeline. Each one is a function the scheduler can call.
echo 'def extract():\n return pull_rows_since_last_run()\n\ndef transform(rows):\n return dedupe_and_type(rows)\n\ndef load(rows):\n return write_to_warehouse(rows)' > ~/orchestration/dags/daily_load.py - 03
Declare the dependencies
depends_on is the whole point of a DAG - the scheduler works out the order, retries and parallelism from it.
echo '\n\ndaily_load = DAG(\n dag_id="daily_load",\n schedule_interval="@daily",\n start_date=datetime(2026, 1, 1),\n)\n\nextract_task = PythonOperator(task_id="extract", python_callable=extract)\ntransform_task = PythonOperator(task_id="transform", python_callable=transform, depends_on=[extract_task])\nload_task = PythonOperator(task_id="load", python_callable=load, depends_on=[transform_task])' >> ~/orchestration/dags/daily_load.py - 04
Verify the DAG
cat prints the file. Every task needs a dependency, or it runs at the wrong time.
cat ~/orchestration/dags/daily_load.py - 05
Submit for validation
The validator checks the three tasks and the dependency declaration.
03Stuck?
0 of 3 revealedTake a hint without losing the lab — each one points at the thing to look at, not at the answer.
04AI Tutor
guides, never solvesTell me what you're stuck on — I'll nudge, not solve. Which requirement are you looking at?
04Notebook
saved locally in this browserlearner@lab — ubuntu:24.04
Session is disposable
↑/↓ recalls commands · Ctrl+Enter submits · type help for the command list