Paths/MLOps Engineer/ML Pipelines / Your First Pipeline
Declare the Pipeline DAG
Lay out ~/pipeline with one file per step and a pipeline.yaml that names the execution order — the shape every orchestrator (Airflow, Dagster, Kubeflow) recognises instantly.
Validator
IDLE — checks run live as you type commands; submit when done
01Requirements
live checks- ~/pipeline/steps contains ingest.txt, features.txt and train.txt○
- pipeline.yaml declares the step order ingest,features,train○
02Walkthrough
8 steps- 01
Create the pipeline root
One directory for the whole pipeline definition.
mkdir ~/pipeline - 02
Move into it
Everything below is created relative to ~/pipeline.
cd ~/pipeline - 03
Add the steps directory
One file per step keeps the DAG inspectable — anyone can see what runs without reading code.
mkdir steps - 04
Declare the ingest step
The first step of every ML pipeline — pulling raw data from its source.
echo read_raw_data > steps/ingest.txt - 05
Declare the feature step
Feature engineering — transforming raw data into model-ready columns.
echo transform_features > steps/features.txt - 06
Declare the train step
The final step — fitting the model on the prepared features.
echo fit_model > steps/train.txt - 07
Write the execution order
The DAG in one line. Orchestrators read exactly this kind of ordered declaration to build their dependency graph.
echo steps=ingest,features,train > pipeline.yaml - 08
Inspect and submit
cat prints the pipeline definition. The validator checks every step file and the declared order.
cat pipeline.yaml
03Stuck?
0 of 3 revealedTake a hint without losing the lab — each one points at the thing to look at, not at the answer.
04AI Tutor
guides, never solvesTell me what you're stuck on — I'll nudge, not solve. Which requirement are you looking at?
04Notebook
saved locally in this browserlearner@lab — ubuntu:24.04
Session is disposable
↑/↓ recalls commands · Ctrl+Enter submits · type help for the command list