Module · Data Engineer
Batch Ingestion & ETL
Load the data twice without breaking anything - the art of idempotent pipelines.
TOPIC 01
Reliable Loads
Jobs get retried, sources repeat themselves, and the table must still end up correct.
What you'll practice
- Load Data Idempotently — guided · beginner
- Extract Only What Changed — practice · intermediate
A command you'll meet
cat ~/pipelines/load_ingest.sqlIf running the job twice changes the result, the job is not finished. Open the task below for the full walkthrough.
Hands-On Tasks in This Topic (2)
Load Data Idempotently
Pipelines get retried - the load has to converge to the same table every time. Write the merge that updates changed rows and inserts the new ones.
Extract Only What Changed
Re-reading the whole table every hour is how pipelines fall over. Write the incremental extract that only asks for rows newer than the last watermark.
TOPIC 02
Transforms & Contracts
Clean the data on the way in, and make the producer tell you what it sends.
What you'll practice
- Deduplicate in SQL — practice · intermediate
- Write a Data Contract — challenge · advanced
A command you'll meet
cat ~/pipelines/contracts/orders.yamlThe contract is how the producer and the consumer stop arguing. Open the task below for the full walkthrough.
Hands-On Tasks in This Topic (2)
Deduplicate in SQL
Sources repeat themselves, and types arrive as strings. Write the transform that keeps the newest row per key and casts the columns.
Write a Data Contract
The producer owns the shape of the data. Publish the contract, state how breaking changes are handled, and nobody is surprised on a Monday morning.