Paths/Data Engineer/Batch Ingestion & ETL / Reliable Loads
Extract Only What Changed
Re-reading the whole table every hour is how pipelines fall over. Write the incremental extract that only asks for rows newer than the last watermark.
Validator
⏱ 0:00
IDLE — checks run live as you type commands; submit when done
01Requirements
live checks- File ~/pipelines/incremental.py exists○
- The script reads the current watermark○
- The script filters the source on the watermark○
- The script writes the new watermark back○
02Walkthrough
1 of 4 revealed- 01
Write the watermark helpers
The watermark is the high-water mark of the last successful run. Keep it in the metadata store, not in the code.
echo 'def read_watermark(table):\n return metadata.get(table, "1970-01-01T00:00:00Z")\n\ndef write_watermark(table, value):\n metadata.set(table, value)' > ~/pipelines/incremental.py - 3 locked · each failed submit reveals one
03Stuck?
0 of 3 revealedTake a hint without losing the lab — each one points at the thing to look at, not at the answer.
04AI Tutor
guides, never solvesTell me what you're stuck on — I'll nudge, not solve. Which requirement are you looking at?
04Notebook
saved locally in this browserlearner@lab — ubuntu:24.04
Session is disposable
↑/↓ recalls commands · Ctrl+Enter submits · type help for the command list