Module · Data Engineer
Change Data Capture
Read the log instead of polling the table, and keep the schema stable while you do it.
TOPIC 01
Reading the Log
Replication slots, publications and the offset that tells you where you were.
What you'll practice
- Configure the Replication Slot — guided · beginner
- Resume From a Checkpoint — practice · intermediate
A command you'll meet
cat ~/cdc/config.yamlCDC is how you get changes without hammering the source table. Open the task below for the full walkthrough.
Hands-On Tasks in This Topic (2)
Configure the Replication Slot
The slot is a place in the write-ahead log that a reader can follow. Write the configuration that points it at your tables.
Resume From a Checkpoint
A connector that dies has to come back to exactly where it was. Write the offset policy that makes that possible.
TOPIC 02
Schema Evolution
The source table will change anyway. Decide how you meet it halfway.
What you'll practice
- Handle a Column Change — practice · intermediate
- Make the Sink Idempotent — challenge · advanced
A command you'll meet
cat ~/cdc/evolution.yamlA capture pipeline is a long conversation with a schema you do not control. Open the task below for the full walkthrough.
Hands-On Tasks in This Topic (2)
Handle a Column Change
A renamed column or a dropped one breaks every consumer downstream. Write the rule for meeting the change halfway.
Make the Sink Idempotent
At-least-once delivery means the same change can arrive twice. Write the strategy that makes the second copy a no-op.