Module · Data Engineer
Streaming & Event Pipelines
Events, topics and the delivery guarantees you can actually promise.
TOPIC 01
Event & Topic Design
Name the events, give them a schema, and choose the key that keeps the order.
What you'll practice
- Version an Event Schema — guided · beginner
- Lay Out Your Topics — practice · intermediate
A command you'll meet
cat ~/streaming/schemas/order_placed.jsonAn event is a promise to every consumer that reads it. Open the task below for the full walkthrough.
Hands-On Tasks in This Topic (2)
Version an Event Schema
Every event needs a schema, a version and a timestamp - consumers build their state from it. Write the contract for the order placed event.
Lay Out Your Topics
One topic per event, split into partitions, with a retention window you chose on purpose. Write the layout.
TOPIC 02
Delivery Semantics
At-least-once or exactly-once - and the code you write to live with the answer.
What you'll practice
- Choose a Delivery Guarantee — challenge · advanced
- Handle Late and Out-of-Order Events — practice · intermediate
A command you'll meet
cat ~/streaming/semantics.mdGuarantees are cheap to promise and expensive to keep. Open the task below for the full walkthrough.
Hands-On Tasks in This Topic (2)
Choose a Delivery Guarantee
Brokers promise at-least-once, and exactly-once is a promise about state, not about the wire. Write down what you promise and how you keep it.
Handle Late and Out-of-Order Events
Network delays mean events arrive late. Decide how long you wait, and what happens to the ones that arrive after the deadline.