Module · AI Engineer
LLM Evaluation
Measure what your prompts and models actually do, before your users tell you.
TOPIC 01
Building Eval Datasets
The dataset is the product. Write test cases and golden answers worth trusting.
What you'll practice
- Write an Evaluation Dataset — guided · beginner
- Curate a Golden Answer Set — practice · intermediate
A command you'll meet
cat ~/evals/dataset.jsonlOne JSON object per line - diff friendly, streaming friendly. Open the task below for the full walkthrough.
Hands-On Tasks in This Topic (2)
Write an Evaluation Dataset
An eval dataset is a list of cases the model has to survive. Write three cases for a support assistant - one line of JSON each.
Curate a Golden Answer Set
Golden answers are the reference the model is measured against, so they need a human behind them. Write the approved answer and record who reviewed it.
TOPIC 02
Metrics & Release Gates
Turn a run into numbers, and turn the numbers into a gate that blocks bad releases.
What you'll practice
- Report the Metrics of a Run — practice · intermediate
- Gate Releases on Eval Results — challenge · advanced
A command you'll meet
cat ~/evals/gate.yamlA metric nobody gates on is a metric nobody reads. Open the task below for the full walkthrough.
Hands-On Tasks in This Topic (2)
Report the Metrics of a Run
A run is only useful if it produces comparable numbers. Write the report with the accuracy, the latency and the cost per request.
Gate Releases on Eval Results
A gate is a rule that fails the build. Write the gate configuration - the accuracy floor, the regression limit and what happens when either is broken.