Paths/AI Engineer/LLM Evaluation / Building Eval Datasets
Curate a Golden Answer Set
Golden answers are the reference the model is measured against, so they need a human behind them. Write the approved answer and record who reviewed it.
Validator
⏱ 0:00
IDLE — checks run live as you type commands; submit when done
01Requirements
live checks- Directory ~/evals/golden exists○
- File ~/evals/golden/refund-answer.txt exists○
- The answer states the refund window○
- File ~/evals/golden/review.txt records the reviewer and the date○
02Walkthrough
1 of 5 revealed- 01
Create the golden directory
Golden answers live apart from the dataset - they are reviewed, versioned and signed off.
mkdir -p ~/evals/golden - 4 locked · each failed submit reveals one
03Stuck?
0 of 3 revealedTake a hint without losing the lab — each one points at the thing to look at, not at the answer.
04AI Tutor
guides, never solvesTell me what you're stuck on — I'll nudge, not solve. Which requirement are you looking at?
04Notebook
saved locally in this browserlearner@lab — ubuntu:24.04
Session is disposable
↑/↓ recalls commands · Ctrl+Enter submits · type help for the command list