Paths/AI Engineer/LLM Evaluation / Building Eval Datasets
Write an Evaluation Dataset
An eval dataset is a list of cases the model has to survive. Write three cases for a support assistant - one line of JSON each.
Validator
IDLE — checks run live as you type commands; submit when done
01Requirements
live checks- Directory ~/evals exists○
- File ~/evals/dataset.jsonl exists○
- Every case has an id, an input and an expected output○
- The dataset contains at least three cases○
02Walkthrough
6 steps- 01
Create the evals directory
Every eval run needs its dataset - keep them in one folder.
mkdir ~/evals - 02
Write the first case
One JSON object per line. id names the case, input is what the user says, expected is what a good answer looks like.
echo '{"id": "refund-1", "input": "How long do refunds take?", "expected": "Refunds are processed within 5 business days"}' > ~/evals/dataset.jsonl - 03
Add the second case
>> appends a line instead of overwriting the file. Every case should probe a different failure mode.
echo '{"id": "policy-1", "input": "Can I return a used item?", "expected": "Items must be unused and in the original packaging"}' >> ~/evals/dataset.jsonl - 04
Add the third case
Include the hard case - the one your users hit and your prompt fails on.
echo '{"id": "edge-1", "input": "Refund my order but it was delivered to the wrong address", "expected": "Offer a replacement shipment and confirm the refund window"}' >> ~/evals/dataset.jsonl - 05
Check the line count
wc -l counts the cases - three is the smallest dataset that can catch anything.
wc -l ~/evals/dataset.jsonl - 06
Submit for validation
The validator checks the file and the fields every case needs.
03Stuck?
0 of 3 revealedTake a hint without losing the lab — each one points at the thing to look at, not at the answer.
04AI Tutor
guides, never solvesTell me what you're stuck on — I'll nudge, not solve. Which requirement are you looking at?
04Notebook
saved locally in this browserlearner@lab — ubuntu:24.04
Session is disposable
↑/↓ recalls commands · Ctrl+Enter submits · type help for the command list