Module · AI Engineer
LLMOps
Evaluation, observability and serving in production.
TOPIC 01
Prompt Versioning and Evals
Treat prompts like code — commit each version to git and grade every change against test cases.
Prompt Versioning and Evals
A prompt that works is an asset; a prompt nobody can reproduce is a liability. LLMOps borrows two habits from software engineering: commit every prompt version to git, and before shipping a change, run it past a set of test cases — an eval — and record the score.
What you'll practice
- Version Your Prompts in Git — practice · beginner
- Write an Eval Results File — challenge · beginner
A command you'll meet
git commit -m "v1 prompt"Open any task below to get the full step-by-step walkthrough — every command is explained, and the lab validator checks what you actually built.
Hands-On Tasks in This Topic (2)
Version Your Prompts in Git
Prompts change constantly, and "which wording was live last Tuesday?" is only answerable if every version is committed. Put a v1 system prompt and an improved v2 into a git repository as two separate commits.
Write an Eval Results File
An eval is a set of test cases a prompt must survive, plus the score from the latest run. Create a cases file with three test cases and record a passing score in results.txt.
TOPIC 02
Deploying & Rolling Out Models
The endpoint that serves the model, and the rollout that replaces it without downtime.
What you'll practice
- Configure the Inference Endpoint — practice · intermediate
- Plan a Canary Rollout — challenge · advanced
A command you'll meet
cat ~/llmops/endpoint.yamlA model rollout is a traffic problem before it is a model problem. Open the task below for the full walkthrough.
Hands-On Tasks in This Topic (2)
Configure the Inference Endpoint
Everything the serving layer needs is configuration - which model, where it listens, how long it may think and what happens when it fails. Write it.
Plan a Canary Rollout
Never swap a model in one step. Send a slice of the traffic to the candidate, watch the metrics, and know exactly when to roll back.