Paths/DevOps Engineer/Observability / Alerting
Write a Prometheus Alert Rule
Alert rules are YAML files too. Write a Prometheus rule that fires when the error rate stays too high — and learn the two halves of every alert: expression and duration.
Validator
IDLE — checks run live as you type commands; submit when done
01Requirements
live checks- File ~/alerts/rules.yml exists○
- It defines an alert named HighErrorRate with a Prometheus expression○
02Walkthrough
8 steps- 01
Create the rules directory
Prometheus loads rule files from configured paths — one directory keeps them tidy.
mkdir ~/alerts - 02
Start the rule file
groups wrap rules; a rule has name, expr, for and labels. The expr is PromQL.
echo 'groups:' > ~/alerts/rules.yml - 03
Add the group and alert name
HighErrorRate says exactly what broke — alert names are the first thing you see at 3am, so make them readable.
echo '- name: app-alerts' >> ~/alerts/rules.yml - 04
Add the expression
expr: is PromQL — here: rate of 5xx responses over 5 minutes divided by total request rate, above 5%. That's 'more than 5% of requests failing'.
echo ' rules:' >> ~/alerts/rules.yml - 05
Finish the rule
Write the rule body: '- alert: HighErrorRate', ' expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.05', ' for: 5m'. The for clause stops blips from paging you.
echo ' - alert: HighErrorRate' >> ~/alerts/rules.yml - 06
Add the expression line
The PromQL condition — more than 5% of requests failing over 5 minutes. Indentation is cosmetic in this lab; the validator reads the content.
echo expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) >0.05 >> ~/alerts/rules.yml - 07
Add the duration clause
The for clause is what stops a 30-second blip from paging someone — the alert must stay firing continuously for 5 minutes first.
echo for: 5m >> ~/alerts/rules.yml - 08
Verify and submit
cat prints the finished rule — alert name, expression and duration. Then submit.
cat ~/alerts/rules.yml
03Stuck?
0 of 3 revealedTake a hint without losing the lab — each one points at the thing to look at, not at the answer.
04AI Tutor
guides, never solvesTell me what you're stuck on — I'll nudge, not solve. Which requirement are you looking at?
04Notebook
saved locally in this browserlearner@lab — ubuntu:24.04
Session is disposable
↑/↓ recalls commands · Ctrl+Enter submits · type help for the command list