Observability & Reliability
Prometheus & Observability
Metrics, log aggregation, and real-time operational telemetry for distributed systems.
Observability answers unknown-unknown questions about system health. Prometheus provides a pull-based time-series metrics engine with PromQL querying, paired with Alertmanager for intelligent paging and Grafana for operational visualization.
Virtual Lab Ready
Automated testing sandbox running in your browser.
Launch First Lab ⚡No local installation required
Hands-on Practice
Interactive Terminal Labs
Real environment tasks with automated validation. Write real commands, verify state, and debug errors.
Count HTTP 5xx Errors
Topic: Metrics from Logs
Every dashboard number starts as raw events. The app's access log is at /var/log/app/access.log — count the HTTP 500 responses and record the number your dashboard would show.
Key Checks:
- Directory ~/report exists
- File ~/report/5xx.txt exists and contains the 500 count
Extract Errors for the Incident Report
Topic: Log Triage
During an incident you hand the on-call engineer a filtered log, not a gigabyte file. Extract every ERROR line from yesterday's app log into a dedicated file.
Key Checks:
- File ~/report/errors.log exists
- It contains only ERROR lines from the 2026-09-27 application log
Write a Prometheus Alert Rule
Topic: Alerting
Alert rules are YAML files too. Write a Prometheus rule that fires when the error rate stays too high — and learn the two halves of every alert: expression and duration.
Key Checks:
- File ~/alerts/rules.yml exists
- It defines an alert named HighErrorRate with a Prometheus expression
Write the SLO and Its Error Budget
Topic: SLOs & Error Budgets
The product team promised customers 99.9% availability this quarter — a promise nobody can act on until it is written down as two numbers. Record the target in ~/slo/slo.txt and work out the error budget it implies into ~/slo/budget.txt.
Key Checks:
- Directory ~/slo exists containing slo.txt and budget.txt
- slo.txt records the availability target of 99.9
Build a Dashboard Definition
Topic: Dashboards & Burn Rates
Dashboards are code - a JSON definition that can be reviewed and deployed. Build one for the API with a request-rate panel and an error-rate panel.
Key Checks:
- Directory ~/dashboards exists
- File ~/dashboards/api.json exists
Alert on Error Budget Burn
Topic: Dashboards & Burn Rates
Paging on every error wastes attention. Burn-rate alerting pages when the error budget is being consumed far too fast - write the rule that does it.
Key Checks:
- Directory ~/alerts exists
- File ~/alerts/burn-rate.yml exists
System Design
Metrics Scraping & Time-Series Engine
Understand how this tool operates internally before running production workloads.
Internal Mechanics
Prometheus scrapes HTTP /metrics endpoints from targets discovered via Kubernetes API or static configs. It stores samples in a custom TSDB on local disk and evaluates alerting rules periodically.
Mental Model Rule of Thumb
Never treat Prometheus & Observability as a black box. Understanding its boundary with the kernel, filesystem, and network stack prevents 90% of production outages.
Core Architectural Subsystems
Prometheus Server
Scrapes targets, stores time series in TSDB, and executes PromQL queries.
Exporters (node_exporter)
Translates Linux kernel and application statistics into Prometheus text format.
Alertmanager
Deduplicates, groups, and routes firing alerts to Slack, PagerDuty, or email.
Grafana
Visualization dashboards rendering time-series queries into panels and graphs.
Developer Reference
Command Reference & Cheatsheet
Curated commands, flags, and one-liners tested for everyday operational productivity.
rate(http_requests_total{status=~'5..'}[5m])Calculate per-second rate of 5xx HTTP server errors over last 5 minutes
sum by (service) (rate(http_requests_total[5m]))Aggregate total request throughput grouped by microservice name
histogram_quantile(0.99, sum(rate(http_duration_seconds_bucket[5m])) by (le))Compute 99th percentile (P99) request latency across endpoints
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode='idle'}[5m])) * 100)Calculate real-time CPU utilization percentage per node
Fundamental Primitives
Key Terminology & Primitives
Core abstractions you must master to communicate and build with confidence.
The 4 Golden Signals
Latency (response time), Traffic (requests/sec), Errors (rate of 5xxs), and Saturation (CPU/memory capacity).
Metric Types
Counter (monotonically increasing), Gauge (can rise and fall), Histogram (binned latency counts), Summary.
High Cardinality
Uncontrolled label values (e.g. user_id or IP addresses) that explode memory usage in time-series databases.
PromQL rate() vs irate()
rate() calculates per-second average rate over a time range; irate() calculates instantaneous rate between the last two data points.
Production War-Stories
Common Pitfalls & Debugging Playbook
The exact errors that take down production systems and how to diagnose and resolve them.
TSDB OOM crash due to high cardinality
Symptom
Prometheus memory usage spirals until killed by Linux kernel OOM killer.
Resolution
Never inject request IDs, UUIDs, or timestamps as Prometheus label values. Keep label cardinality strictly bounded.
Flapping alerts causing pager fatigue
Symptom
Alerts fire and resolve every 60 seconds during minor traffic spikes.
Resolution
Add a 'for: 5m' clause to alerting rules so conditions must persist continuously before triggering notification.
Reliability Standards
Production Readiness Checklist
Review this checklist before deploying workloads to staging or production clusters.
- ✓Define Service Level Objectives (SLOs) and Error Budgets for critical services.
- ✓Implement the 4 Golden Signals across all external and internal APIs.
- ✓Audit metrics for high cardinality labels before promoting to staging.
- ✓Test alert routing paths and PagerDuty escalations in game-day drills.
- ✓Configure retention policies and remote storage write (Cortex/Thanos/Mimir) for long-term historical metrics.
Full Curriculum Track
Master Prometheus & Observability as part of the DevOps Engineer path
Tools do not exist in isolation. Learn how Prometheus & Observability connects with other systems, CI/CD pipelines, and infrastructure patterns.