SRE Prometheus and Grafana
Prometheus and Grafana form one of the most popular open-source pairs for monitoring. Prometheus collects and stores numbers. Grafana turns those numbers into charts that people read at a glance.
The Roles of Each Tool
Imagine a weather station. Sensors record temperature every minute and store the readings. A display board shows graphs for visitors. Prometheus plays the sensor and storage role. Grafana plays the display board.
How Prometheus Works
Prometheus uses a pull model. Every fifteen seconds or so, it visits each service at an address such as /metrics, reads the current numbers, and saves them with a time stamp. Services only need to publish their numbers on a web page.
[App A /metrics] ←──┐
[App B /metrics] ←──┼── pull ── [Prometheus] ── query ──→ [Grafana]
[Server Exporter]←──┘ │
└── rules ──→ [Alertmanager] → [Email / Chat / Pager]
Metric Types
| Type | Behavior | Example |
|---|---|---|
| Counter | Only goes up | Total requests served |
| Gauge | Goes up and down | Current memory in use |
| Histogram | Counts values in buckets | Request duration spread |
| Summary | Reports percentiles directly | Response time quantiles |
Labels
Labels add detail to a metric. A metric named http_requests_total can carry labels for method, status, and service. Labels let you slice the same number many ways, such as errors for one service only.
Keep label values limited. A label that holds user IDs creates millions of series and overloads the database. Experts call this a high cardinality problem.
Querying with PromQL
PromQL is the query language of Prometheus. A few short patterns cover most daily needs.
Requests per Second
rate(http_requests_total[5m])
Error Ratio
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))Ninety-Fifth Percentile Latency
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
The rate function converts a growing counter into a speed. The square brackets set the time window.
Alert Rules and Alertmanager
Prometheus evaluates alert rules on a schedule. A rule can say that an error ratio above 2 percent for five minutes should fire an alert. Alertmanager receives the alert, groups related alerts, removes duplicates, silences planned work, and sends messages to the right team.
Building a Grafana Dashboard
- Add Prometheus as a data source.
- Create a dashboard and add a panel.
- Paste a PromQL query and choose a chart type.
- Set units, thresholds, and clear titles.
- Arrange panels in the order of the golden signals.
- Save the dashboard and share the link.
Example: Shop Health Board
+----------------------+----------------------+ | Requests per second | Error ratio | | ~~~~/\~~~~ | ______/\_____ | +----------------------+----------------------+ | p95 latency | Database connections | | ____/\______ | ________/---- 80% | +----------------------+----------------------+
The four panels map to traffic, errors, latency, and saturation. An on-call engineer sees the story within seconds.
Good Habits
- Name metrics clearly, with units such as seconds or bytes.
- Store dashboards as code so changes pass review.
- Use recording rules for expensive queries that many charts share.
- Plan long-term storage if you need data beyond a few weeks.
Common Mistakes
- Putting unbounded values into labels.
- Building dashboards with dozens of panels nobody reads.
- Alerting on raw CPU instead of user-facing symptoms.
- Forgetting to monitor Prometheus itself.
Key Takeaways
Prometheus pulls and stores numbers, PromQL asks questions about them, Alertmanager routes warnings, and Grafana draws the pictures. Together they give SRE teams a clear window into service health.