SRE Golden Signals
Large systems produce thousands of metrics. Nobody can read them all during an outage. The four golden signals give engineers a short checklist that reveals the health of almost any service in minutes.
The Restaurant Picture
Imagine a busy restaurant. The manager watches four things: how long guests wait for food, how many guests arrive, how many plates go back to the kitchen, and how full the dining room is. Those four observations describe the restaurant better than a hundred small counters. The golden signals work the same way for software.
The Four Signals
1. Latency
Latency measures how long a request takes. Separate successful requests from failed ones, because a fast error hides real slowness. Track percentiles instead of averages. An average of 200 milliseconds can hide the fact that 1 user in 100 waits 8 seconds.
| Percentile | Meaning | Sample Value |
|---|---|---|
| p50 | Half of users wait less than this | 120 ms |
| p95 | 95 of 100 users wait less than this | 450 ms |
| p99 | 99 of 100 users wait less than this | 2,100 ms |
2. Traffic
Traffic counts how much demand the system receives. Web services measure requests per second. Databases measure queries per second. Streaming services measure bytes delivered. A sudden jump signals a campaign or an attack. A sudden drop signals a broken entry point.
3. Errors
Errors count failed requests. Include loud failures such as HTTP 500 responses and quiet failures such as a 200 response that carries the wrong content. Express errors as a rate, such as 0.4 percent of requests, so the number stays meaningful at any traffic level.
4. Saturation
Saturation shows how full the most limited resource has become. CPU, memory, disk, network, and connection pools all fill up. Services slow down well before they reach 100 percent, so teams alert at a safe margin such as 80 percent.
Where the Signals Sit
User │ Latency: time from click to page ▼ [Load Balancer] ← Traffic: requests per second │ ▼ [Web Application] ← Errors: failed responses per 100 │ ▼ [Database] ← Saturation: connections used / connections allowed
Reading the Signals Together
One signal rarely tells the whole story. Patterns across signals point to causes.
| Pattern | Likely Cause |
|---|---|
| Traffic up, latency up, saturation high | Capacity too small for demand |
| Traffic steady, errors up right after a release | Faulty code change |
| Traffic down, errors flat | Users cannot reach the service |
| Latency up, saturation low | Slow dependency or network trouble |
Example: Online Ticket Sale
A concert ticket page normally handles 300 requests per second. At noon the sale opens, and traffic climbs to 4,000. Latency at p99 rises from 1 second to 9 seconds. The database connection pool reaches 98 percent. Errors stay near zero until the pool fills, and they jump to 12 percent. The dashboard tells the whole story in one glance: demand grew, a shared resource saturated, and users began to fail. The team adds database connections and enables a waiting-room page, and the signals return to normal.
Turning Signals into Alerts
- Alert on error rate above the target for five minutes.
- Alert on p99 latency above the promise made to users.
- Warn on saturation above 80 percent so the team acts before users suffer.
- Watch traffic for sharp drops, since silence often signals trouble.
Common Mistakes
- Reading averages and missing slow tails.
- Mixing failed and successful requests in a single latency chart.
- Tracking CPU only and ignoring queues or connection limits.
- Building dashboards with 40 charts when four would answer the question.
Key Takeaways
Latency, traffic, errors, and saturation form a compact health check for any service. Place these four charts at the top of every dashboard, and your on-call engineer will find the problem faster.