System Design Monitoring
Monitoring means watching a system to learn how it behaves and to catch problems early. A running system produces a constant stream of signals. Monitoring collects those signals, stores them, and turns them into charts and alerts.
Why Monitoring Matters
Servers fail at night. Traffic spikes during a sale. A slow database query hides inside a rarely used feature. Teams that watch their systems fix issues before customers complain. Teams that skip monitoring learn about outages from angry users.
Layman example
A car dashboard shows speed, fuel, and engine heat. A warning light appears before the engine overheats. Monitoring gives a software system the same dashboard.
The Three Pillars
Metrics
Metrics are numbers measured over time. CPU usage, memory use, request count, and error rate all qualify. Charts of metrics reveal trends and sudden jumps.
Logs
Logs are text records of events. A log line might say that a payment failed at 10:42 with a specific error code. Logs explain what happened in detail.
Traces
A trace follows one request through every service it touches. Each step records its start and end time. A trace shows exactly where a slow request lost its time.
Request #8841 (total 920 ms) |-- API Gateway ........... 20 ms |-- Order Service ......... 80 ms |-- Payment Service ....... 780 ms <-- slow step found |-- Email Service ......... 40 ms
The Four Golden Signals
Google's site reliability engineers suggest four signals as the starting set for any service.
| Signal | Question It Answers |
|---|---|
| Latency | How long do requests take? |
| Traffic | How much demand hits the system? |
| Errors | How many requests fail? |
| Saturation | How full are the key resources? |
Monitoring Pipeline
[Services] --> [Agents collect data] --> [Storage]
|
+------------------+----------------+
v v
[Dashboards] [Alert Rules]
(charts for people) (notify the on-call engineer)
Dashboards
A dashboard puts the most important charts on one screen. A good dashboard answers one question at a glance: "Is the system healthy right now?" Keep each dashboard focused. Crowded screens hide the real problem.
Alerting
Alerts push a message to a person when a metric crosses a limit. An alert might fire when the error rate passes 5 percent for five minutes.
Rules for Good Alerts
- Alert on symptoms users feel, such as slow pages or failed checkouts.
- Attach a clear message that says what broke and where to look.
- Route urgent alerts to a phone and minor alerts to chat or email.
- Delete alerts that nobody acts on.
Too many alerts cause alert fatigue. Tired engineers start ignoring messages, and a real emergency slips through.
SLI, SLO, and SLA
- SLI (Service Level Indicator): a measured number, such as the share of requests that succeed.
- SLO (Service Level Objective): the target for that number, such as 99.9 percent success each month.
- SLA (Service Level Agreement): a promise to customers, often with penalties when the team misses the target.
The gap between 100 percent and the SLO is the error budget. Teams spend this budget on new releases. A used-up budget tells the team to slow down and fix reliability.
Health Checks and Uptime Probes
Synthetic checks mimic real users. A small script opens the homepage or runs a test login every minute from several locations. A failed script raises an alert before real users notice.
Best Practices
- Add a unique request ID to every log line so teams can follow one request across services.
- Store logs in a central place, not on individual servers.
- Keep sensitive data such as passwords out of logs.
- Review dashboards and alerts after each incident.
- Track business numbers too, such as orders per minute.
Popular Tools
Teams commonly use Prometheus and Grafana for metrics and dashboards, the ELK stack for logs, and OpenTelemetry with Jaeger for traces. Cloud providers offer managed versions of the same ideas.