SRE Logging Best Practices
Logs tell the story of what a program did and why. During an outage, a clear log saves hours, and a messy log wastes them. Good logging habits turn raw text into a reliable record that any engineer can search and understand.
What Is a Log?
A log is a time-stamped message that a program writes when something happens. Think of a ship captain's diary. Each entry records the time, the event, and the condition of the ship. Years later, investigators read the diary to learn what went wrong. Application logs serve the same purpose for software.
Structured Logs Beat Plain Text
Plain text logs read well for humans and poorly for machines. Structured logs use a fixed format, usually JSON, so tools can filter and count fields.
Plain Text
2026-11-01 10:02:11 Payment failed for user 4821 amount 59.00
Structured
{"time":"2026-11-01T10:02:11Z","level":"ERROR","service":"payments",
"event":"payment_failed","order_id":"A-9921","amount":59.00,
"trace_id":"7f3a9c"}A search for every payment_failed event in the payments service takes seconds with the structured version.
Log Levels
| Level | Use | Example |
|---|---|---|
| DEBUG | Detailed steps for developers | Cache lookup for key product-12 |
| INFO | Normal milestones | Order created |
| WARN | Unexpected event that the system survived | Retry after a slow response |
| ERROR | A failed operation | Payment declined by the gateway |
| FATAL | The program must stop | Cannot reach the database at startup |
Production usually runs at INFO level. Engineers raise the level to DEBUG for a short time while they investigate.
What to Include in Every Entry
- Time stamp in a standard format with a time zone
- Service name and version
- Level and a short event name
- Request or trace identifier
- Useful context such as order number or region
What Never to Log
Logs spread across many systems and stay for months. Keep sensitive data out of them.
- Passwords, tokens, and secret keys
- Full card numbers and bank details
- Health records and government IDs
- Personal data beyond what the investigation needs
Mask or hash identifiers when you must keep them.
The Correlation Identifier
A single click can pass through ten services. Give each request a unique identifier at the entry point and pass it along. Every service writes the identifier into its logs. A search for that one value returns the complete journey, in order.
[Web] id=7f3a9c → [Cart] id=7f3a9c → [Payments] id=7f3a9c → [Bank API] id=7f3a9c
search "7f3a9c" → full story across all four logs
A Central Logging Pipeline
[App Containers] → [Log Agent] → [Collector and Parser] → [Search Store]
↓
[Dashboards] [Alerts] [Archive]
Central storage lets engineers search all services from one screen. Servers that vanish during scaling or failures leave no local files behind, so central storage protects the evidence.
Control Volume and Cost
- Set retention by value: 14 days of detailed logs, 90 days of summaries, and one year in cheap archive storage.
- Sample noisy, low-value events such as health check hits.
- Drop DEBUG lines in production by default.
- Alert on sudden jumps in log volume, which often signal a loop or a bug.
Logs, Metrics, and Traces Together
A metric says that errors rose. A trace shows where the slow step sits. A log explains the exact failure. Link the three through shared identifiers so engineers move from chart to trace to log in a few clicks.
Example: Finding a Silent Failure
Customers report missing confirmation emails. The metric dashboard looks green. An engineer searches logs for level ERROR in the email service and finds thousands of "template_not_found" events after the last release. A missing file caused the failure. The team restores the file and adds an alert on email errors.
Common Mistakes
- Writing vague messages such as "error happened."
- Logging inside tight loops and flooding storage.
- Mixing time zones across services.
- Keeping secrets in log lines.
Key Takeaways
Write structured entries, use clear levels, attach a correlation identifier, protect sensitive data, and manage retention. Strong logs turn confusing incidents into short investigations.