SRE Troubleshooting Methods
Troubleshooting is a skill that anyone can learn. Strong engineers follow a method instead of guessing. A clear method keeps the mind calm, saves time, and avoids new damage during stressful moments.
Think Like a Doctor
A doctor listens to symptoms, forms ideas about causes, orders tests, and chooses a treatment. Doctors avoid treating before they understand. SRE troubleshooting follows the same path: observe, guess, test, and fix.
The Four-Step Loop
[Observe Symptoms] → [Form Hypotheses] → [Test the Best Hypothesis]
↑ ↓
└──────── [Fix and Verify] ←───────────────┘
(if the test fails, return to Observe)
Observe
Collect facts. Which users suffer? Which region? When did it start? Which charts moved? Write the answers in the incident notes.
Form Hypotheses
List several possible causes and rank them by likelihood and by how cheap they are to check.
Test
Check one hypothesis at a time. A clear yes or no answer moves the search forward.
Fix and Verify
Apply the smallest safe change. Watch the signals and confirm that users recover.
Start with Recent Changes
Most failures follow a change. Ask what changed in the last hour, the last day, and the last week: releases, configuration edits, traffic shifts, certificate dates, and vendor notices. A change timeline often reveals the cause within minutes.
Divide and Conquer
Split the request path into layers and test the middle. Each test removes half of the search area.
[User] → [DNS] → [Load Balancer] → [Web App] → [Cache] → [Database]
ok ok ok slow ok ok
↑
Problem found here
The USE Method
For each resource, such as CPU, memory, disk, and network, check three things.
| Letter | Meaning | Question |
|---|---|---|
| U | Utilization | How busy is the resource? |
| S | Saturation | Does work wait in a queue? |
| E | Errors | Does the resource report failures? |
A quick pass over every resource often exposes the bottleneck.
Useful Questions
- What works, and what does not?
- Does the problem affect everyone or a group?
- Can we reproduce it?
- When did it last work?
- What differs between a healthy server and a sick one?
Example: Slow Search Page
- Observation: p95 search latency rose from 300 ms to 4 seconds at 9:10 AM. Only mobile users in one region feel it.
- Recent changes: a cache configuration update shipped at 9:05 AM.
- Hypothesis: the new setting shortened cache lifetime and sent more queries to the database.
- Test: the cache hit ratio dropped from 92 percent to 18 percent, and database CPU doubled.
- Fix: revert the configuration. Latency returns to normal within three minutes.
- Follow-up: add a cache hit ratio alert.
Keep Notes as You Work
Write each test, result, and decision in the incident channel. Notes prevent repeated work, help the next shift, and feed the postmortem.
Stay Safe While Investigating
- Prefer read-only checks before changes.
- Change one thing at a time.
- Ask for a second pair of eyes before risky commands.
- Prefer rollback over a fresh fix during an outage.
Common Mistakes
- Jumping to a favorite cause without evidence.
- Changing five settings at once.
- Ignoring a rollback option.
- Working silently and duplicating effort.
Key Takeaways
Observe facts, rank hypotheses, test one idea at a time, and verify the fix. Check recent changes early, split the system into layers, and record every step.