SRE Root Cause Analysis

Restoring service ends the pain. Understanding why the failure happened prevents the next one. Root cause analysis, often shortened to RCA, is the careful search for the deeper reasons behind an incident.

What Is a Root Cause?

A root cause is a condition that, once removed, stops the same failure from returning. Think of a leaking ceiling. Mopping the floor treats the symptom. Finding the broken roof tile treats the cause. Most real incidents have several causes working together, so teams often speak of contributing factors instead of a single culprit.

Symptom, Trigger, and Cause

Term	Meaning	Example
Symptom	What users see	Checkout pages time out
Trigger	The event that started the failure	A configuration change at 10:02
Root cause	The weakness that allowed the trigger to hurt	No validation on configuration values

Method 1: The Five Whys

Ask "why" repeatedly until the answer points to a process or design gap. Five is a guide, and the real number varies.

Problem: Checkout failed for 40 minutes.
Why 1: The database refused new connections.
Why 2: The connection limit was set to 10 instead of 1,000.
Why 3: A new configuration file shipped with a test value.
Why 4: The release process does not check values against safe ranges.
Why 5: Nobody owns validation rules for configuration.
Action: Add range checks to the release pipeline and assign an owner.

Method 2: The Fishbone Diagram

A fishbone diagram groups possible causes into categories. The team brainstorms in each branch and marks the likely ones.

  People            Process             Technology
     \                 |                   /
      \                |                  /
       ──────────────────────────────────────→  [Checkout Outage]
      /                |                  \
     /                 |                   \
  Environment       Monitoring          Dependencies

Questions for Each Branch

• People: Did the on-call engineer have the right access and training?
• Process: Did a review or approval step exist, and did it work?
• Technology: Did the code, config, or hardware behave as designed?
• Environment: Did load, region, or timing play a role?
• Monitoring: Did alerts fire quickly and point to the problem?
• Dependencies: Did an outside service change behavior?

Method 3: Timeline Review

List every event in time order, including quiet moments before the alert. Gaps in the timeline reveal missing monitoring. Surprising delays reveal slow decisions. Many causes appear only when the whole sequence sits on one page.

Method 4: Fault Tree

              [Checkout Down]
                    │
          ┌─────────OR─────────┐
          ↓                    ↓
 [Database Unavailable]   [Payment API Fails]
          │
   ┌─────AND──────┐
   ↓              ↓
[Low Limit]  [High Traffic]

Read the tree from top to bottom. An OR gate means any branch alone can cause the failure. An AND gate means both conditions must occur together.

Avoid the Human Error Trap

Many reports stop at "an engineer made a mistake." People make mistakes in every workplace. A strong analysis asks why the system allowed one mistake to cause so much damage. Better checks, safer defaults, and automated tests turn a human slip into a harmless event.

Turning Findings into Actions

Finding	Action	Owner	Due
No range check on configuration	Add validation to the pipeline	Release team	Two weeks
Alert fired 15 minutes late	Lower the alert threshold	Monitoring team	One week

Each action needs a named owner and a date. Track actions to completion, because an unfinished list repeats the same outage.

Common Mistakes

• Naming a single cause for a complex failure.
• Blaming a person instead of a process.
• Writing vague actions such as "be more careful."
• Filing the report and never checking the results.

Key Takeaways

Pick a method that fits the incident, gather facts before opinions, look for system gaps, and assign concrete actions. Good analysis makes the next incident smaller or removes it entirely.