SRE Runbooks and Playbooks

At three in the morning, nobody thinks clearly. A runbook gives a tired engineer a clear path from alert to fix. Teams that write good runbooks recover faster and train new members with less stress.

What Is a Runbook?

A runbook is a short guide that explains how to handle one specific task or alert. Think of a recipe card. A recipe lists the ingredients, the order of steps, and the signs that the dish is ready. A runbook lists the checks, the commands, and the signs that the service has recovered.

What Is a Playbook?

A playbook covers a larger situation. It describes roles, decision points, and communication for events such as a regional outage or a security incident. Runbooks give detailed steps. Playbooks give the overall plan.

Parts of a Good Runbook

1. Title and owner: the alert name and the team that maintains the page.
2. Meaning: one or two sentences on what the alert says about users.
3. Impact: who feels the problem and how badly.
4. Checks: commands or dashboards that confirm the cause.
5. Fix steps: numbered actions in a safe order.
6. Verification: signs that prove the fix worked.
7. Escalation: who to call when the steps fail.

Example Runbook: Disk Space Above 90 Percent

Meaning

The log server disk is almost full. Writes may stop and the application may crash.

Checks

• Open the disk dashboard and confirm usage above 90 percent.
• List the biggest folders to find the source of growth.

Fix

1. Compress log files older than seven days.
2. Delete compressed files older than thirty days.
3. Confirm usage drops below 75 percent.

Escalation

Call the platform team lead if usage stays above 85 percent after cleanup.

Decision Flow Diagram

[Alert Fires]
     ↓
[Open Runbook]
     ↓
[Run Checks] → Cause confirmed? → No → [Escalate]
     ↓ Yes
[Apply Fix Steps]
     ↓
[Verify Recovery] → Recovered? → No → [Escalate]
     ↓ Yes
[Close Alert and Note Findings]

Writing Style Tips

• Use short sentences and plain words.
• Start each step with a verb.
• Show the exact command and the expected output.
• Warn before dangerous actions, such as deleting data.
• Link every alert directly to its runbook.

Keeping Runbooks Fresh

An old runbook misleads. Teams review pages after every incident and fix any step that failed. New engineers test runbooks during training, because fresh eyes spot unclear steps. Mark every page with a last-reviewed date.

From Runbook to Automation

Repeated runbook steps signal a chance to automate. Start with a script that performs the safe parts. Add a button that runs the script after a human approves. The final stage runs the fix automatically and sends a note to the team. Each stage lowers toil and speeds recovery.

Common Mistakes

• Writing pages that explain theory and skip commands.
• Storing runbooks in a place that goes offline during an outage.
• Leaving out the verification step.
• Creating one huge page for many alerts.

Key Takeaways

Runbooks turn knowledge from a few experts into guidance that anyone on call can use. Keep each page short, test it often, and replace steps with automation when you can.