SRE Disaster Recovery and Backups
Disks fail, regions go offline, and people delete the wrong table. A disaster recovery plan decides in advance how a service comes back. Teams that rehearse recovery lose less data and less time.
Backups and Disaster Recovery
A backup is a saved copy of data. Disaster recovery is the full plan for restoring the service, including people, systems, and order of actions. Imagine a house key. A spare key hidden in the garden is a backup. A written plan that says who fetches it, where to sleep, and who calls the locksmith is disaster recovery.
Two Numbers Every Plan Needs
Recovery Point Objective (RPO)
RPO states how much data the business accepts to lose, measured in time. An RPO of 15 minutes means the latest backup must never be older than 15 minutes.
Recovery Time Objective (RTO)
RTO states how long the service may stay down. An RTO of one hour means the team must restore service within sixty minutes.
Last good backup Disaster Service restored
│ │ │
───────●───────────────────X─────────────────────●──────→ time
│←──── RPO ────────→│←────── RTO ────────→│
(data you may lose) (time you may be down)
The 3-2-1 Backup Rule
- 3 copies of important data.
- 2 different storage types.
- 1 copy kept in another location.
Add one more guard: keep a copy that attackers cannot change or delete, often called an immutable backup.
Recovery Strategies
| Strategy | How It Works | Recovery Speed | Cost |
|---|---|---|---|
| Backup and restore | Rebuild from saved copies | Hours to days | Lowest |
| Pilot light | Keep core data replicated and servers off | Tens of minutes | Low |
| Warm standby | Run a small copy of the full system | Minutes | Medium |
| Active-active | Run full systems in two regions at once | Seconds | Highest |
Pick the cheapest strategy that still meets the RPO and RTO of the business.
Example: Regional Failure
Normal: Users → [Region A: Live] ──replicates──→ [Region B: Standby]
Disaster: Users → [Region A: Down]
↓ traffic switch
Users → [Region B: Takes Over]
A store runs warm standby in a second region. Database changes copy across every few seconds. When Region A loses power, the team switches DNS and promotes the standby database. Users see a short pause of eight minutes. The store loses about ten seconds of orders, which fits both objectives.
Building a Recovery Plan
- List critical services and rank them by business value.
- Set an RPO and RTO for each service.
- Choose a strategy and automate the setup.
- Write the recovery steps as clear runbooks.
- Assign owners and backup owners.
- Test on a schedule and record the results.
Test Your Restores
A backup that nobody restores is an unproven hope. Schedule restore drills every quarter. Restore into a clean environment, check that data opens correctly, and measure the real recovery time. Many teams discover broken backup jobs only during a drill, which is the best moment to find them.
Protect Against Human Error and Attacks
- Enable versioning so deleted files return.
- Encrypt backups and guard keys separately.
- Limit who can delete backups.
- Monitor backup jobs and alert on failures.
Common Mistakes
- Storing backups in the same location as the original data.
- Writing a plan and never rehearsing it.
- Forgetting configuration, secrets, and DNS records.
- Setting RPO and RTO with no input from the business.
Key Takeaways
Set clear RPO and RTO numbers, follow the 3-2-1 rule, choose a strategy that fits the budget, and test restores often. Preparation turns a disaster into a planned event.