SRE Release Engineering and Safe Deployments

A new drug goes through clinical trials before reaching patients — small groups first, then larger groups, with careful monitoring at each stage. If problems appear early, the trial stops before more people are affected. Safe deployment in SRE follows the same staged logic: release to a small slice of production, measure, then expand if everything looks good.

Why Deployments Are High-Risk Moments

Most production incidents trace back to a recent change — a new deployment, a configuration update, or an infrastructure modification. Change is the primary cause of instability in production systems. Release engineering is the discipline of making change as safe as possible without making it so slow that teams cannot deliver new features.

Root Cause Analysis of Incidents (typical distribution):
----------------------------------------------------------
Code change / new deployment:           45%
Configuration change:                   20%
Infrastructure change:                  15%
Capacity / traffic causes:              10%
External dependency failure:             7%
Unknown / hardware:                      3%

Continuous Integration and Continuous Delivery

Continuous Integration (CI)

Continuous Integration means every code change is automatically built and tested before it can be merged. A developer pushes code, and the CI system immediately runs unit tests, integration tests, and static analysis. If anything fails, the merge is blocked.

Developer pushes code
       ↓
CI System runs:
  - Unit tests (fast, seconds)
  - Integration tests (minutes)
  - Security scans
  - Code style checks
       ↓
All pass? → Code merges to main branch
Any fail? → Developer notified; merge blocked

Continuous Delivery (CD)

Continuous Delivery extends CI by automatically preparing every passing build for deployment. The build is packaged, versioned, and pushed to a staging environment for final validation. A human approves the final production release.

Continuous Deployment

Continuous Deployment goes one step further: every passing build deploys to production automatically, without human approval. This requires high confidence in automated tests and robust rollback mechanisms.

Deployment Strategies

Blue-Green Deployment

Two identical production environments exist side by side — Blue (current live) and Green (new version). The new release deploys to Green first. After validation, all traffic switches from Blue to Green in seconds. If problems appear, traffic switches back to Blue instantly.

Blue-Green Deployment:
-----------------------
BEFORE RELEASE:
  Blue: v1.5 (100% of traffic)
  Green: v1.6 (0% of traffic)

RELEASE STEP 1: Deploy v1.6 to Green. Test it.

AFTER RELEASE:
  Blue: v1.5 (0% of traffic)
  Green: v1.6 (100% of traffic)

PROBLEM FOUND:
  Green: v1.6 (0% of traffic) ← traffic removed instantly
  Blue: v1.5 (100% of traffic) ← old version restored

Canary Deployment

A canary deployment sends a small percentage of real production traffic to the new version first — like a miner's canary in a coal mine that detects danger before the miners enter. The SRE team monitors the canary for errors and performance degradation. If everything looks healthy, they gradually increase the traffic percentage until 100 percent of users are on the new version.

Canary Rollout Stages:
-----------------------
Stage 1:  1% of users → v2.0      Monitor 30 minutes
Stage 2:  10% of users → v2.0     Monitor 1 hour
Stage 3:  25% of users → v2.0     Monitor 1 hour
Stage 4:  50% of users → v2.0     Monitor 2 hours
Stage 5:  100% of users → v2.0    Rollout complete ✅

If SLO degrades at any stage → Rollback to previous version instantly

Feature Flags

Feature flags are code switches that enable or disable a feature without redeploying. A new feature ships in the codebase but hidden behind a flag. The flag is turned on for specific users, regions, or percentages of traffic. If the feature causes problems, a team member flips the flag off — no deployment needed.

Feature Flag in Action:
------------------------
Code is deployed with new payment feature behind a flag.

Flag OFF: All users see the old payment flow.
Flag ON for 5% of users: New payment flow tested on a small audience.
Flag ON for 50% of users: Expanded rollout after 5% looks healthy.
Flag ON for everyone: Full release. No redeployment required.

Problem found at 50%: Flip flag back to OFF. Instant mitigation.

Rollback Strategy

Every deployment must have a defined rollback plan before it is executed. Rollback is the process of reverting to the previous stable version when a deployment causes problems. The faster the rollback, the less error budget is consumed during an incident.

What Makes Rollback Fast

Automation: the rollback command runs in one step, not twenty manual steps.
Database compatibility: new code must handle the existing database schema; rollback must handle the new schema.
Immutable artifacts: deployments use versioned, pre-built packages, not live code changes.

Deployment Gates and Quality Checks

A deployment gate is an automated check that must pass before the deployment proceeds to the next stage. Common gates include:

All automated tests pass.
No critical security vulnerabilities in the build.
Error rate in the canary does not exceed a threshold.
Latency p99 in the canary does not exceed a threshold.
Relevant dashboard reviewed and healthy.

Key Points

Most production incidents trace back to a recent change — release engineering reduces that risk.
CI/CD automates testing and delivery, making releases faster and more reliable.
Canary deployments expose new code to a small fraction of users before full rollout.
Feature flags decouple code deployment from feature activation for instant control.
Every deployment needs a pre-defined, tested rollback plan before it is executed.

Previous lessons

Back to courses

Next lessons