SRE Kubernetes Reliability
Kubernetes runs containers across many machines and restarts them when they fail. The platform offers strong tools for reliability, and wrong settings can still cause outages. Knowing the key features helps SRE teams build services that survive failures.
Kubernetes in Simple Words
Think of a shipping port. Containers arrive in many sizes. Cranes place each one on a ship, and a control tower tracks every container. If a ship breaks down, the tower moves the containers to another ship. Kubernetes acts as the control tower for software containers.
Building Blocks
| Object | Plain Meaning |
|---|---|
| Pod | Smallest unit, holds one or more containers |
| Deployment | Keeps a chosen number of identical pods running |
| Service | Gives pods a stable address and balances traffic |
| Node | A machine that runs pods |
| Namespace | A separate area for a team or environment |
Cluster Picture
[Service: shop-web]
/ | \
[Pod 1] [Pod 2] [Pod 3]
│ │ │
[Node A] [Node B] [Node C]
If Node B fails, Kubernetes starts a replacement pod on a healthy node. The Service sends traffic only to pods that report ready.
Reliability Settings That Matter
Replicas
Run at least three replicas for important services, spread across different nodes and zones. One replica turns every restart into an outage.
Liveness and Readiness Probes
A liveness probe restarts a stuck container. A readiness probe keeps traffic away from a container that is not ready. A startup probe gives slow programs extra time to begin.
Resource Requests and Limits
A request reserves the minimum CPU and memory a pod needs. A limit caps the maximum. Missing requests lead to crowded nodes and unpredictable slowdowns. Tight memory limits kill pods that exceed them.
Rolling Updates
A rolling update replaces pods a few at a time. Settings such as maxUnavailable control how many pods may disappear during the change, so capacity never collapses.
Pod Disruption Budgets
A pod disruption budget tells Kubernetes the minimum number of pods that must stay up during planned work such as node upgrades.
Autoscaling
The Horizontal Pod Autoscaler adds pods when load rises and removes them when load falls. Set sensible minimum and maximum counts so scaling stays safe and affordable.
Common Failure Messages
| Message | Meaning | First Check |
|---|---|---|
| CrashLoopBackOff | Container starts and fails repeatedly | Read the container logs |
| OOMKilled | Container exceeded its memory limit | Raise the limit or fix a memory leak |
| ImagePullBackOff | Image cannot download | Check image name and registry access |
| Pending | No node has room for the pod | Check requests and node capacity |
Example: Safe Node Upgrade
A team upgrades ten nodes. Each service has three replicas and a pod disruption budget that allows one pod to be down. Kubernetes drains one node at a time, moves its pods, and waits for readiness before continuing. Users notice nothing, and the upgrade finishes in an hour.
Useful Starter Commands
kubectl get pods kubectl describe pod shop-web-1 kubectl logs shop-web-1 kubectl rollout status deployment/shop-web kubectl rollout undo deployment/shop-web
Reliability Checklist
- Three or more replicas across zones
- Readiness and liveness probes on every service
- Requests and limits for CPU and memory
- Pod disruption budgets for critical services
- Monitoring for restarts, pending pods, and node pressure
Common Mistakes
- Copying a liveness probe that is too strict, which restarts healthy pods under load.
- Running all replicas on a single node.
- Leaving containers with no memory limit.
- Treating Kubernetes as a replacement for good application design.
Key Takeaways
Kubernetes supplies self-healing, scaling, and safe updates. Reliable setups use multiple replicas, honest probes, clear resource settings, and disruption budgets.