Kubernetes Backup and Upgrades
Clusters run for years, and two chores define their long-term health: protecting data against loss and moving to new Kubernetes versions on schedule. Both tasks reward preparation. A tested backup turns a disaster into an inconvenience, and a rehearsed upgrade turns a risky night into routine work.
The House Maintenance Analogy
Homeowners keep copies of important papers in a safe place and schedule regular repairs before small cracks become big leaks. Backups play the role of the paper copies. Upgrades play the role of repairs, since each Kubernetes release brings security fixes and new features.
What You Need to Back Up
| Item | Where it lives | Backup method |
|---|---|---|
| Cluster state (all objects) | etcd database | etcd snapshot |
| Application manifests | Git repository | Git history |
| Persistent volume data | Disks and storage systems | Volume snapshots or Velero |
| Certificates and keys | /etc/kubernetes/pki | Secure file copy |
Backing Up etcd
etcd stores every object in the cluster: deployments, secrets, config maps, and more. A snapshot captures that state at one moment.
ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-snapshot.db \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key
etcdutl snapshot status /backup/etcd-snapshot.db
Automate the command with a scheduled job and copy snapshots to storage outside the cluster. Encrypt them, because they contain Secret data. Managed services such as EKS, GKE, and AKS back up the control plane for you.
Restoring etcd
etcdutl snapshot restore /backup/etcd-snapshot.db \
--data-dir=/var/lib/etcd-restored
Point the etcd static pod manifest at the restored data directory, and the control plane returns to the saved state. Practice this restore in a test cluster before an emergency forces it.
Application-Level Backups with Velero
Velero backs up Kubernetes objects and persistent volumes together. It works on managed clusters where etcd stays hidden, and it moves workloads between clusters.
velero backup create shop-backup --include-namespaces shop
velero backup get
velero restore create --from-backup shop-backup
velero schedule create daily-shop --schedule="0 2 * * *" \
--include-namespaces shop
Diagram: Backup Layers
┌─────────────────────────────────────────────┐
│ Git manifests and Helm values │
├─────────────────────────────────────────────┤
│ etcd snapshot full cluster state │
├─────────────────────────────────────────────┤
│ Velero namespaces + volume data │
├─────────────────────────────────────────────┤
│ Database dump application-aware copies │
└─────────────────────────────────────────────┘
│ all copied to off-cluster storage
▼
Cloud bucket in another region
Version Rules You Must Know
- Kubernetes releases a new minor version (such as 1.31 to 1.32) about three times a year.
- The project supports roughly the three most recent minor versions with patches.
- Upgrade one minor version at a time. Skipping versions is not supported.
- The API server upgrades first. Kubelets may trail the API server by a few minor versions, and the exact skew policy appears in the release documentation.
- kubectl stays within one minor version of the API server.
Preparing for an Upgrade
- Read the release notes for removed APIs and behavior changes.
- Scan manifests for deprecated API versions.
- Take an etcd snapshot and a Velero backup.
- Upgrade a staging cluster first and run application tests.
- Confirm PodDisruptionBudgets exist so draining nodes keeps services alive.
Upgrading a kubeadm Cluster
Control Plane Node
sudo apt-get update
sudo apt-get install -y kubeadm=1.32.0-1.1
sudo kubeadm upgrade plan
sudo kubeadm upgrade apply v1.32.0
kubectl drain control-1 --ignore-daemonsets
sudo apt-get install -y kubelet=1.32.0-1.1 kubectl=1.32.0-1.1
sudo systemctl restart kubelet
kubectl uncordon control-1
Worker Nodes, One at a Time
kubectl drain worker-1 --ignore-daemonsets --delete-emptydir-data
# on worker-1:
sudo apt-get install -y kubeadm=1.32.0-1.1
sudo kubeadm upgrade node
sudo apt-get install -y kubelet=1.32.0-1.1
sudo systemctl restart kubelet
# back on the control plane:
kubectl uncordon worker-1
Package version strings differ by repository, so confirm the exact version with apt-cache madison kubeadm.
Diagram: Rolling Node Upgrade
Step 1: cordon and drain worker-1 ──▶ pods move to worker-2, worker-3 Step 2: upgrade kubelet on worker-1 Step 3: uncordon worker-1 ──▶ node accepts pods again Step 4: repeat for worker-2, then worker-3
Managed Cluster Upgrades
EKS, GKE, and AKS upgrade the control plane with one command or a console click. Node pools upgrade through surge or rolling settings. Many providers offer release channels that apply upgrades on a schedule.
Verification After Upgrade
kubectl version
kubectl get nodes
kubectl get pods -A
kubectl get events -A --sort-by=.metadata.creationTimestamp
Confirm that every node reports the new version, system pods run, and test traffic reaches key services.
Key Points
- etcd snapshots protect cluster state, and Velero protects applications with their data.
- Store backups outside the cluster and test restores.
- Upgrade one minor version at a time, control plane first.
- Drain and uncordon nodes to keep applications online during upgrades.