Kubernetes Troubleshooting

Every Kubernetes user meets a pod that refuses to start. Calm, repeatable steps solve almost every case. A good troubleshooter works like a doctor: check the symptoms, read the evidence, form a theory, and test it.

The Troubleshooting Loop

 Observe ──▶ Describe ──▶ Read logs ──▶ Form theory ──▶ Fix ──▶ Verify
    ▲                                                            │
    └──────────────── repeat if still broken ────────────────────┘

The Four Core Commands

kubectl get pods                         # what is the status?
kubectl describe pod  NAME               # what do the events say?
kubectl logs NAME                        # what did the app print?
kubectl logs NAME --previous             # what did the crashed container print?

Run describe first. The Events section at the bottom explains most failures in plain words.

Pod Status Guide

Status	Meaning	Typical cause
Pending	Scheduler has not placed the pod	Not enough CPU or memory, taints, unbound volume
ContainerCreating	Node prepares the container	Slow image pull, missing Secret or ConfigMap, network plugin issue
ImagePullBackOff	Image download failed	Wrong image name, missing tag, bad registry credentials
CrashLoopBackOff	Container starts and exits repeatedly	Application error, wrong command, failed probe
OOMKilled	Container exceeded its memory limit	Limit too low or memory leak
Evicted	Node removed the pod	Node ran out of disk or memory
Running but not Ready	Readiness probe fails	Application not healthy or wrong probe path

Diagram: Decision Tree for a Broken Pod

 Pod not working
      │
      ├── Pending ─────────────▶ describe: look for "FailedScheduling"
      │                            ├─ Insufficient cpu/memory ─▶ add nodes or lower requests
      │                            ├─ Taint not tolerated ─────▶ add toleration
      │                            └─ PVC unbound ─────────────▶ check StorageClass
      │
      ├── ImagePullBackOff ────▶ check image name, tag, pull secret
      │
      ├── CrashLoopBackOff ────▶ logs --previous
      │                            ├─ stack trace ─▶ fix application
      │                            └─ exit 137 ────▶ memory limit (OOMKilled)
      │
      └── Running, not Ready ──▶ check readiness probe and endpoints

Case 1: ImagePullBackOff

kubectl describe pod web-7d9f
...
Warning  Failed   Failed to pull image "nginx:1.99": not found

The tag does not exist. Correct the tag in the deployment and apply the change.

Case 2: CrashLoopBackOff

kubectl logs api-5c8d --previous
Error: environment variable DB_HOST is not set

The container exits on startup because a setting is missing. Add the variable through a ConfigMap and restart the rollout.

Case 3: Service Returns No Response

Walk the path from the service backward.

kubectl get service shop-service
kubectl get endpoints shop-service
kubectl get pods --show-labels

Empty endpoints mean the service selector matches no ready pods. Compare the selector with pod labels and check readiness.

 Client ──▶ Service ──▶ Endpoints ──▶ Pod ──▶ Container port
              ▲            ▲           ▲          ▲
          selector?    any listed?  Ready?    port correct?

Case 4: Pending Pod

kubectl describe pod batch-job-1
Warning  FailedScheduling  0/3 nodes available: 3 Insufficient memory.

No node has enough free memory for the request. Reduce the request, free resources, or add a node.

Look at the Whole Cluster

kubectl get events --sort-by=.metadata.creationTimestamp
kubectl get nodes
kubectl describe node worker-1
kubectl top nodes
kubectl top pods
kubectl get pods -n kube-system

Node conditions such as MemoryPressure and DiskPressure reveal sick machines. System pods in kube-system must run before any application can behave.

Get Inside the Pod

kubectl exec -it web-7d9f -- sh
kubectl exec web-7d9f -- env
kubectl port-forward pod/web-7d9f 8080:80

Minimal images often ship without a shell. Attach a temporary debug container to cover that gap:

kubectl debug -it web-7d9f --image=busybox:1.36 --target=web

The debug container shares the process space of the target container, so you can inspect files and network without changing the original image.

Testing Network Connectivity

kubectl run tmp --rm -it --image=busybox:1.36 -- sh
/ # nslookup shop-service
/ # wget -qO- http://shop-service:80

A failed nslookup points to DNS. A successful lookup with a failed request points to the service, endpoints, or network policy.

Reading Events Like a Story

Events record what the cluster tried and what went wrong. Read them from top to bottom, and each line explains one step of the pod journey: the scheduler picked a node, the kubelet pulled an image, the container started, a probe failed, the container restarted. The failing step names the problem area. A scheduling failure points to capacity. An image failure points to the registry. A probe failure points to the application.

Events expire after about an hour, so check them soon after a failure. A central logging and event system keeps the record longer.

Exit Codes Worth Knowing

Exit code	Meaning
0	Normal exit
1	Application error
137	Killed, often by the out-of-memory killer
139	Segmentation fault
143	Stopped by a termination signal

The exit code appears under Last State in the output of kubectl describe pod.

Good Habits

• Change one thing at a time and re-test after each change.
• Keep labels, probes, and resource settings consistent across manifests.
• Send logs to a central system so evidence survives pod deletion.
• Write down fixes in a team runbook.

Key Points

• Start with get, describe, and logs.
• Pod status names point directly at the likely cause.
• Follow traffic step by step: service, endpoints, pod, container.
• kubectl debug opens a shell even inside minimal images.