Grafana Kubernetes Monitoring
Kubernetes moves containers between machines, restarts them on failure, and scales them up and down. That flexibility makes manual checking impossible. Monitoring gives you a live picture of nodes, namespaces, and pods, and shows trouble before users notice. This project installs a complete monitoring stack with a single Helm command.
The Airport Control Tower Analogy
An airport control tower tracks every plane: where it is, how fast it moves, and whether it needs help. Controllers do not fly the planes. They watch the whole sky and send alerts when something looks wrong. Grafana plays the control tower for a cluster. Pods are the planes, nodes are the runways, and namespaces are the airlines.
+----------------------- Cluster -----------------------+
| Node 1 Node 2 Node 3 |
| [pod][pod] [pod][pod][pod] [pod] |
+---+----------------------+-------------------+--------+
| metrics | |
+----------+-----------+-------------------+
v
Prometheus --> Grafana --> Alerts
Meet the Components
| Component | Role |
|---|---|
| Prometheus | Collects and stores metrics |
| Node Exporter | Reports hardware numbers for each node |
| kube-state-metrics | Reports the state of Kubernetes objects, such as pod phases and deployment counts |
| cAdvisor (built into the kubelet) | Reports container CPU and memory use |
| Grafana | Draws dashboards |
Install With Helm
The kube-prometheus-stack chart bundles all components, plus a set of ready-made dashboards and alert rules.
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install monitoring prometheus-community/kube-prometheus-stack \
--namespace monitoring --create-namespace
Check progress with the command below. Wait until every pod shows Running.
kubectl get pods -n monitoring
Open Grafana
kubectl port-forward -n monitoring svc/monitoring-grafana 3000:80
Open http://localhost:3000. The chart creates the admin user with a generated password. Read it from the Kubernetes secret, as the chart notes explain, and change it right away. Prometheus is already connected as a data source.
Explore the Built-in Dashboards
Open Dashboards and look for the folder created by the chart. Useful dashboards include views for the whole cluster, individual nodes, namespaces, workloads, and pods. Click through them to see typical layouts and queries.
Key Metrics for Your Own Panels
Pods by Phase
sum by (phase) (kube_pod_status_phase)
A pie chart or bar chart shows how many pods run, wait, or fail.
CPU by Namespace
sum by (namespace) (
rate(container_cpu_usage_seconds_total{container!=""}[$__rate_interval])
)
Memory by Namespace
sum by (namespace) (container_memory_working_set_bytes{container!=""})
Container Restarts
sum by (namespace, pod) (
increase(kube_pod_container_status_restarts_total[1h])
)
A table sorted by value reveals pods that crash repeatedly. Pods that restart many times in an hour deserve attention at once.
Node Readiness
sum(kube_node_status_condition{condition="Ready", status="true"})
Add Variables
Create a variable named namespace with the query label_values(kube_pod_info, namespace). Filter panels with namespace="$namespace". Teams open the dashboard and see only their own workloads.
Understand Requests and Limits
Every container can declare a CPU and memory request, which is the amount the scheduler reserves, and a limit, which is the ceiling. Comparing real usage with requests reveals waste and risk. Usage far below the request means the team reserves capacity it never uses. When usage nears the limit, the container may slow down or restart.
sum by (namespace) (
rate(container_cpu_usage_seconds_total{container!=""}[$__rate_interval])
) / sum by (namespace) (
kube_pod_container_resource_requests{resource="cpu"}
)
A result near 1 means usage matches the reservation. Expect a result of 0.1 from a namespace that reserves ten times more CPU than it needs.
Follow a Pod Problem
- The restart table shows a pod with twelve restarts in an hour.
- The memory panel shows the same pod climbing to its limit before each restart.
- The team raises the memory limit or fixes the leak in the application.
- The restart count drops to zero, and the chart confirms the fix.
Layout Suggestion
+-------+-------+-------+-------+
| Nodes | Pods | Failed| Restarts| <-- stat row
| Ready | Running| Pods | (1h) |
+-------+-------+-------+-------+
| CPU by namespace | Memory by namespace |
+------------------+---------------------+
| Pods by phase | Top restarting pods |
+------------------+---------------------+
Alert Ideas
| Alert | Signal |
|---|---|
| Pod crash looping | Restarts above 3 in 15 minutes |
| Pod stuck pending | Phase Pending for more than 10 minutes |
| Node not ready | Ready condition false for 5 minutes |
| Memory pressure | Namespace memory near its quota |
The chart already ships many rules like these. Review them and tune the routing with notification policies.
Good Practices
- Set resource requests and limits on every workload, so usage charts have meaning.
- Expose Grafana through an ingress with HTTPS and single sign-on.
- Use persistent volumes for Prometheus and Grafana, so data survives pod restarts.
- Limit retention and series count to control storage cost.