Kubernetes Monitoring and Logging with Prometheus and Grafana
Running workloads in Kubernetes without monitoring is like driving with no dashboard — the engine might be overheating and you would not know until the car stops. Prometheus collects and stores metrics. Grafana turns those metrics into readable dashboards and alerts. Together, they form the most widely used observability stack for Kubernetes.
The Two Pillars: Metrics vs Logs
Before diving in, it helps to understand the difference between what Prometheus handles and what a logging stack handles.
┌────────────────────────────────────────────────────────┐ │ Observability in Kubernetes │ │ │ │ METRICS (Prometheus) LOGS (Loki / EFK) │ │ ───────────────── ────────────────── │ │ Numbers over time Text events over time │ │ │ │ "CPU is at 85%" "Error: DB conn timeout" │ │ "500 requests/sec" "User 42 logged in" │ │ "Memory growing for 10m" "Pod restarted: OOMKilled" │ │ │ │ Good for: alerts, Good for: debugging, │ │ capacity, SLOs tracing incidents │ └────────────────────────────────────────────────────────┘
This page covers metrics with Prometheus and Grafana first, then covers log aggregation with Loki — all three tools integrate closely and are often deployed together.
How Prometheus Works
Prometheus is a pull-based metrics system. Instead of applications pushing data to a central server, Prometheus scrapes HTTP endpoints that expose metrics in a plain-text format. These endpoints are called targets.
┌────────────────────────────────────────────────────────────┐ │ Prometheus Data Flow │ │ │ │ Application Pods Prometheus Server │ │ ───────────────── ───────────────── │ │ │ │ [Pod A] /metrics ◄──scrape── Prometheus │ │ [Pod B] /metrics ◄──scrape── every 15s │ │ [Node ] /metrics ◄──scrape── │ │ │ │ │ ▼ │ │ Time-Series DB │ │ (on-disk TSDB) │ │ │ │ │ ▼ │ │ Query with PromQL │ │ Feed to Grafana │ └────────────────────────────────────────────────────────────┘
The Metrics Format
A Prometheus metrics endpoint returns plain text. Each line is a metric with its name, optional labels, and a current value:
# HELP http_requests_total Total number of HTTP requests
# TYPE http_requests_total counter
http_requests_total{method="GET",status="200"} 1027
http_requests_total{method="POST",status="500"} 14
Labels are key-value pairs that let you filter and group the same metric across many dimensions — by HTTP method, response status, namespace, pod name, and so on.
Deploying Prometheus with kube-prometheus-stack
The recommended way to deploy Prometheus in Kubernetes is the kube-prometheus-stack Helm chart. It installs Prometheus, Grafana, Alertmanager, and a set of pre-built dashboards and alert rules in one command.
# Add the Helm repo helm repo add prometheus-community \ https://prometheus-community.github.io/helm-charts helm repo update # Install into a dedicated namespace helm install kube-prometheus-stack \ prometheus-community/kube-prometheus-stack \ --namespace monitoring \ --create-namespace
After installation, the following components run in the monitoring namespace:
- Prometheus — Scrapes and stores metrics.
- Alertmanager — Receives alerts from Prometheus and routes them to Slack, PagerDuty, email, etc.
- Grafana — Visualises metrics on dashboards.
- kube-state-metrics — Exposes Kubernetes object state (Deployment replicas, Pod status, etc.) as metrics.
- node-exporter — A DaemonSet that exports CPU, memory, and disk metrics from every node.
Service Discovery: How Prometheus Finds Pods
In a dynamic environment like Kubernetes, Pods come and go. Prometheus does not use a static list of IP addresses. It uses Kubernetes service discovery to automatically find what to scrape.
ServiceMonitor
A ServiceMonitor is a custom resource (from the kube-prometheus-stack) that tells Prometheus which Services to scrape and on which port and path.
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: my-app-monitor
namespace: production
labels:
release: kube-prometheus-stack # Prometheus picks this up
spec:
selector:
matchLabels:
app: my-app # Matches the Service label
endpoints:
- port: http-metrics # The port name in the Service
path: /metrics
interval: 30s
Prometheus finds the ServiceMonitor, reads its selector, finds matching Services, resolves their Pod endpoints, and starts scraping. New Pods created by a scaling event are discovered automatically within one scrape interval.
PromQL: Querying Your Metrics
Prometheus Query Language (PromQL) is how you extract meaning from raw metrics. It reads like a filter pipeline — select a metric, apply label filters, transform with functions.
Instant Query: CPU Usage per Pod
rate(container_cpu_usage_seconds_total
{namespace="production"}[5m])
This calculates the per-second CPU usage rate over the last 5 minutes, filtered to the production namespace. The result is one time series per container.
Memory Usage as a Percentage of Limit
container_memory_working_set_bytes
{namespace="production"}
/
container_spec_memory_limit_bytes
{namespace="production"}
* 100
Request Error Rate
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))
* 100
This gives the percentage of HTTP requests returning 5xx status codes over the last 5 minutes. If this number climbs above 1%, an alert fires.
Alertmanager: From Metric to Notification
A PrometheusRule resource defines alert conditions. When the condition is true for longer than a specified duration, Prometheus sends the alert to Alertmanager, which routes it to the right team.
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: app-alerts
namespace: production
spec:
groups:
- name: app.rules
rules:
- alert: HighErrorRate
expr: |
sum(rate(http_requests_total{status=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m])) * 100 > 5
for: 5m
labels:
severity: critical
annotations:
summary: "High HTTP error rate detected"
description: "Error rate is {{ $value }}%"
Alertmanager receives this alert and can route it based on labels. A severity: critical alert goes to PagerDuty and wakes someone up. A severity: warning alert posts to a Slack channel for morning review.
Grafana: Building Dashboards
Grafana connects to Prometheus as a data source and lets you build visual dashboards from PromQL queries. The kube-prometheus-stack pre-installs a full set of dashboards covering:
- Node CPU, memory, disk, and network usage.
- Kubernetes cluster resource requests vs limits.
- Pod and container-level metrics.
- Deployment and StatefulSet health.
- Kubernetes API server and etcd performance.
Accessing Grafana
# Port-forward to access Grafana locally kubectl port-forward svc/kube-prometheus-stack-grafana \ 3000:80 -n monitoring # Default credentials (change immediately in production) # Username: admin # Password: prom-operator
Adding a Custom Panel
Inside Grafana, create a new dashboard, add a panel, select Prometheus as the data source, and enter a PromQL query. Grafana renders the query result as a line graph, bar chart, stat, or gauge. You arrange panels into a layout that tells the story of your application's health at a glance.
The Four Golden Signals
Every production dashboard should track at least these four signals. They come from Google's Site Reliability Engineering practices and apply to any service:
┌──────────────────────────────────────────────────────────┐ │ Four Golden Signals │ │ │ │ LATENCY How long requests take │ │ Metric: p50, p95, p99 response time │ │ │ │ TRAFFIC How much load the system handles │ │ Metric: requests per second │ │ │ │ ERRORS How often requests fail │ │ Metric: error rate (4xx + 5xx %) │ │ │ │ SATURATION How full the system is │ │ Metric: CPU%, memory%, queue depth │ └──────────────────────────────────────────────────────────┘
Logging in Kubernetes
Container logs are ephemeral. When a Pod restarts or gets rescheduled, its logs disappear. You need a log aggregation system that collects, stores, and indexes logs so you can query them after an incident.
The PLG Stack: Promtail + Loki + Grafana
Loki is the log storage and query system built by the Grafana team. It is designed specifically for Kubernetes log aggregation and integrates directly into Grafana alongside Prometheus dashboards.
┌─────────────────────────────────────────────────────────────┐ │ PLG Log Pipeline │ │ │ │ Pods Promtail Loki │ │ ──── ──────── ──── │ │ stdout/stderr (DaemonSet) Stores logs │ │ written to Tails log files Indexes labels │ │ /var/log/ Adds labels Accepts queries │ │ containers/ (namespace, pod, from Grafana │ │ container) ─────────── │ │ ─────────► LogQL queries │ │ Pushes to Loki in Grafana UI │ └─────────────────────────────────────────────────────────────┘
Installing Loki with Helm
helm repo add grafana https://grafana.github.io/helm-charts helm repo update helm install loki grafana/loki-stack \ --namespace monitoring \ --set grafana.enabled=false \ # Grafana already installed --set promtail.enabled=true
Querying Logs with LogQL
LogQL is Loki's query language. It looks similar to PromQL but operates on log streams.
# All error logs from the production namespace
{namespace="production"} |= "error"
# Count error log lines per minute
sum(rate({namespace="production"} |= "error" [1m]))
# Filter by pod name and search for a specific exception
{namespace="production", pod=~"api-.*"} |= "NullPointerException"
Grafana shows Prometheus metrics and Loki logs on the same dashboard. You can click a spike on a CPU graph and immediately open the logs from that Pod at that exact time — without switching tools.
The EFK Alternative
The other common logging stack is Elasticsearch + Fluentd/Fluent Bit + Kibana (EFK). Fluentd or Fluent Bit runs as a DaemonSet and forwards container logs to Elasticsearch. Kibana provides the query UI.
EFK is more powerful for full-text search at massive scale but requires significantly more resources — Elasticsearch clusters are heavyweight. The PLG (Promtail + Loki + Grafana) stack is lighter and integrates more cleanly with a Prometheus-based monitoring setup.
Monitoring Best Practices for Kubernetes
Use Resource Requests and Limits — Then Monitor Them
Set CPU and memory requests and limits on every container. Then build dashboards that show actual usage vs requests. This reveals over-provisioned services wasting cluster resources and under-provisioned services at risk of OOMKill.
Alert on Symptoms, Not Causes
Alert when users are affected (high error rate, slow response time), not on every infrastructure metric. An alert on "CPU above 80%" fires constantly. An alert on "error rate above 1% for 5 minutes" fires when something is actually broken.
Set Up HPA Metrics
Horizontal Pod Autoscaler can scale deployments based on custom Prometheus metrics using the metrics-server or KEDA. A queue-consumer service can scale based on queue depth, for example.
Persist Prometheus Data
By default, Prometheus stores data in an emptyDir that disappears when the Pod restarts. In production, configure a PersistentVolumeClaim for the Prometheus data directory so historical metrics survive Pod restarts.
# In kube-prometheus-stack values.yaml
prometheus:
prometheusSpec:
storageSpec:
volumeClaimTemplate:
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 50Gi
Key Points
- Prometheus scrapes metrics from HTTP endpoints and stores them as time-series data.
- Deploy the full stack quickly with the
kube-prometheus-stackHelm chart. - ServiceMonitors tell Prometheus which Services to scrape, enabling automatic discovery of new Pods.
- PromQL queries metrics; PrometheusRules define alert conditions; Alertmanager routes alerts to the right teams.
- Grafana visualises Prometheus metrics and Loki logs on the same dashboards.
- Track the four golden signals: Latency, Traffic, Errors, and Saturation for every production service.
- Use Loki for lightweight log aggregation that integrates with Grafana; use EFK for large-scale full-text search requirements.
- Persist Prometheus storage and alert on user-facing symptoms rather than infrastructure thresholds.