Grafana Performance Tuning

A dashboard that takes thirty seconds to load defeats its own purpose. People stop opening it, and trouble goes unnoticed. Slow dashboards usually trace back to a few common causes. Fixing those causes brings the load time down to a couple of seconds and protects your data sources from needless strain.

The Kitchen Rush Analogy

A restaurant kitchen slows down when twenty orders arrive at once, each asking for complex dishes. Dashboards behave the same way. Every panel sends one or more queries, and every query asks a data source to do work. More panels, wider time ranges, and heavier queries create a rush that the data source struggles to serve.

Load time = number of queries x cost of each query

Dashboard A:  8 panels  x light queries   -->  1 second
Dashboard B: 40 panels  x heavy queries   --> 30 seconds

Measure Before You Fix

Guessing wastes time. Measure first and fix the biggest problem.

Query Inspector

Open a panel menu, choose Inspect, and select Query. The inspector lists each request with its duration and the amount of data returned. Slow queries stand out at once.

Browser Developer Tools

Open the network tab of your browser and reload the dashboard. Long bars point to slow requests, and the number of bars shows how many queries run in parallel.

Panel Load Times

The inspect view shows how long rendering takes as well. A fast query with a slow render signals too many series or too many data points.

Reduce the Number of Queries

Trim Panels

Every panel must justify its place. Remove panels nobody reads, and split a giant dashboard into focused ones linked together.

Collapse Rows

Grafana skips queries for panels in collapsed rows. Collapse detail sections and let viewers open them on demand.

Combine Queries

Several panels often ask for the same data. Use the Dashboard data source to reuse the result of one query across many panels.

Make Each Query Cheaper

Control Data Points

A line chart cannot show more points than the pixels on the screen. Lower Max data points in the query options, or raise the Min interval, to cut the volume of returned data.

Before:  1-second interval, 24 hours  --> 86,400 points per series
After:   1-minute interval, 24 hours  -->  1,440 points per series

Narrow the Time Range

Default to a short range such as the last hour or six hours. Viewers can widen the range when needed.

Filter Early

Restrict queries with labels or WHERE conditions so the data source scans less. Avoid queries that return thousands of series only to hide most of them in the panel.

Precompute Heavy Calculations

Prometheus recording rules calculate expensive expressions on a schedule and store the result as a new metric. Panels read the stored answer and skip the repeated work.

Without recording rule:  Panel --> heavy calculation over millions of samples
With recording rule:     Panel --> reads a ready-made metric

Use Variables Wisely

The All option in a multi-value variable expands into a long list of items, and a panel may draw hundreds of lines. Limit the options, set a sensible default, or disable All for large lists. Chained variables that refresh on every time range change add extra queries. Set the variable refresh to On dashboard load for lists that rarely change.

Tune the Refresh Rate

A dashboard that refreshes every five seconds runs all its queries twelve times a minute, for every viewer. Match the refresh rate to the data. Metrics collected every minute need no faster refresh. Set a minimum refresh interval in the Grafana server settings to stop users from choosing extreme rates.

Choose Panels With Care

• Tables with thousands of rows render slowly. Limit the rows and sort by importance.
• Time series panels with hundreds of lines slow the browser. Show the top ten series.
• Logs panels with huge result sets drain memory. Set a line limit.

Tune the Grafana Server

Grafana stores users, dashboards, and settings in its own database. The default is SQLite, which works well for small setups. A busy installation or a high-availability cluster should use PostgreSQL or MySQL.

Small team            Large team or cluster
+---------+           +---------+   +---------+
| Grafana |           | Grafana |   | Grafana |
| SQLite  |           +----+----+   +----+----+
+---------+                |             |
                       +---v-------------v---+
                       | PostgreSQL / MySQL  |
                       +---------------------+

Give the server enough memory and CPU, and place it close to its data sources on the network. Long network distances add delay to every query.

Tune the Data Sources

Grafana can only be as fast as the systems behind it. Index SQL columns that queries filter on. Keep Prometheus label values under control, because a label with millions of unique values slows every query. Set query timeouts on data sources so one runaway query cannot hold up the rest.

Performance Checklist

Check	Target
Panels per dashboard	Twenty or fewer
Default time range	One to six hours
Refresh rate	Matches data collection pace
Detail rows	Collapsed
Heavy expressions	Recorded in advance
Series per panel	Ten to twenty
Grafana database	PostgreSQL or MySQL for large setups