Grafana Alert State History
An alert that fired at 3 AM and cleared by 3:10 AM leaves no trace on a normal dashboard. Teams still need to answer questions the following morning. What fired? How long did it last? Did it repeat? State history records every change of alert state, so you can review the past with confidence.
The Flight Recorder Analogy
Aircraft carry a flight recorder that logs every event during a flight. Investigators read the log after the landing to learn what happened and when. Alert state history serves as the flight recorder for your alert rules. Each change of state gets a timestamp and the values that caused it.
Alert rule: High CPU on web-02
10:00 Normal
10:12 Pending (CPU = 88)
10:17 Alerting (CPU = 91) --> notification sent
10:31 Normal (CPU = 62) --> resolved message
Review the Alert States
| State | Meaning |
|---|---|
| Normal | The condition is false |
| Pending | The condition is true, and the pending timer runs |
| Alerting | The condition held for the full pending period |
| NoData | The query returned no data |
| Error | The query or an expression failed |
| Recovering | The condition cleared, and Grafana waits to confirm recovery on versions that support it |
Open the History
Open Alerting and select Alert rules. Click a rule name to open its detail page. Look for the History tab or the Show state history option, depending on your version. A timeline shows each state change with its time and the labels of the alert instance.
State timeline for rule "High CPU"
web-01 [Normal ][Normal ][Normal ][Normal ]
web-02 [Normal][Pending][ALERTING ][Normal ]
web-03 [Normal ][Normal][Pend][Normal][Normal ]
10:00 10:15 10:30 10:45
Each row represents one alert instance. A single rule can create many instances, one for each server or each label combination.
Where Grafana Stores History
Grafana writes state changes to a backend. Older setups use the Grafana database through annotations. Newer setups can send the records to Loki, which handles large volumes better and keeps history for longer. Ask your administrator which backend your server uses, since that choice affects how long records last.
Alert rule evaluates
|
v
State changes --> History backend (database or Loki)
|
v
Alert detail page, State timeline panel, Annotations
Show History on a Dashboard
Alert Annotations
Dashboards display alert state changes as annotations on time series panels. A red marker appears when an alert fires, and a green marker appears when it resolves. Hover over the marker to read the rule name. This overlay helps readers connect a metric spike to its alert.
Alert List Panel
The Alert list panel shows current alerts or a stream of recent state changes. Add it to a team dashboard to display what fired in the last day.
State Timeline Panel
Query the history backend and feed the result to a State timeline panel. The panel draws coloured bars for each state, and long red bars reveal unstable services at once.
Questions the History Answers
- How many times did this rule fire last week?
- How long did each incident last?
- Did the alert flap between Normal and Alerting several times in an hour?
- Which instances caused the most alerts?
- Did a notification go out at the right time?
Spot Flapping Alerts
A flapping alert switches state again and again around its threshold. It floods channels and teaches people to ignore messages.
CPU %
90 | /\ /\ /\ /\
85 |- /- \/- \/- \/- \- - - threshold
80 |/
+------------------------
Normal-Alerting-Normal-Alerting ...
The history shows the repeated changes in one view. Fix flapping with a longer pending period, a smoother query that uses an average over several minutes, or a threshold with more margin.
Use History in Reviews
Run a weekly review with the team. Sort rules by the number of state changes. Rules at the top of the list either guard a shaky service or need tuning. Delete rules nobody acts on, since silent noise hides real signals.
Troubleshooting
No History Appears
Check that your server has state history enabled and that the rule evaluated at least once. New rules need a little time to build a record.
History Ends Too Early
The backend keeps records for a limited time. Ask your administrator to extend the retention period, or move history to Loki.