Grafana Notification Policies
An alert rule decides when something is wrong. A notification policy decides who hears about it and how often. Without policies, every alert would reach the same inbox, and the right person would drown in noise. Policies route each alert to the correct team, group related alerts together, and control how often reminders arrive.
The Post Office Analogy
A post office sorts letters by address. Local mail goes to one truck, international mail goes to another, and registered mail gets special handling. Notification policies sort alerts by their labels in the same way. An alert labelled team=payments travels to the payments channel, and an alert labelled team=network travels to the network team.
Alert (team=payments, severity=critical)
|
v
+------------------+
| Policy tree |
| sorts by label |
+----+-------+-----+
| |
v v
Payments Network
Slack PagerDuty
The Policy Tree
Policies form a tree. The top of the tree is the default policy. It catches every alert that no other policy claims. Child policies sit below it, and each child holds matching rules.
Default policy --> contact point: Email to ops
|
+-- team = payments --> Slack #payments
| |
| +-- severity = critical --> PagerDuty payments
|
+-- team = network --> Slack #network
Grafana walks the tree from the top for each alert. It moves down to the deepest policy that matches and uses that policy's contact point. A critical payments alert reaches PagerDuty, a low severity payments alert reaches Slack, and an alert with no team label reaches the default email.
Matching Labels
A policy contains one or more label matchers. The matchers use the same operators as silences.
| Operator | Meaning |
|---|---|
| = | Label equals the value |
| != | Label differs from the value |
| =~ | Label matches a pattern |
| !~ | Label does not match a pattern |
Consistent labels on your alert rules make routing simple. Agree on a small set, such as team, severity, and env, and apply them to every rule.
Create a Nested Policy
- Open Alerting and choose Notification policies.
- Click New child policy under the policy you want to extend.
- Add a matcher, such as team = payments.
- Choose a contact point.
- Save the policy.
Continue Matching
Grafana stops at the first matching sibling by default. Switch on Continue matching subsequent sibling nodes when an alert needs to reach several places. A critical alert can notify both the team channel and the on-call pager.
Grouping Alerts
A failing database might trigger fifty alerts at once. Fifty separate messages overwhelm the reader. Grouping bundles related alerts into one message. The Group by setting names the labels that define a group.
Group by: alertname, cluster
Without grouping: 50 messages, one per alert
With grouping: 1 message listing 50 alerts for cluster "prod-1"
Timing Settings
Three timers shape the rhythm of messages for each group.
| Timer | Purpose | Typical Value |
|---|---|---|
| Group wait | Time to collect alerts before the first message | 30 seconds |
| Group interval | Time to wait before sending news about a group that changed | 5 minutes |
| Repeat interval | Time before a reminder about an unresolved alert | 4 hours |
t=0 First alert fires
t=30s Group wait ends --> message 1 (all alerts so far)
t=2m New alert joins group
t=5m30s Group interval passes --> message 2 (update)
t=4h Repeat interval passes --> reminder
Short timers speed up delivery and increase message volume. Long timers calm the stream and delay the news. Pick values that match the urgency of each team.
Mute Timings in Policies
A policy can carry mute timings. Alerts routed through that policy stay quiet during the scheduled windows. Attach nightly mute timings to low severity branches only.
Test Your Routing
The policy page includes a preview feature. Enter sample labels and Grafana shows which policy and which contact point would receive the alert. Test critical, low severity, and unlabelled cases before you rely on the tree.
Design Tips
- Keep the tree shallow. Two or three levels cover most teams.
- Set a useful default contact point, so no alert disappears.
- Route by team first, and by severity second.
- Review the tree whenever a team changes ownership.