Grafana Annotations

Annotations mark specific moments in time directly on your charts. They appear as vertical lines or highlighted regions on time-series panels. Annotations answer a critical question: "What happened here?" — correlating events like deployments, incidents, and configuration changes with the metric data you see on the chart.

The Timeline Analogy

Imagine a stock price chart. A vertical line at a specific date labelled "CEO resigned" explains a sudden price drop. Without that label, the drop looks random. Annotations work the same way on your infrastructure charts — they place context directly where the event occurred so anyone reading the chart understands why a metric changed.

CPU Usage Chart with Annotations:
100 |
 80 |                  │
 60 |      ────────────│──────
 40 |  ────             │     ────
 20 |                   │
  0 |__________________________ time
    12:00      13:00    13:30    14:00
                         │
                         └── Annotation: "Deployed v2.4.1"

Types of Annotations

Manual Annotations

You add manual annotations directly on a dashboard chart. Click on a point on a time-series panel to mark a single moment, or click and drag to mark a time range. A dialog asks for a description and optional tags.

Single point annotation:  │  (vertical line at one moment)
Time range annotation:    │▓▓▓▓▓▓▓▓│  (shaded region between two times)

Native Annotations

Native annotations are stored inside Grafana's own database. They persist across sessions and are visible to all users who open the dashboard. Manual annotations you add by clicking the chart become native annotations automatically.

Query Annotations

Query annotations pull event data from an external data source and display it automatically on panels. If your deployment system writes deployment timestamps to a database, Grafana can query that database and draw an annotation line for every deployment — without anyone clicking the chart manually.

Adding a Manual Annotation

Method 1 – Click on the Chart

Hold Ctrl (or Cmd on macOS) and click on any point on a time-series panel. A dialog box opens asking for a description and tags. Type a message like Deployed version 2.4.1 and click Save. The annotation line appears immediately.

Method 2 – Annotate a Range

Hold Ctrl and click-and-drag across the chart to mark a time range. A shaded region appears with the annotation text. Use this to mark the duration of an incident — from when it started to when it was resolved.

Managing Native Annotations

Go to Dashboard settings (gear icon) → Annotations to see and manage annotation sources. A default source called Annotations & Alerts is always present. This source displays both manual annotations and Grafana alert state changes as vertical event lines.

Showing Alert State Changes as Annotations

When an alert fires or resolves, Grafana automatically creates an annotation on the dashboard panels that use the same data source and time range. A red dashed line marks when the alert fired; a green dashed line marks when it resolved. This creates a visual record of every incident directly on the affected metric charts.

CPU Chart with Alert Annotations:
100 |
 90 | ─ ─ ─ ─ ─ ─ ─ ─ ─   (threshold)
 80 |      ┊             ┊
 60 |  ─── │ ─────────── │ ────  CPU line
 40 |      │             │
    |      │             │
    |   🔴 Firing      🟢 Resolved

Query Annotations – Automatic from Data Source

Query annotations automatically generate annotation lines from data stored in a database or time-series tool. This is the most powerful way to annotate because it requires no manual input after the initial setup.

Setting Up a Query Annotation

In Dashboard Settings → Annotations, click New query.

Name:        Deployments
Data source: MySQL
Query:
  SELECT
    deployed_at AS time,
    CONCAT('Deploy: ', version) AS text,
    'deployment' AS tags
  FROM deployments
  WHERE $__timeFilter(deployed_at)

Grafana runs this query whenever the dashboard loads or refreshes. Each row in the result becomes one annotation line. The time column sets the position, text is the tooltip label, and tags are optional tags for filtering.

Annotation from Prometheus

Name:        Restart Events
Data source: Prometheus
Query:       changes(process_start_time_seconds[5m]) > 0
Step:        60s

This query detects moments when a process restarted. An annotation appears on the chart every time a service restart is detected.

Annotation Tags and Filtering

Tags on annotations allow filtering. On the dashboard, open the annotation layer controls (the annotation icon at the top of time-series panels) to toggle visibility of annotations by tag. For example, show only deployment annotations and hide incident annotations, or vice versa.

Annotation API

Grafana provides an HTTP API for creating annotations programmatically. Your CI/CD pipeline can call this API at the end of each deployment to add a deployment annotation automatically.

POST http://localhost:3000/api/annotations
Content-Type: application/json
Authorization: Bearer <API_KEY>

{
  "time":     1709294400000,
  "isRegion": false,
  "text":     "Deployed v2.4.1 to production",
  "tags":     ["deployment", "production"]
}

The time field is a Unix timestamp in milliseconds. Most CI/CD tools like GitHub Actions, Jenkins, and GitLab CI can execute this API call as a pipeline step. After adding the API call to your pipeline, every future deployment automatically annotates your Grafana dashboards without any manual action.

Hiding Annotations on a Dashboard

Click the annotation icon that appears at the top left of time-series panels on the dashboard (a small bookmark or flag icon) to toggle annotation visibility on and off without removing them. Each annotation source in the dashboard settings also has a toggle to disable it globally for all panels on that dashboard.