Grafana Tempo Tracing

Metrics tell you that checkout runs slowly. Logs show errors from individual services. Neither one tells you which step inside a single request caused the delay. Traces fill that gap. Grafana Tempo stores traces at scale, and Grafana draws them as clear timelines that point straight to the slow step.

The Parcel Tracking Analogy

A parcel tracking page lists every stop: picked up, sorted at the hub, loaded on a truck, delivered. Each stop shows a time. When the parcel arrives late, you see exactly which stop caused the delay. A trace follows a user request the same way through every service it touches.

Request: Place Order   (trace ID: 7f3a9c...)

Browser --> API Gateway --> Orders --> Payments --> Database
              12 ms        30 ms     480 ms       40 ms
                                      ^
                                  slow step

Trace Vocabulary

Term	Plain Meaning
Trace	The full journey of one request
Span	One step inside the journey, with a start time and a duration
Trace ID	A unique number that links all spans of one request
Parent span	A step that calls other steps
Service name	The application that produced a span

What Tempo Does

Tempo is the storage and search backend for traces. Applications send spans to Tempo, often through an OpenTelemetry collector. Tempo keeps the spans in low-cost object storage. Grafana queries Tempo and displays the results.

+-----------+   spans   +-----------+  store  +---------+
| Your App  | --------> | Collector | ------> |  Tempo  |
+-----------+           +-----------+         +----+----+
                                                   |
                                              query|
                                             +-----v-----+
                                             |  Grafana  |
                                             +-----------+

Add the Tempo Data Source

1. Open Connections and choose Add new connection.
2. Search for Tempo and select it.
3. Enter the Tempo address, such as http://tempo:3200.
4. Click Save and test.

Find a Trace

Search by Trace ID

Open Explore, choose the Tempo data source, and paste a trace ID. Grafana loads that single trace.

Search by Conditions

Tempo provides a query language named TraceQL. A TraceQL query describes the spans you want, wrapped in curly braces.

{ resource.service.name = "payments" }

{ resource.service.name = "payments" && duration > 500ms }

{ span.http.status_code = 500 }

The first query finds traces that touch the payments service. A second query narrows the result to spans slower than half a second. The last query lists spans that returned an HTTP 500 error.

Read the Trace View

The trace view draws a waterfall. Each bar represents a span. The bar position shows when the span started, and the bar length shows how long it ran.

Place Order                |=====================================| 560 ms
  API Gateway              |==|                                    12 ms
  Orders Service              |====|                               30 ms
    Payments Service              |====================|           480 ms
      Card Check                   |=======|                       90 ms
      Database Write                          |=======|            40 ms
    Send Email                                         |=|         8 ms

Indented bars belong to the span above them. The Payments span dominates the timeline, which tells the team where to look first. Click any bar to see attributes such as the HTTP method, the database statement, or an error message.

Connect Traces to Logs and Metrics

Traces become far more useful when they link to other signals.

• Traces to logs: configure the Tempo data source to open matching Loki logs for a span, using the trace ID or service name.
• Logs to traces: add a derived field to the Loki data source that extracts the trace ID from a log line and turns it into a link.
• Traces to metrics: link a span to a Prometheus query for the same service and time.

Slow metric --> Trace shows slow span --> Logs of that span --> Root cause

Service Graph

Tempo can generate a service graph from trace data. The graph draws each service as a node and each call as an arrow, with request rates and error rates along the way. Use it to learn the architecture of a system you did not build.

[Gateway] --> [Orders] --> [Payments] --> [Database]
                  |
                  +--> [Email]

A Troubleshooting Walkthrough

1. A latency panel shows checkout slowing down at 14:05.
2. Open Explore with Tempo and run a TraceQL query for slow checkout spans.
3. Open a slow trace and spot the Payments span at 480 ms.
4. Click the span and follow the link to its logs.
5. Read a log line that reports a card service timeout.

The team finds the cause in minutes and skips the hand search through every service.

Tips for Useful Traces

• Name services clearly, because names appear on every span.
• Add attributes that matter, such as order type or region, and avoid personal data.
• Sample high-volume traffic to control storage cost.
• Keep trace IDs in your log lines.