SRE OpenTelemetry
Modern services call dozens of other services for a single click. Finding the slow or broken piece requires data from every hop. OpenTelemetry, often shortened to OTel, gives teams a single open standard for collecting that data.
What Is OpenTelemetry?
OpenTelemetry is a set of open-source tools, libraries, and specifications that create and send traces, metrics, and logs. Think of a universal power plug. Without the plug, each device needs a different cable. With the plug, any device works in any socket. OTel works as the universal plug between applications and monitoring backends.
Three Kinds of Signals
| Signal | Describes | Question Answered |
|---|---|---|
| Traces | The path of one request across services | Where did the time go? |
| Metrics | Numbers over time | How much and how often? |
| Logs | Text records of events | What exactly happened? |
Main Pieces
API and SDK
The API defines how code creates telemetry. The SDK implements the rules for sampling, batching, and exporting. Libraries exist for Java, Python, Go, JavaScript, .NET, and other languages.
Instrumentation
Automatic instrumentation attaches to common frameworks and records web requests and database calls with almost no code. Manual instrumentation lets developers add custom spans, such as "apply discount code."
Collector
The Collector is a small service that receives telemetry, processes it, and forwards it. It has three parts: receivers accept data, processors filter or enrich it, and exporters send it to a backend.
Data Flow Diagram
[Service A]──┐
[Service B]──┼→ [OTel Collector] → [Tracing Backend]
[Service C]──┘ receive [Metrics Backend]
process [Log Backend]
export
Switching backends means changing only Collector settings. Application code stays the same, which avoids lock-in.
Understanding Traces and Spans
A trace represents one request. A span represents one piece of work inside it. Spans nest like folders, and each span records a start time, a duration, and details such as status.
Trace: Checkout request (total 1,250 ms)
|-- web-frontend [=========================] 1,250 ms
|-- cart-service [=====] 200 ms
|-- payment-service [==========================] 900 ms
|-- fraud-check [==] 80 ms
|-- bank-api-call [=======================] 780 ms
The picture shows the cause in seconds: the bank call consumes most of the time. Without a trace, the team might spend an afternoon blaming the web frontend.
Context Propagation
Each service passes a trace identifier to the next service through request headers. The identifier links spans from different machines into one story. A missing hand-off creates a broken trace, so teams verify that every service forwards the header.
Sampling
Storing every trace costs money. Sampling keeps a share of them.
- Head sampling decides at the start of a request, such as keeping 10 percent.
- Tail sampling decides after the request ends, so the system keeps every slow or failed trace.
Tail sampling preserves the most useful traces and trims the boring ones.
Getting Started
- Pick one important service and enable automatic instrumentation.
- Run a Collector and point the service at it.
- Send data to a free or existing backend.
- Add custom spans around key business steps.
- Link trace identifiers into log lines to jump between signals.
- Expand service by service.
Common Mistakes
- Instrumenting everything at once and drowning in data.
- Adding personal data to span details.
- Forgetting to propagate headers through message queues.
- Keeping every trace and facing a surprise storage bill.
Key Takeaways
OpenTelemetry standardizes how services produce traces, metrics, and logs. Use the Collector to route data, trace context to connect services, and sampling to control cost.