gRPC in Production
Running gRPC in production means caring about observability, performance, schema evolution, and deployment safety. This topic covers the practical patterns that experienced teams use to operate gRPC services reliably at scale.
The Production Readiness Checklist
┌─────────────────────────────────────────────────┬──────┐ │ Item │ Done │ ├─────────────────────────────────────────────────┼──────┤ │ TLS enabled on all connections │ [ ] │ │ Authentication interceptor in place │ [ ] │ │ Deadlines set on every outgoing call │ [ ] │ │ Health check service registered │ [ ] │ │ Kubernetes liveness and readiness probes set │ [ ] │ │ Graceful shutdown implemented │ [ ] │ │ Panic recovery interceptor in place │ [ ] │ │ Structured logging per call │ [ ] │ │ Distributed tracing configured │ [ ] │ │ Metrics exported (Prometheus) │ [ ] │ │ Rate limiting configured │ [ ] │ │ Max message size limits set │ [ ] │ │ Reflection disabled in production │ [ ] │ │ Proto breaking change detection in CI │ [ ] │ └─────────────────────────────────────────────────┴──────┘
Observability — The Three Pillars
Observability means being able to understand what your system is doing from the outside. The three pillars are logs, metrics, and traces.
┌────────────┬──────────────────────────────────────────────────────┐ │ Pillar │ What It Tells You │ ├────────────┼──────────────────────────────────────────────────────┤ │ Logs │ What happened on a specific call (request, response, │ │ │ error message, call duration, user ID) │ ├────────────┼──────────────────────────────────────────────────────┤ │ Metrics │ Aggregated counts and durations across all calls │ │ │ (requests/sec, error rate, p99 latency) │ ├────────────┼──────────────────────────────────────────────────────┤ │ Traces │ The journey of one request across multiple services │ │ │ (A called B called C — which step was slow?) │ └────────────┴──────────────────────────────────────────────────────┘
Metrics with Prometheus
The go-grpc-prometheus library instruments your gRPC server automatically. It exports standard metrics that Prometheus scrapes and Grafana displays.
import (
grpc_prometheus "github.com/grpc-ecosystem/go-grpc-prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
"net/http"
)
func main() {
grpcServer := grpc.NewServer(
grpc.ChainUnaryInterceptor(
grpc_prometheus.UnaryServerInterceptor,
),
grpc.ChainStreamInterceptor(
grpc_prometheus.StreamServerInterceptor,
),
)
pb.RegisterOrderServiceServer(grpcServer, &orderServer{})
// Enable histogram metrics for latency percentiles
grpc_prometheus.EnableHandlingTimeHistogram()
grpc_prometheus.Register(grpcServer)
// Expose Prometheus metrics on a separate HTTP port
go func() {
http.Handle("/metrics", promhttp.Handler())
http.ListenAndServe(":9090", nil)
}()
lis, _ := net.Listen("tcp", ":50051")
grpcServer.Serve(lis)
}
Metrics exposed automatically:
grpc_server_started_total — call count by method and type
grpc_server_handled_total — completed calls by status code
grpc_server_handling_seconds — latency histogram (p50, p90, p99)
grpc_server_msg_received_total — messages received (streaming)
grpc_server_msg_sent_total — messages sent (streaming)
Distributed Tracing with OpenTelemetry
OpenTelemetry is the standard for distributed tracing. It attaches a trace ID to every request. As the request travels from service to service, each hop adds a span. You can see the full call chain in tools like Jaeger or Zipkin.
Trace for one user request:
┌─────────────────────────────────────────────────────────────────┐
│ Trace ID: abc-123 │
│ │
│ [API Gateway] 0ms ──────────────────────────────────── 220ms │
│ [OrderService] 5ms ──────────────────────────── 215ms │
│ [InventoryService] 8ms ──────────── 80ms │
│ [DB query] 10ms ──── 30ms │
│ [PaymentService] 85ms ───────────────────── 210ms │
│ [External API] 90ms ──────────────────── 205ms ← SLOW │
└─────────────────────────────────────────────────────────────────┘
The trace immediately shows that the external payment API caused 120ms of the total 220ms.
Setup in Go:
import (
"go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
"google.golang.org/grpc"
)
grpcServer := grpc.NewServer(
grpc.StatsHandler(otelgrpc.NewServerHandler()),
)
conn, _ := grpc.NewClient("server:443",
grpc.WithStatsHandler(otelgrpc.NewClientHandler()),
grpc.WithTransportCredentials(creds),
)
Performance Tuning
Message Compression
Large payloads benefit from compression. gRPC supports gzip and snappy.
Enable per call or per channel:
// Client — compress all outgoing calls
conn, _ := grpc.NewClient("server:443",
grpc.WithDefaultCallOptions(grpc.UseCompressor(gzip.Name)),
grpc.WithTransportCredentials(creds),
)
// Server — automatically decompresses what client sends
// No extra config needed; gRPC detects grpc-encoding header
Compression sweet spot:
• Useful for large messages (>1KB of text/JSON nested in proto)
• Not useful for small messages — CPU cost exceeds savings
• Not useful for already-compressed data (images, videos)
Maximum Message Size
Default max message size is 4MB. Change it explicitly:
// Server — increase to 10MB
grpcServer := grpc.NewServer(
grpc.MaxRecvMsgSize(10 * 1024 * 1024),
grpc.MaxSendMsgSize(10 * 1024 * 1024),
)
// Client — increase to 10MB
conn, _ := grpc.NewClient("server:443",
grpc.WithDefaultCallOptions(
grpc.MaxCallRecvMsgSize(10 * 1024 * 1024),
grpc.MaxCallSendMsgSize(10 * 1024 * 1024),
),
grpc.WithTransportCredentials(creds),
)
Better practice for very large data:
Use server streaming or client streaming to send data in chunks
instead of raising the limit. This keeps memory usage predictable.
Keepalive
Long-lived connections go idle. Firewalls and NAT gateways silently drop
idle TCP connections after minutes of inactivity. Keepalive pings detect
and recover from these silent drops.
Server keepalive config:
import "google.golang.org/grpc/keepalive"
grpcServer := grpc.NewServer(
grpc.KeepaliveParams(keepalive.ServerParameters{
MaxConnectionIdle: 15 * time.Minute, // close idle connections after 15min
MaxConnectionAge: 30 * time.Minute, // max age per connection
MaxConnectionAgeGrace: 5 * time.Second, // graceful close window
Time: 5 * time.Second, // ping client every 5s
Timeout: 1 * time.Second, // close if no pong in 1s
}),
grpc.KeepaliveEnforcementPolicy(keepalive.EnforcementPolicy{
MinTime: 5 * time.Second, // min time between client pings
PermitWithoutStream: true,
}),
)
Client keepalive:
conn, _ := grpc.NewClient("server:443",
grpc.WithKeepaliveParams(keepalive.ClientParameters{
Time: 10 * time.Second,
Timeout: 5 * time.Second,
PermitWithoutStream: true,
}),
grpc.WithTransportCredentials(creds),
)
Schema Evolution in Production
Proto schema changes must follow safe rules so existing clients and servers keep working.
Safe changes (backward compatible — deploy anytime):
✔ Add a new field with a new field number
✔ Add a new method to a service
✔ Add a new value to an enum (except as default=0)
✔ Change a field name (field number stays the same)
Unsafe changes (breaking — require coordinated rollout):
✘ Remove a field or reuse its number
✘ Change a field's type
✘ Remove a service method
✘ Change a method's request or response type
Breaking change detection in CI with Buf:
# Install Buf CLI
brew install bufbuild/buf/buf
# buf.yaml — configure the registry for comparison
version: v1
breaking:
use:
- FILE
# Run in CI to block merges that break the schema
buf breaking --against .git#branch=main
Deployment Strategy
Safe rollout sequence for a breaking proto change: Phase 1: Deploy new server that supports BOTH old and new schema Old clients → send old messages → new server handles them (backward compat) Monitor for errors, verify old clients still work Phase 2: Deploy new clients that send new schema New clients → send new messages → new server handles them Monitor for errors Phase 3: (Optional) Remove old schema support from server Only when all clients have been updated This is the "expand and contract" pattern — also known as the "parallel change" or "strangler fig" pattern.
gRPC-Gateway — Exposing REST and gRPC Together
grpc-gateway generates an HTTP/JSON reverse proxy from proto annotations.
This lets you serve both gRPC and REST from one Go process.
Proto annotation:
import "google/api/annotations.proto";
service UserService {
rpc GetUser (GetUserRequest) returns (User) {
option (google.api.http) = {
get: "/v1/users/{user_id}" ← REST route generated automatically
};
}
}
Architecture:
REST Client ──HTTP/JSON──► grpc-gateway ──gRPC──► gRPC Server
gRPC Client ──gRPC──────────────────────────────► gRPC Server
Both client types can reach the same server code.
Load Testing with ghz
ghz is a gRPC load testing tool (like Apache ab or wrk for REST):
# 1000 requests, 50 concurrent workers
ghz --insecure \
--proto order.proto \
--call order.OrderService.GetOrder \
-d '{"order_id": 1}' \
-n 1000 -c 50 \
localhost:50051
Output:
Summary:
Count: 1000
Total: 2.34s
Slowest: 85.3ms
Fastest: 2.1ms
Average: 14.8ms
Requests/sec: 427.4
Response time histogram:
2.1 [50] |████
15.0 [700] |████████████████████████████████████████████████
30.0 [200] |████████████
85.3 [50] |████
Summary
Production gRPC requires TLS, authentication, deadlines, health checks, and graceful shutdown as a baseline. Add observability with Prometheus metrics, structured logging per call, and OpenTelemetry traces to see across service boundaries. Tune performance with compression, keepalive pings, and appropriate message size limits. Evolve proto schemas safely using the expand-and-contract pattern, and catch breaking changes in CI with the Buf CLI. Use grpc-gateway when external clients need a REST interface alongside the gRPC API.