gRPC Load Balancing
When one server cannot handle all incoming calls, you run multiple copies — called replicas. A load balancer distributes calls across those replicas so no single instance is overwhelmed. gRPC load balancing has some unique characteristics because it uses long-lived HTTP/2 connections.
Why gRPC Load Balancing Is Different
A traditional HTTP/1.1 load balancer opens a new connection for every request. It can distribute load simply by deciding which server gets the next connection.
gRPC reuses one long-lived HTTP/2 connection and multiplexes many calls over it. If the load balancer assigns all calls to one connection, all calls go to one server — even if five other servers are idle.
HTTP/1.1 load balancing — works at connection level: Request 1 → new connection → Server A Request 2 → new connection → Server B ← natural distribution Request 3 → new connection → Server A gRPC naive load balancing — fails at connection level: All calls → one long-lived connection → Server A (overloaded) Server B, C, D → idle gRPC correct load balancing — must work at call (stream) level: Call 1 → Server A Call 2 → Server B ← each call picks a server independently Call 3 → Server C
Three Load Balancing Approaches
┌──────────────────────────┬─────────────────────────────────────────────┐ │ Approach │ How It Works │ ├──────────────────────────┼─────────────────────────────────────────────┤ │ Proxy-based (L7 proxy) │ Traffic flows through a proxy (Envoy, etc.) │ │ Client-side │ Client holds list of servers, picks one │ │ Lookaside (xDS) │ External controller tells client where to go│ └──────────────────────────┴─────────────────────────────────────────────┘
Approach 1: Proxy-Based Load Balancing
An L7 (application-layer) proxy sits between clients and servers. Envoy is the most widely used proxy for gRPC. It understands HTTP/2 streams and distributes individual gRPC calls across backend servers.
Architecture: ┌──────────┐ ┌─────────────┐ ┌────────────┐ │ Client A │──────►│ │──────►│ Server 1 │ ├──────────┤ │ Envoy │──────►│ Server 2 │ │ Client B │──────►│ Proxy │──────►│ Server 3 │ ├──────────┤ │ (L7) │ └────────────┘ │ Client C │──────►│ │ └──────────┘ └─────────────┘ Clients connect to ONE address (the proxy). Proxy maintains connections to ALL backends. Proxy distributes calls using round-robin or least-connections. Benefits: • Clients need no discovery logic • Backends can be added/removed without client changes • Proxy can enforce rate limits, retries, timeouts centrally Drawback: • Proxy is an extra network hop (adds ~1–5ms latency) • Proxy itself must be highly available
Envoy config snippet for gRPC load balancing:
clusters:
- name: order_service
type: STRICT_DNS
lb_policy: ROUND_ROBIN
http2_protocol_options: {} ← tells Envoy to use HTTP/2
load_assignment:
cluster_name: order_service
endpoints:
- lb_endpoints:
- endpoint:
address: { socket_address: { address: order-1, port_value: 50051 } }
- endpoint:
address: { socket_address: { address: order-2, port_value: 50051 } }
- endpoint:
address: { socket_address: { address: order-3, port_value: 50051 } }
Approach 2: Client-Side Load Balancing
The client itself holds the list of server addresses and picks one per call. No proxy required. This removes the extra hop but adds complexity to the client.
Client-side LB flow: 1. Client queries DNS or a service registry for "order-service" 2. DNS returns: [10.0.0.1, 10.0.0.2, 10.0.0.3] 3. Client opens one connection to each address 4. For each new call, client picks one connection (round-robin) 5. When a server is removed, client stops using its connection ┌──────────────────────────────────┐ │ gRPC Client │ │ │ │ resolver ──► [10.0.0.1:50051] │──► Server 1 │ [10.0.0.2:50051] │──► Server 2 │ [10.0.0.3:50051] │──► Server 3 │ │ │ balancer: round-robin │ │ → call 1 goes to Server 1 │ │ → call 2 goes to Server 2 │ │ → call 3 goes to Server 3 │ └──────────────────────────────────┘
Go — enable client-side round-robin load balancing:
import (
"google.golang.org/grpc/balancer/roundrobin"
_ "google.golang.org/grpc/health/grpc_health_v1" // health check support
)
conn, err := grpc.NewClient(
"dns:///order-service.default.svc.cluster.local:50051",
grpc.WithTransportCredentials(creds),
grpc.WithDefaultServiceConfig(`{"loadBalancingPolicy":"round_robin"}`),
)
// The "dns:///" prefix tells gRPC to use DNS resolution
// DNS returns multiple A records (one per pod)
// round_robin distributes calls across all resolved addresses
Load Balancing Policies
┌──────────────────────┬────────────────────────────────────────────────┐ │ Policy │ How It Picks a Server │ ├──────────────────────┼────────────────────────────────────────────────┤ │ round_robin │ Cycles through servers in order │ │ pick_first │ Always uses the first available server │ │ least_request │ Picks the server with fewest active calls │ │ random │ Picks a random server each time │ │ ring_hash │ Maps a key to a consistent server (useful for │ │ │ cache affinity) │ └──────────────────────┴────────────────────────────────────────────────┘
Approach 3: xDS / Lookaside Load Balancing
xDS (discovery service protocol from Envoy) lets a central control plane push routing, load balancing, and health information directly to clients in real time. Istio and Envoy use this protocol. The client gets smarter routing decisions without a proxy in the data path.
xDS architecture:
┌─────────────────────┐
│ Control Plane │ (Istio / Traffic Director / xDS server)
│ (xDS server) │
└──────────┬──────────┘
│ Pushes: server list, health, weights, routing rules
▼
┌─────────────────────┐
│ gRPC Client │ (has built-in xDS support in grpc-go 1.33+)
│ with xDS resolver │
└──────────┬──────────┘
│ Connects directly to healthy backends
▼
┌────────────────────────────────┐
│ Server 1 │ Server 2 │ Server 3│
└────────────────────────────────┘
No proxy in the data path = zero extra latency.
Control plane handles discovery = no client-side DNS polling.
Load Balancing in Kubernetes
Problem: Kubernetes Service (ClusterIP) uses iptables for L4 TCP load
balancing. It load-balances connections, not gRPC calls. One gRPC
connection goes to one pod.
Solutions:
┌──────────────────────────┬────────────────────────────────────────┐
│ Solution │ How │
├──────────────────────────┼────────────────────────────────────────┤
│ Headless Service │ DNS returns all pod IPs; client does │
│ + client-side LB │ round-robin across them │
├──────────────────────────┼────────────────────────────────────────┤
│ Istio / Linkerd │ Sidecar proxies intercept gRPC, │
│ (service mesh) │ load-balance at call level │
├──────────────────────────┼────────────────────────────────────────┤
│ Ingress with Envoy │ External traffic → Envoy → pods │
└──────────────────────────┴────────────────────────────────────────┘
Headless Service YAML:
apiVersion: v1
kind: Service
metadata:
name: order-service
spec:
clusterIP: None ← headless: DNS returns all pod IPs
selector:
app: order-service
ports:
- port: 50051
Summary
gRPC's persistent HTTP/2 connections break naive connection-level load balancers. Use an L7 proxy like Envoy for the simplest deployment with a central control point. Use client-side round-robin with DNS when you want zero extra hops. Use xDS in a service mesh for dynamic, control-plane-managed routing at scale. In Kubernetes, use a headless Service or a service mesh to ensure calls are distributed across all pods.