gRPC Retries and Backoff

Networks fail in small ways all the time. A server restarts, a connection drops, or a request lands at a bad moment. A retry sends the same call again and often succeeds. Done carelessly, retries multiply traffic and crash a struggling server. Done carefully, they hide short glitches from users.

The Phone Call Analogy

You call a friend and hear a busy tone. You wait a moment and call again. If the line stays busy, you wait longer before the next attempt. You never redial a hundred times per second. That patience is called backoff.

Attempt 1 ──✖ fails ── wait 0.1s
Attempt 2 ──✖ fails ── wait 0.2s
Attempt 3 ──✖ fails ── wait 0.4s
Attempt 4 ──✔ success

Which Errors Deserve a Retry

Status Code	Retry?	Reason
UNAVAILABLE	Yes	Server or network is temporarily down
DEADLINE_EXCEEDED	Sometimes	Safe only when the call is idempotent
RESOURCE_EXHAUSTED	Carefully	Server asks for slower traffic
INVALID_ARGUMENT	No	The request itself is wrong
NOT_FOUND	No	Retrying returns the same answer
PERMISSION_DENIED	No	Retrying changes nothing

Idempotency

An idempotent call gives the same result when repeated. Reading a profile is idempotent. Charging a credit card is not, because a retry might charge twice. Retry only safe calls, or add a unique request ID so the server detects duplicates.

message ChargeRequest {
  string idempotency_key = 1;   // server remembers this key
  int64  amount_cents    = 2;
}

The server stores each key with its result. A repeated key returns the stored result and skips a second charge.

Built In Retry Policy

gRPC clients accept a retry policy inside a service config. The policy lives in JSON, and the channel applies it with no extra code in your calls.

{
  "methodConfig": [{
    "name": [{ "service": "shop.OrderService" }],
    "retryPolicy": {
      "maxAttempts": 4,
      "initialBackoff": "0.1s",
      "maxBackoff": "1s",
      "backoffMultiplier": 2,
      "retryableStatusCodes": ["UNAVAILABLE"]
    }
  }]
}

Policy Fields

• maxAttempts counts the first try plus retries. gRPC caps the value at 5.
• initialBackoff sets the first wait.
• backoffMultiplier grows each following wait.
• maxBackoff limits the longest wait.
• retryableStatusCodes lists the codes that trigger a retry.

Attach the Policy in Go

const serviceConfig = `{
  "methodConfig": [{
    "name": [{ "service": "shop.OrderService" }],
    "retryPolicy": {
      "maxAttempts": 4,
      "initialBackoff": "0.1s",
      "maxBackoff": "1s",
      "backoffMultiplier": 2,
      "retryableStatusCodes": ["UNAVAILABLE"]
    }
  }]
}`

conn, err := grpc.NewClient("orders:50051",
  grpc.WithTransportCredentials(creds),
  grpc.WithDefaultServiceConfig(serviceConfig),
)

Jitter

Imagine a thousand clients that all fail at the same second. If each waits exactly 100 milliseconds, all thousand retry together and hit the server in one wave. Jitter adds a random delay to each wait. The built in policy applies jitter automatically, which spreads retries across time.

Retry Throttling

A server in trouble gets worse when every client retries. Retry throttling stops retries when too many calls fail. The client keeps a token bucket. Failures drain tokens, successes refill them, and an empty bucket blocks further retries.

Deadlines Still Rule

All attempts share one overall deadline. Retries never extend the total time. A call with a two second deadline stops retrying at two seconds, whatever the policy says.

Hedging

Hedging sends a second copy of a slow call before the first one finishes. The client takes the first reply and cancels the rest. Hedging cuts tail latency for idempotent reads, and it increases server load. Use it sparingly.

Summary

Retry transient errors such as UNAVAILABLE, wait longer between attempts, add jitter, and respect deadlines. Protect non idempotent calls with idempotency keys. Retry throttling prevents a small failure from growing into an outage.