gRPC Deadlines and Timeouts
A deadline is a fixed point in time after which a gRPC call must stop, whether or not it has finished. Deadlines prevent one slow operation from holding up an entire chain of services. They are one of the most important reliability patterns in distributed systems.
Deadline vs Timeout — The Difference
Timeout: a duration from now (e.g., "give up after 5 seconds") Deadline: a fixed timestamp (e.g., "give up at 14:05:03.456 UTC") gRPC uses deadlines internally. You usually set a timeout and the library converts it to a deadline by adding the duration to the current time. now = 14:05:00.000 timeout = 3 seconds deadline = 14:05:03.000 Why deadlines are better than timeouts for distributed systems: Service A calls Service B (2s timeout), B calls Service C (2s timeout) A's clock: 0ms → calls B B's clock: 1ms → calls C with a NEW 2s timeout Result: total possible wait = 2s (A waits for B) + 2s (B waits for C) = 4s With deadline propagation: C gets only the time A has remaining → prevents cascade
The Cascade Problem Without Deadlines
Without deadline propagation: ┌────────┐ request ┌────────┐ request ┌────────┐ │ Client │──────────►│ API │──────────►│ DB │ │ │ │ server │ │ server │ (stuck for 30s) │ │ │ │ │ │ │ │ waits indefinitely (hangs) │ └────────┘ └────────┘ └────────┘ Result: the client waits forever; connections pile up; memory fills up With deadline propagation: ┌────────┐ deadline: 14:05:03 ┌────────┐ deadline: 14:05:03 ┌───────┐ │ Client │─────────────────────►│ API │─────────────────────►│ DB │ │ │ │ server │ │ server│ │ │◄──DEADLINE_EXCEEDED ─┤ │◄──DEADLINE_EXCEEDED ─┤ │ └────────┘ at 14:05:03 └────────┘ at 14:05:03 └───────┘ All three stop at the same wall-clock time.
Setting a Deadline on the Client (Go)
import (
"context"
"time"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
)
func callWithDeadline(client pb.UserServiceClient, userId int32) (*pb.User, error) {
// Option 1: Timeout (simpler — converts to deadline internally)
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
defer cancel() // always call cancel to release resources
resp, err := client.GetUser(ctx, &pb.GetUserRequest{UserId: userId})
if err != nil {
st, _ := status.FromError(err)
if st.Code() == codes.DeadlineExceeded {
return nil, fmt.Errorf("GetUser timed out after 3 seconds")
}
return nil, err
}
return resp, nil
}
// Option 2: Absolute deadline
func callWithAbsoluteDeadline(client pb.UserServiceClient) {
deadline := time.Now().Add(5 * time.Second)
ctx, cancel := context.WithDeadline(context.Background(), deadline)
defer cancel()
resp, err := client.GetUser(ctx, &pb.GetUserRequest{UserId: 1})
// ...
}
Checking the Deadline on the Server (Go)
func (s *server) GetUser(ctx context.Context,
req *pb.GetUserRequest) (*pb.User, error) {
// Check if caller already cancelled or deadline already passed
if ctx.Err() != nil {
return nil, status.FromContextError(ctx.Err()).Err()
}
// Check how much time is left before starting expensive work
deadline, ok := ctx.Deadline()
if ok {
remaining := time.Until(deadline)
if remaining < 50*time.Millisecond {
// Not enough time to do useful work
return nil, status.Error(codes.DeadlineExceeded,
"insufficient time remaining")
}
log.Printf("Time remaining for request: %v", remaining)
}
// Do the actual work
user, err := s.db.FetchUser(ctx, req.UserId) // pass ctx to DB call
if err != nil {
return nil, err
}
return user, nil
}
Propagating Deadlines Through a Chain
The key rule: always pass the incoming ctx to every downstream call. Never create a fresh context.Background() for a downstream call — you would discard the original deadline.
Service A → Service B → Service C
// CORRECT — deadline propagates through the chain
func (s *OrderService) CreateOrder(ctx context.Context, req *pb.CreateOrderRequest) (*pb.Order, error) {
// ctx contains the deadline from the original caller
// Pass ctx to inventory check — inherits remaining deadline
inv, err := s.inventoryClient.Reserve(ctx, &inv.ReserveRequest{...})
if err != nil { return nil, err }
// Pass ctx to payment — inherits remaining deadline
pay, err := s.paymentClient.Charge(ctx, &pay.ChargeRequest{...})
if err != nil { return nil, err }
return &pb.Order{...}, nil
}
// WRONG — creates a new context, ignores original deadline
func (s *OrderService) CreateOrder(ctx context.Context, req *pb.CreateOrderRequest) (*pb.Order, error) {
freshCtx := context.Background() // ← discard! new context has no deadline
inv, err := s.inventoryClient.Reserve(freshCtx, ...) // runs forever
// ...
}
How gRPC Sends Deadlines on the Wire
gRPC encodes the deadline as a relative timeout in the request headers: grpc-timeout: 3000m (3000 milliseconds = 3 seconds) grpc-timeout: 5S (5 seconds) grpc-timeout: 500000u (500000 microseconds = 500ms) The receiving server converts this relative timeout back into an absolute deadline using its own clock at the moment the request arrives. Units: H = hours M = minutes S = seconds m = milliseconds u = microseconds n = nanoseconds
Context Cancellation
A client can cancel a call at any time, not just when the deadline expires. Cancellation signals the server to stop work early and release resources.
Client cancels manually:
ctx, cancel := context.WithCancel(context.Background())
// Start the call in a goroutine
go func() {
resp, err := client.LongRunningExport(ctx, req)
// err will be codes.Canceled if cancel() was called
}()
// User pressed "Cancel" button after 2 seconds
time.Sleep(2 * time.Second)
cancel() // signals the server to stop
Server detects cancellation:
func (s *server) LongRunningExport(ctx context.Context,
req *pb.ExportRequest) (*pb.ExportResult, error) {
for i, row := range s.fetchAllRows() {
// Check context every few iterations
if i%100 == 0 {
select {
case <-ctx.Done():
return nil, status.Error(codes.Canceled,
"export cancelled by client")
default:
}
}
processRow(row)
}
return &pb.ExportResult{...}, nil
}
Recommended Deadline Values
┌────────────────────────────────┬───────────────────────────────┐ │ Operation Type │ Suggested Deadline │ ├────────────────────────────────┼───────────────────────────────┤ │ Fast DB lookup (by primary key)│ 500ms – 1s │ │ Auth token validation │ 500ms │ │ Complex search / query │ 2s – 5s │ │ Payment processing │ 10s – 15s │ │ File upload / export │ 30s – 120s │ │ Background batch job │ No deadline (use task queue) │ └────────────────────────────────┴───────────────────────────────┘
Summary
Deadlines set a fixed expiry time on a gRPC call. Always set a deadline on outgoing calls — calls without deadlines can block forever and cascade failures through your system. Propagate the incoming context to every downstream call so the deadline is respected end to end. Use ctx.Err() on the server to detect when a client has cancelled or when the deadline has passed before doing expensive work.