gRPC Health Checks
A health check tells an external system — a load balancer, Kubernetes, or a deployment tool — whether a gRPC server is ready to receive traffic. Without health checks, a load balancer sends calls to a server that has not finished starting up, or that has lost its database connection. Calls fail silently for users.
Why Health Checks Matter
Without health checks:
Pod starts up ──► Kubernetes marks it "Running"
──► Load balancer sends traffic immediately
──► Server is still loading config ← calls fail
Pod loses DB ──► Server process is still alive
──► Load balancer sends traffic
──► Every call returns INTERNAL error
With health checks:
Pod starts up ──► Returns NOT_SERVING until ready
──► Kubernetes waits before sending traffic
Pod loses DB ──► Returns NOT_SERVING
──► Load balancer removes pod from rotation
──► Other pods handle traffic while this one reconnects
The gRPC Health Checking Protocol
The gRPC community standardised a health checking protocol. It defines a single proto service that every server can implement. Load balancers and orchestrators speak this protocol natively.
// Standard proto definition (from grpc/health/v1/health.proto):
syntax = "proto3";
package grpc.health.v1;
message HealthCheckRequest {
string service = 1; // "" = overall server, or a specific service name
}
message HealthCheckResponse {
enum ServingStatus {
UNKNOWN = 0;
SERVING = 1; // ready to handle traffic
NOT_SERVING = 2; // not ready — exclude from load balancing
SERVICE_UNKNOWN = 3; // service name not registered
}
ServingStatus status = 1;
}
service Health {
// Unary — ask once, get current status
rpc Check (HealthCheckRequest) returns (HealthCheckResponse);
// Server streaming — watch for status changes in real time
rpc Watch (HealthCheckRequest) returns (stream HealthCheckResponse);
}
Implementing Health Checks in Go
import (
"google.golang.org/grpc"
"google.golang.org/grpc/health"
"google.golang.org/grpc/health/grpc_health_v1"
)
func main() {
grpcServer := grpc.NewServer()
// Register your business services
pb.RegisterOrderServiceServer(grpcServer, &orderServer{})
pb.RegisterUserServiceServer(grpcServer, &userServer{})
// Create and register the health server
healthSrv := health.NewServer()
grpc_health_v1.RegisterHealthServer(grpcServer, healthSrv)
// Set initial status — NOT_SERVING while loading
healthSrv.SetServingStatus("", grpc_health_v1.HealthCheckResponse_NOT_SERVING)
healthSrv.SetServingStatus("order.OrderService",
grpc_health_v1.HealthCheckResponse_NOT_SERVING)
// Simulate startup loading (DB connection, config, etc.)
if err := connectToDatabase(); err != nil {
log.Fatalf("DB connection failed: %v", err)
}
// Mark as ready once everything is loaded
healthSrv.SetServingStatus("", grpc_health_v1.HealthCheckResponse_SERVING)
healthSrv.SetServingStatus("order.OrderService",
grpc_health_v1.HealthCheckResponse_SERVING)
lis, _ := net.Listen("tcp", ":50051")
log.Println("Server ready on :50051")
grpcServer.Serve(lis)
}
Dynamic Status Updates
Health status is not static. Update it whenever your server's readiness changes — for example, when a downstream dependency goes down.
type orderServer struct {
pb.UnimplementedOrderServiceServer
db *sql.DB
healthSrv *health.Server
}
// Background goroutine monitors DB connection health
func (s *orderServer) monitorDatabase() {
for {
err := s.db.Ping()
if err != nil {
// DB unreachable — stop accepting traffic
s.healthSrv.SetServingStatus("order.OrderService",
grpc_health_v1.HealthCheckResponse_NOT_SERVING)
log.Println("DB unreachable — marked NOT_SERVING")
} else {
// DB back — resume accepting traffic
s.healthSrv.SetServingStatus("order.OrderService",
grpc_health_v1.HealthCheckResponse_SERVING)
}
time.Sleep(10 * time.Second)
}
}
Checking Health from the Command Line
The grpc_health_probe tool sends a health check request and prints the result. Kubernetes uses it in liveness and readiness probe configs.
Install:
go install github.com/grpc-ecosystem/grpc-health-probe@latest
Usage:
# Check overall server
grpc_health_probe -addr=localhost:50051
# Check a specific service
grpc_health_probe -addr=localhost:50051 -service=order.OrderService
# With TLS
grpc_health_probe -addr=server:443 \
-tls -tls-ca-cert=ca.crt \
-service=order.OrderService
Output:
healthy: SERVING
unhealthy: NOT_SERVING (exit code 1)
Kubernetes Liveness and Readiness Probes
Liveness probe: Is the process still alive and not deadlocked?
If it fails repeatedly, Kubernetes restarts the pod.
Readiness probe: Is the server ready to handle user traffic?
If it fails, Kubernetes removes the pod from the Service endpoints.
YAML configuration:
containers:
- name: order-service
image: mycompany/order-service:latest
ports:
- containerPort: 50051
livenessProbe:
exec:
command:
- /bin/grpc_health_probe
- -addr=:50051
initialDelaySeconds: 10 # wait 10s before first check
periodSeconds: 10 # check every 10s
failureThreshold: 3 # restart after 3 consecutive failures
readinessProbe:
exec:
command:
- /bin/grpc_health_probe
- -addr=:50051
- -service=order.OrderService
initialDelaySeconds: 5 # wait 5s — server starts faster than DB connects
periodSeconds: 5
failureThreshold: 2 # remove from LB after 2 consecutive failures
Health Check Flow in a Rolling Deployment
Rolling deployment with health checks: Step 1: New pod starts new-pod readiness probe → NOT_SERVING → Kubernetes does NOT send traffic Step 2: New pod finishes loading config and connecting to DB new-pod readiness probe → SERVING → Kubernetes adds pod to Service endpoints Step 3: Old pod receives a SIGTERM (graceful shutdown signal) old-pod marks itself NOT_SERVING → Kubernetes removes from Service endpoints old-pod waits for in-flight calls to complete (drain period) old-pod exits cleanly Result: zero downtime, no requests dropped during the deployment
Graceful Shutdown with Health
func main() {
grpcServer := grpc.NewServer()
healthSrv := health.NewServer()
grpc_health_v1.RegisterHealthServer(grpcServer, healthSrv)
// Catch OS shutdown signals
quit := make(chan os.Signal, 1)
signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
go func() {
<-quit // wait for signal
// Step 1: Stop accepting new calls by marking NOT_SERVING
healthSrv.SetServingStatus("", grpc_health_v1.HealthCheckResponse_NOT_SERVING)
log.Println("Marked NOT_SERVING — waiting for in-flight calls to finish")
// Step 2: Wait for load balancer to notice (2 × readiness probe period)
time.Sleep(10 * time.Second)
// Step 3: Gracefully stop — waits for active streams to complete
grpcServer.GracefulStop()
log.Println("Server stopped cleanly")
}()
lis, _ := net.Listen("tcp", ":50051")
grpcServer.Serve(lis)
}
Watch — Real-Time Health Streaming
The Watch RPC streams status changes as they happen.
Load balancers that support it get instant updates instead of polling.
Client — watch for status changes:
import "google.golang.org/grpc/health/grpc_health_v1"
healthClient := grpc_health_v1.NewHealthClient(conn)
stream, _ := healthClient.Watch(ctx,
&grpc_health_v1.HealthCheckRequest{Service: "order.OrderService"})
for {
resp, err := stream.Recv()
if err != nil {
break
}
fmt.Printf("Status changed to: %v\n", resp.Status)
// SERVING, NOT_SERVING, UNKNOWN
}
Summary
The gRPC health checking protocol exposes a Health service with Check (unary) and Watch (streaming) methods. Implement it in every gRPC server. Set status to NOT_SERVING on startup until all dependencies are ready, and update status dynamically when dependencies fail or recover. Use grpc_health_probe in Kubernetes liveness and readiness probes to enable zero-downtime deployments and automatic traffic isolation during failures.