gRPC File Transfer

gRPC messages have a size limit, which defaults to 4 MB on the receiving side. Photos, documents, and videos often exceed that size. Streaming solves the problem. The sender cuts the file into small chunks and sends them one after another. The receiver reassembles the chunks into the original file.

The Moving Truck Analogy

A sofa does not fit through a letterbox. Movers load boxes into a truck, drive them over, and rebuild the furniture. Chunked streaming moves a large file the same way: small boxes travel through the connection and the receiver puts the pieces back together.

File (50 MB)
   │ slice into 64 KB chunks
   ▼
[chunk 1][chunk 2][chunk 3] ... [chunk 800]
   │ stream
   ▼
Server writes each chunk to disk ──▶ Complete file

Upload Contract

A client streaming method handles uploads. The first message carries file details. Later messages carry data. A oneof field keeps both shapes in one message type.

service FileService {
  rpc Upload (stream UploadRequest) returns (UploadReply);
  rpc Download (DownloadRequest) returns (stream DownloadReply);
}

message UploadRequest {
  oneof payload {
    FileInfo info  = 1;   // first message only
    bytes    chunk = 2;   // every later message
  }
}

message FileInfo {
  string file_name = 1;
  string content_type = 2;
}

message UploadReply {
  string file_id = 1;
  int64  size_bytes = 2;
}

Server Side Upload

func (s *server) Upload(stream pb.FileService_UploadServer) error {
  var out *os.File
  var total int64

  for {
    req, err := stream.Recv()
    if err == io.EOF {
      break
    }
    if err != nil {
      return err
    }

    switch p := req.Payload.(type) {
    case *pb.UploadRequest_Info:
      out, err = os.Create(safeName(p.Info.FileName))
      if err != nil {
        return status.Error(codes.Internal, "cannot create file")
      }
      defer out.Close()
    case *pb.UploadRequest_Chunk:
      if out == nil {
        return status.Error(codes.InvalidArgument, "file info must come first")
      }
      n, _ := out.Write(p.Chunk)
      total += int64(n)
    }
  }
  return stream.SendAndClose(&pb.UploadReply{FileId: "f-1001", SizeBytes: total})
}

Client Side Upload

stream, _ := client.Upload(ctx)
stream.Send(&pb.UploadRequest{Payload: &pb.UploadRequest_Info{
  Info: &pb.FileInfo{FileName: "report.pdf"},
}})

buf := make([]byte, 64*1024)
for {
  n, err := file.Read(buf)
  if n > 0 {
    stream.Send(&pb.UploadRequest{Payload: &pb.UploadRequest_Chunk{Chunk: buf[:n]}})
  }
  if err == io.EOF {
    break
  }
}
reply, _ := stream.CloseAndRecv()

Choose a Chunk Size

Chunk Size	Effect
Under 16 KB	Many messages, high overhead
32 KB to 1 MB	Good balance for most files
Over 2 MB	Memory pressure and near the default limit

Verify with Checksums

Networks and disks rarely corrupt data, yet verification costs little. The sender computes a SHA-256 hash while it reads the file. The receiver computes its own hash while it writes. The two values must match.

Sender hash:   9c1f...a42b
Receiver hash: 9c1f...a42b   ✔ match → accept
Receiver hash: 7d02...e911   ✖ mismatch → reject and retry

Resume Interrupted Transfers

Large uploads fail midway on weak networks. Add an offset field so that a client can continue from the last confirmed byte instead of starting over. The server reports how many bytes it already holds.

Download with Server Streaming

The server reads the file in chunks and sends each chunk through the stream. The client appends each chunk to a local file. Check the stream context in the loop so a cancelled download stops reading from disk.

Safety Rules

• Clean the file name. Strip path pieces such as ../ so uploads stay inside the storage folder.
• Enforce a maximum total size and reject larger uploads early.
• Check the content type against an allowed list.
• Authenticate every upload call.
• Store large files in object storage and keep only the reference in your database.

Summary

Chunked streaming moves files of any size over gRPC. Send file details first, stream data in chunks near 64 KB, verify with checksums, and support resume with offsets. Validate names and sizes to keep the server safe.