gRPC Protocol Buffers Basics

Protocol Buffers (also called Protobuf) is the language gRPC uses to describe data. Think of it as a stricter, faster alternative to JSON. Google designed it to pack data into the smallest possible space while still being easy for machines to decode at high speed.

Why Not Just Use JSON?

JSON stores both the key name and the value together in every message. Protobuf replaces key names with short field numbers. The savings add up fast across millions of messages.

Same data in JSON vs Protobuf:

JSON (text, 65 bytes):
{
  "id": 1001,
  "name": "Alice",
  "active": true
}

Protobuf (binary, ~10 bytes):
  08 E9 07 12 05 41 6C 69 63 65 18 01

• Field names gone — replaced by numbers (08, 12, 18)
• No quotation marks, no curly braces
• Booleans and integers encoded in 1–2 bytes

How Protobuf Encodes Data

Every field in a proto message has a field number and a wire type. The field number identifies which field the data belongs to. The wire type tells the decoder how many bytes to read.

Protobuf Wire Types:
┌───────────┬────────────────┬────────────────────────────────┐
│ Wire Type │ Meaning        │ Used For                       │
├───────────┼────────────────┼────────────────────────────────┤
│     0     │ Varint         │ int32, int64, bool, enum       │
│     1     │ 64-bit         │ double, fixed64                │
│     2     │ Length-delim.  │ string, bytes, nested message  │
│     5     │ 32-bit         │ float, fixed32                 │
└───────────┴────────────────┴────────────────────────────────┘

Encoding example for field: string name = 2;
  Tag byte = (field_number << 3) | wire_type
           = (2 << 3) | 2
           = 18 decimal = 0x12

  Full encoding: 0x12 [length] [bytes of "Alice"]

Variable-Length Integers (Varint)

Small numbers are common in real data. Protobuf uses a trick called varint encoding to store small numbers in fewer bytes than large ones.

Varint encoding:
  Number 1   → 1 byte  (01)
  Number 127 → 1 byte  (7F)
  Number 128 → 2 bytes (80 01)
  Number 300 → 2 bytes (AC 02)

Compare to fixed int32 (always 4 bytes):
  Number 1   → 4 bytes (00 00 00 01)
  Number 127 → 4 bytes (00 00 00 7F)

Benefit: if most of your IDs are below 128, each one costs 1 byte instead of 4.

Scalar Data Types

Protobuf has a fixed set of built-in types. Each maps to a native type in most programming languages.

┌────────────────┬──────────────┬──────────────────────────────┐
│ Protobuf Type  │ Default Value│ Description                  │
├────────────────┼──────────────┼──────────────────────────────┤
│ double         │ 0.0          │ 64-bit floating point        │
│ float          │ 0.0          │ 32-bit floating point        │
│ int32          │ 0            │ 32-bit integer               │
│ int64          │ 0            │ 64-bit integer               │
│ uint32         │ 0            │ Unsigned 32-bit integer      │
│ uint64         │ 0            │ Unsigned 64-bit integer      │
│ bool           │ false        │ True or false                │
│ string         │ ""           │ UTF-8 text                   │
│ bytes          │ empty        │ Raw binary data              │
└────────────────┴──────────────┴──────────────────────────────┘

Default Values and Missing Fields

Every Protobuf field has a default value. If a field is not set, it is simply absent from the encoded bytes. The decoder fills in the default when it reads the message. This is different from JSON, where a missing field is ambiguous — it could mean "not set" or "the sender forgot it."

Proto3 defaults:
  int32  → 0
  string → "" (empty string)
  bool   → false
  enum   → first value (0)
  message → not set (null)

Practical impact:
  If a user's score is 0, that field is NOT sent in the binary.
  If a user's name is empty, that field is NOT sent.
  This keeps message sizes tiny for sparse data.

Backward and Forward Compatibility

Protobuf is designed so that old code can read messages written by new code, and new code can read messages written by old code. This property is called schema evolution.

Version 1 of a message:
  message User {
    int32  id   = 1;
    string name = 2;
  }

Version 2 adds a new field:
  message User {
    int32  id    = 1;
    string name  = 2;
    string email = 3;   ← new field
  }

What happens:
  Old code reads a v2 message → ignores field 3, works fine
  New code reads a v1 message → email is empty string, works fine
  Rule: NEVER reuse a field number once it has been used

Nested Messages

A Protobuf message can contain another message as a field. This is similar to a JSON object inside another JSON object, but encoded more efficiently.

Proto definition:
  message Address {
    string street = 1;
    string city   = 2;
  }

  message User {
    int32   id      = 1;
    string  name    = 2;
    Address address = 3;   ← nested message
  }

Encoded binary (conceptual view):
  [field 1: id=1001] [field 2: name="Alice"] [field 3: [field 1: street="..."] [field 2: city="Berlin"]]

Repeated Fields

A repeated field is a list. It can hold zero, one, or many values of the same type.

Proto definition:
  message Order {
    int32          order_id = 1;
    repeated string items    = 2;
  }

Equivalent in other languages:
  Java:   List<String> items
  Python: list[str] items
  Go:     []string Items

Encoded: each item in the list gets its own length-delimited entry
  with the same field number (2).

Summary

Protocol Buffers encode data as compact binary using field numbers instead of key names. Varint encoding keeps small integers tiny. Default values avoid sending empty fields. Nested messages and repeated fields handle complex structures. Schema evolution rules let old and new code coexist without breaking each other.