GCP Pub/Sub

Cloud Pub/Sub is GCP's fully managed messaging service that enables asynchronous communication between different parts of an application. Pub/Sub stands for Publish/Subscribe — one component publishes messages, and one or more components subscribe to receive and process them, all without direct connections between the sender and receiver.

Imagine a newspaper. The publisher (news agency) prints and distributes newspapers without knowing who reads them. Subscribers (readers) receive their newspaper on their own schedule. Pub/Sub works the same way for applications — the sender sends a message and moves on, while receivers process messages at their own pace.

Why Use Pub/Sub?

In a tightly coupled system, Service A calls Service B directly. If Service B is slow or offline, Service A fails too. Pub/Sub introduces a buffer between services:

Without Pub/Sub (Tight Coupling):
Service A ──HTTP──▶ Service B
(If B is slow, A waits. If B crashes, A fails.)

With Pub/Sub (Loose Coupling):
Service A ──publish──▶ [Pub/Sub Topic] ──deliver──▶ Service B
(A publishes and moves on. B processes at its own speed.)

Core Concepts

Topic

A topic is the named channel to which messages are published. Publishers send messages to a topic, not directly to subscribers.

Subscription

A subscription is an attachment to a topic that defines how messages are delivered to subscribers. Multiple subscriptions can exist on one topic — each subscription receives a copy of every message.

Message

A message is a unit of data sent through Pub/Sub. It consists of a data payload (up to 10 MB) and optional key-value attributes (metadata).

Publisher A ──▶ Topic: "orders"
Publisher B ──▶     │
                    │
         ┌──────────┴──────────┐
         ▼                     ▼
Subscription: "billing"   Subscription: "shipping"
         │                     │
         ▼                     ▼
  Billing Service          Shipping Service

Both the Billing and Shipping services receive every order message independently.

Push vs Pull Delivery

Creating a Topic and Subscription

# Create a topic
gcloud pubsub topics create orders

# Create a pull subscription
gcloud pubsub subscriptions create billing-sub \
  --topic=orders

# Create a push subscription (sends to a Cloud Run URL)
gcloud pubsub subscriptions create shipping-sub \
  --topic=orders \
  --push-endpoint=https://my-shipping-service-xyz.run.app/handle-order

Publishing and Receiving Messages

Publish a message from CLI

gcloud pubsub topics publish orders \
  --message='{"order_id": "ORD-9001", "product": "Laptop", "qty": 1}' \
  --attribute=priority=high

Pull and acknowledge messages from CLI

# Pull up to 5 messages
gcloud pubsub subscriptions pull billing-sub --limit=5 --auto-ack

Python Publisher

from google.cloud import pubsub_v1
import json

project_id = "my-project"
topic_id = "orders"

publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path(project_id, topic_id)

order = {"order_id": "ORD-9001", "product": "Laptop", "qty": 1}
message_data = json.dumps(order).encode("utf-8")

future = publisher.publish(topic_path, data=message_data, priority="high")
print(f"Published message ID: {future.result()}")

Python Subscriber (Pull)

from google.cloud import pubsub_v1
import json

project_id = "my-project"
subscription_id = "billing-sub"

subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path(project_id, subscription_id)

def process_message(message):
    order = json.loads(message.data.decode("utf-8"))
    print(f"Processing order: {order['order_id']}")
    message.ack()  # Acknowledge = tell Pub/Sub the message was processed

streaming_pull = subscriber.subscribe(subscription_path, callback=process_message)
print("Listening for messages...")
streaming_pull.result()

Message Acknowledgment and Retry

Publisher → Topic → Subscription → Subscriber
                                        │
                              ┌─────────┴──────────┐
                              │                    │
                       message.ack()          message.nack()
                     (processed OK)          (failed, retry)
                              │                    │
                      Message deleted        Message redelivered
                      from subscription      after ackDeadline

If a subscriber does not acknowledge a message within the ackDeadline (default: 10 seconds), Pub/Sub redelivers the message. This ensures at-least-once delivery — every message is delivered at least once, even if a subscriber crashes mid-process.

Dead Letter Topic

If a message fails to be acknowledged after a set number of delivery attempts, Pub/Sub can forward it to a Dead Letter Topic for manual inspection.

Normal Topic: "orders"
    │
    │ Message fails 5 delivery attempts
    ▼
Dead Letter Topic: "orders-dead-letter"
    │
    ▼
Alert / Manual investigation

gcloud pubsub subscriptions modify-config billing-sub \
  --dead-letter-topic=orders-dead-letter \
  --max-delivery-attempts=5

Common Pub/Sub Use Cases

Key Takeaways

• Pub/Sub decouples services so they do not depend on each other's availability.
• Publishers send messages to topics; subscribers receive messages via subscriptions.
• Multiple subscriptions on one topic each receive an independent copy of every message.
• Pull delivery suits batch workers; Push delivery suits webhooks and Cloud Run.
• Unacknowledged messages are redelivered automatically — ensuring at-least-once delivery.
• Dead Letter Topics capture repeatedly failing messages for investigation.