RabbitMQ Network Partitions

A network partition (also called a split-brain) occurs when nodes in a RabbitMQ cluster lose the ability to communicate with each other but each group keeps running independently. Each partition thinks the other nodes are dead and continues operating — potentially accepting writes and making conflicting state changes that cannot be automatically reconciled when connectivity is restored.

The Isolated Island Analogy

Imagine a company with offices in two cities connected by a phone line. The phone line breaks. Each office still has staff and keeps working, making decisions and updating their records independently. When the phone line is fixed, both offices have made conflicting changes to the same customer records — and no one knows whose version is correct.

A RabbitMQ network partition is the same problem: two halves of a cluster making independent decisions while disconnected, then needing to reconcile when they reconnect.

What Causes Network Partitions

• Network cable failure between data center racks
• Switch or router failure isolating some nodes
• Cloud provider network hiccup separating availability zones
• Extremely high CPU load causing Erlang heartbeat timeouts
• Firewall rule changes blocking inter-node traffic

Detecting a Partition

rabbitmqctl cluster_status

In a healthy cluster, the partitions section is empty:

{partitions, []}

During a partition, it lists the separated node groups:

{partitions, [{rabbit@node1, [rabbit@node2]},
              {rabbit@node2, [rabbit@node1]}]}

The Management UI also shows a red warning banner at the top of the screen when a partition is detected.

Partition Handling Modes

RabbitMQ provides four configurable modes for handling network partitions. Set the mode in rabbitmq.conf:

cluster_partition_handling = ignore | autoheal | pause_minority | pause_if_all_down

ignore (default — not recommended)

RabbitMQ does nothing automatically. Both sides of the partition keep running and accepting writes. When the partition heals, state conflicts may exist. You must resolve the partition manually. This mode risks split-brain and data divergence — never use it in production without a clear manual recovery plan.

pause_minority (recommended for most clusters)

When a partition occurs, the minority side (the smaller group of nodes) pauses and stops serving clients. The majority side continues operating normally. When connectivity is restored, the minority side resumes and syncs with the majority.

3-node cluster, partition splits into groups of 1 and 2:
  Group of 2 nodes → majority → continues serving
  Group of 1 node  → minority → pauses, refuses connections

This prevents split-brain at the cost of reduced capacity on the minority side. Always use an odd number of nodes so there is always a clear majority.

autoheal

When the partition heals, RabbitMQ automatically chooses a "winning" side and restarts the losers to sync with the winner. The winning side is chosen by the side with more queues. Any messages on the losing nodes that were not on the winning side are lost.

Autoheal recovers automatically but can lose messages. Use it only when availability is more important than message durability.

pause_if_all_down

A hybrid mode. You specify a list of "important" nodes. If all nodes in that list are unreachable, the broker pauses. Otherwise it continues. Useful for multi-datacenter setups where certain nodes are considered authoritative.

Manual Partition Recovery

If you use ignore mode or need manual recovery after autoheal:

1. Identify which partition side has the correct data (the winner).
2. Restart the nodes on the losing side one by one.
3. When a losing node restarts, it joins the cluster fresh (its data is wiped).
4. Messages that existed only on the losing side are permanently lost.

# On a losing-side node:
rabbitmqctl stop_app
rabbitmqctl reset          # wipes this node's data
rabbitmqctl join_cluster rabbit@winning-node
rabbitmqctl start_app

Preventing Network Partitions

• Use redundant network paths: Bond network interfaces and use multiple switches to eliminate single network failure points.
• Keep cluster nodes close together: Nodes in the same data center with low-latency connections are far less likely to experience Erlang heartbeat timeouts.
• Tune heartbeat timeouts: Reduce the Erlang distribution timeout in rabbitmq.conf so partitions are detected quickly and pause_minority engages before clients accumulate much diverged state.
• Use Quorum Queues: Quorum queues handle node failures without requiring partition handling configuration — they simply require a quorum of nodes, making them naturally partition-tolerant.

Recommended Configuration

# rabbitmq.conf
cluster_partition_handling = pause_minority

Combined with an odd-number cluster (3 or 5 nodes) and Quorum Queues, this gives the safest behaviour: the minority side pauses, the majority side continues, no split-brain occurs, and no messages are lost on the surviving side.

Summary

Network partitions occur when cluster nodes cannot communicate but keep running independently. Detect them via rabbitmqctl cluster_status or the Management UI banner. Configure partition handling in rabbitmq.conf: pause_minority is recommended for most clusters — it prevents split-brain by pausing the smaller side. Always use an odd number of nodes so a clear majority always exists. Use Quorum Queues and redundant network paths to minimise both the likelihood and the impact of network partitions.