Kafka Quotas

A shared Kafka cluster serves many teams at once. One runaway application can flood the brokers and slow everyone else. Quotas set limits on how much data or processing time each client may use. Imagine an apartment building with a shared water pipe: each flat gets a flow limit, so one tap left open does not leave the other tenants dry.

Types of Quotas

Quota	Controls	Unit
producer_byte_rate	Data a client may write	Bytes per second per broker
consumer_byte_rate	Data a client may read	Bytes per second per broker
request_percentage	Broker thread time a client may use	Percent of one thread
controller_mutation_rate	Topic and partition changes	Mutations per second

Byte rate quotas apply per broker. A client limited to 10 MB per second may push 10 MB per second to each broker it talks to.

Who Receives a Quota

Kafka identifies quota targets in three ways.

• User: the authenticated principal, available when security is on
• Client ID: a name that each application sets in its configuration
• User and client ID together: the most specific match

Kafka checks the most specific rule first and falls back to defaults when no match exists.

How Throttling Works

Kafka does not reject traffic that exceeds a quota. The broker slows the client instead. It calculates the delay needed to bring the average rate back under the limit and holds the response for that time.

  Producer sends 30 MB in one second   (limit: 10 MB per second)
        |
        v
  Broker accepts the data, then waits
        |
        v
  Response delayed by about 2 seconds
        |
        v
  Average rate returns to 10 MB per second

The client library honors the delay automatically. Producers slow their sends, and consumers wait longer between fetches.

Set a Quota

bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter \
  --add-config 'producer_byte_rate=10485760,consumer_byte_rate=20971520' \
  --entity-type clients --entity-name analytics-app

This command limits the client named analytics-app to 10 MB per second for writes and 20 MB per second for reads on each broker. The change takes effect without a restart.

Set a Default Quota

bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter \
  --add-config 'producer_byte_rate=5242880' \
  --entity-type clients --entity-default

A default protects the cluster from new, unknown clients.

View Existing Quotas

bin/kafka-configs.sh --bootstrap-server localhost:9092 --describe \
  --entity-type clients --entity-name analytics-app

Request Quota Example

--add-config 'request_percentage=50' --entity-type clients --entity-name report-job

The value 50 gives the client half of one network or I/O thread. Heavy fetch patterns that use little bandwidth but burn CPU respond well to this limit.

Replication Throttling

Moving partitions between brokers copies large amounts of data. Replication throttles cap that traffic so that regular producers and consumers keep their speed.

bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter \
  --add-config 'leader.replication.throttled.rate=50000000,follower.replication.throttled.rate=50000000' \
  --entity-type brokers --entity-name 1

Remove the throttle after the move finishes. A forgotten throttle slows future replication and leaves replicas behind.

Monitor Quota Impact

• Watch the throttle-time metrics for producers and consumers
• Compare per-client byte rates with their limits
• Review clients that sit at their limit all day, since they may need a higher value or a design change

Planning Guidelines

Measure normal traffic first and set quotas about 30 to 50 percent above typical peaks. Share the limits with application owners so surprises do not appear during a busy period. Review them every quarter as workloads grow.