Kafka Security Authentication Authorization and Encryption

A Kafka cluster with no security configuration is completely open — any client on the network can connect, read any topic, write to any topic, and delete data. This is acceptable on a local development machine but catastrophic in production. Kafka provides three security pillars: encryption (data is private in transit), authentication (clients prove who they are), and authorization (clients are allowed only what they need). Together they form a production-grade security posture.

The Three Pillars of Kafka Security

KAFKA SECURITY MODEL:

ENCRYPTION (SSL/TLS):
  "Nobody can sniff or tamper with data in transit"
  Protects: network traffic between clients and brokers,
            and between brokers (inter-broker)

AUTHENTICATION (SASL mechanisms):
  "Only known, verified clients can connect"
  Protects: unauthorized clients connecting to the cluster
  Methods: SASL/PLAIN, SASL/SCRAM, SASL/GSSAPI (Kerberos), SASL/OAUTHBEARER, mTLS

AUTHORIZATION (ACLs):
  "Each authenticated client can only do what it is allowed to do"
  Protects: unauthorized reads, writes, topic deletions, consumer group manipulation
  Mechanism: Access Control Lists (ACLs) stored in ZooKeeper or KRaft metadata

All three work together. Authentication without authorization means any
authenticated client can do anything. Encryption without authentication
means data is encrypted but from unknown sources.

Encryption with SSL/TLS

Kafka uses SSL/TLS to encrypt all network communication. Without encryption, anyone on the same network (or with access to network traffic) can read your Kafka messages, which may contain sensitive user data, financial records, or internal business information.

How SSL Works in Kafka

Each broker has an SSL certificate that proves its identity. Clients verify this certificate before establishing a connection — preventing man-in-the-middle attacks. After verification, all traffic flows through an encrypted TLS channel.

SSL SETUP OVERVIEW:

Step 1: Create a Certificate Authority (CA) — the trusted root
Step 2: Create a keystore for each broker (contains broker's private key + certificate)
Step 3: Create a truststore for clients (contains CA certificate to verify brokers)
Step 4: Configure brokers to use SSL listener
Step 5: Configure clients to use SSL with truststore

BROKER CONFIG (server.properties):
  listeners=SSL://broker1:9093
  ssl.keystore.location=/etc/kafka/ssl/broker.keystore.jks
  ssl.keystore.password=changeit
  ssl.key.password=changeit
  ssl.truststore.location=/etc/kafka/ssl/broker.truststore.jks
  ssl.truststore.password=changeit

CLIENT CONFIG (producer.properties / consumer.properties):
  security.protocol=SSL
  ssl.truststore.location=/etc/kafka/ssl/client.truststore.jks
  ssl.truststore.password=changeit

Mutual TLS (mTLS): Two-Way Authentication

Standard SSL verifies only the broker's identity. Mutual TLS requires both the broker and the client to present certificates, so the broker also verifies the client's identity. mTLS serves as both an encryption mechanism and an authentication mechanism simultaneously.

STANDARD TLS:          Client verifies BROKER certificate
                       (one-way — only broker is verified)

MUTUAL TLS (mTLS):     Client verifies BROKER certificate
                       BROKER verifies CLIENT certificate
                       (two-way — both parties verified)

mTLS broker config additions:
  ssl.client.auth=required   ← require client certificates (optional: request)

mTLS client config additions:
  ssl.keystore.location=/etc/kafka/ssl/client.keystore.jks
  ssl.keystore.password=changeit

Authentication with SASL

SASL (Simple Authentication and Security Layer) is the framework Kafka uses for authentication. It sits on top of either a plaintext or SSL connection. Kafka supports several SASL mechanisms.

SASL/PLAIN: Username and Password

The simplest mechanism. Credentials (username and password) are sent as plaintext in the SASL handshake. Always use this over TLS (never over plaintext) because SASL/PLAIN itself does not encrypt credentials. Best for simple internal environments where certificate management is not feasible.

SASL/PLAIN SETUP:

BROKER CONFIG:
  listeners=SASL_SSL://broker1:9094
  security.inter.broker.protocol=SASL_SSL
  sasl.mechanism.inter.broker.protocol=PLAIN
  sasl.enabled.mechanisms=PLAIN
  listener.name.sasl_ssl.plain.sasl.jaas.config=\
    org.apache.kafka.common.security.plain.PlainLoginModule required \
    username="admin" \
    password="admin-secret" \
    user_admin="admin-secret" \
    user_alice="alice-secret" \
    user_bob="bob-secret";

CLIENT CONFIG:
  security.protocol=SASL_SSL
  sasl.mechanism=PLAIN
  sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
    username="alice" \
    password="alice-secret";

SASL/SCRAM: Salted Challenge Response

SCRAM (Salted Challenge Response Authentication Mechanism) is more secure than PLAIN. Passwords are hashed and salted — never transmitted in plaintext even inside SASL. Credentials are stored in ZooKeeper or KRaft metadata and can be added/removed dynamically without broker restart. Recommended over PLAIN for production use.

SASL/SCRAM USER MANAGEMENT:

# Create a user (store credentials in Kafka):
bin/kafka-configs.sh \
  --bootstrap-server localhost:9092 \
  --alter \
  --add-config 'SCRAM-SHA-256=[iterations=8192,password=alice-secret]' \
  --entity-type users \
  --entity-name alice

# Delete a user:
bin/kafka-configs.sh \
  --bootstrap-server localhost:9092 \
  --alter \
  --delete-config 'SCRAM-SHA-256' \
  --entity-type users \
  --entity-name alice

# List all users:
bin/kafka-configs.sh \
  --bootstrap-server localhost:9092 \
  --describe \
  --entity-type users

SASL/GSSAPI (Kerberos): Enterprise Authentication

Kerberos is the standard authentication protocol in enterprise environments with Active Directory or MIT Kerberos. It issues time-limited tickets (tokens) that clients present to prove identity without sending passwords over the network. Use GSSAPI in enterprises where Kerberos is already the authentication standard.

SASL/OAUTHBEARER: Token-Based Authentication

OAuth 2.0 bearer tokens allow clients to authenticate using tokens issued by an external identity provider (like Okta, Auth0, or Azure AD). The client obtains a token from the IdP and presents it to Kafka. Kafka validates the token (locally or via a token validation endpoint). Ideal for cloud-native environments and microservices architectures where OAuth/OIDC is the authentication standard.

Authorization with Access Control Lists (ACLs)

Once a client is authenticated, ACLs determine what it can do. An ACL entry specifies: who (principal/user), can perform what action (read, write, create, delete, describe), on what resource (topic, consumer group, cluster), from where (any host or specific IP).

ACL Structure

ACL FORMAT:
  Principal: User:alice
  Permission: ALLOW (or DENY)
  Operation: READ / WRITE / CREATE / DELETE / ALTER / DESCRIBE / ALL
  Resource: Topic:orders / Group:analytics-team / Cluster
  Host: * (any host) or specific IP

EXAMPLE ACLs:

Allow alice to produce to orders topic from any host:
  Principal=User:alice, Permission=ALLOW, Operation=WRITE, Resource=Topic:orders, Host=*

Allow bob to consume from orders topic as part of warehouse-group:
  Principal=User:bob, Permission=ALLOW, Operation=READ, Resource=Topic:orders, Host=*
  Principal=User:bob, Permission=ALLOW, Operation=READ, Resource=Group:warehouse-group, Host=*

Deny carol from deleting any topic:
  Principal=User:carol, Permission=DENY, Operation=DELETE, Resource=Topic:*, Host=*

Managing ACLs with the Command Line

# Grant alice write access to orders topic:
bin/kafka-acls.sh \
  --bootstrap-server localhost:9094 \
  --command-config admin.properties \
  --add \
  --allow-principal User:alice \
  --operation Write \
  --topic orders

# Grant bob read access to orders topic AND the consumer group:
bin/kafka-acls.sh \
  --bootstrap-server localhost:9094 \
  --command-config admin.properties \
  --add \
  --allow-principal User:bob \
  --operation Read \
  --topic orders \
  --group warehouse-group

# List all ACLs:
bin/kafka-acls.sh \
  --bootstrap-server localhost:9094 \
  --command-config admin.properties \
  --list

# Remove an ACL:
bin/kafka-acls.sh \
  --bootstrap-server localhost:9094 \
  --command-config admin.properties \
  --remove \
  --allow-principal User:alice \
  --operation Write \
  --topic orders

Enable ACLs on the Broker

ACLs are enforced by the Kafka authorizer. Enable it in server.properties:

# For ZooKeeper-based clusters:
authorizer.class.name=kafka.security.authorizer.AclAuthorizer
super.users=User:admin   ← admin bypasses all ACL checks

# For KRaft clusters:
authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer
super.users=User:admin

Security Best Practices

Principle of Least Privilege

Grant each client exactly the permissions it needs and nothing more. A producer for the orders topic needs WRITE on orders only — not READ, not access to other topics, not cluster admin rights. A consumer for the orders topic needs READ on orders and READ on its consumer group — nothing else.

PRINCIPLE OF LEAST PRIVILEGE EXAMPLE:

Order service (producer):
  ✓ WRITE on topic: orders
  ✗ No read access
  ✗ No access to payments, users, or other topics

Warehouse service (consumer):
  ✓ READ on topic: orders
  ✓ READ on group: warehouse-group
  ✗ No write access
  ✗ No access to other topics

Admin user:
  ✓ ALL on Cluster (can create/delete topics, manage ACLs)
  ← Only used for administration, not application code

Separate Internal and External Listeners

Use different listeners for different types of clients. Internal broker-to-broker traffic can use SASL_PLAINTEXT (within a trusted internal network). External client traffic uses SASL_SSL (encrypted and authenticated).

MULTI-LISTENER CONFIGURATION:

listeners=INTERNAL://0.0.0.0:9092,EXTERNAL://0.0.0.0:9094,ADMIN://0.0.0.0:9096

INTERNAL: SASL_PLAINTEXT (inter-broker, trusted network)
EXTERNAL: SASL_SSL (client connections from app servers)
ADMIN:    SASL_SSL (administrative tools only, firewalled)

Firewall rules:
  Port 9092: open only to other broker IPs
  Port 9094: open to application server IP ranges
  Port 9096: open only to admin workstation IPs

Regular Credential Rotation

Rotate SCRAM passwords and SSL certificates regularly. SCRAM passwords can be updated dynamically without broker restart. SSL certificates require reloading — plan for this with rolling restarts or certificate hot-reloading if your environment supports it.

Network Security Complementing Kafka Security

Kafka's built-in security mechanisms complement but don't replace network security. Always deploy Kafka clusters in private network segments (VPC/VLAN). Use firewalls or security groups to restrict which IP ranges can reach broker ports. Never expose Kafka brokers directly to the public internet. Use a proxy or API gateway if external access is required.

Key Points

  • Kafka security has three pillars: encryption (TLS), authentication (SASL), and authorization (ACLs). All three must be configured for a secure production deployment.
  • TLS encrypts data in transit. Mutual TLS (mTLS) also authenticates clients via certificates.
  • SASL mechanisms: PLAIN (simple, over TLS only), SCRAM (secure hashed, recommended), GSSAPI/Kerberos (enterprise AD), OAuthBearer (cloud-native tokens).
  • ACLs define what each authenticated user can do on which resource. Enable the Kafka authorizer and configure super.users for admin bypass.
  • Apply the principle of least privilege: grant each service only the exact permissions it needs.
  • Use separate listeners for internal broker traffic and external client traffic. Firewall Kafka ports to the appropriate IP ranges.
Previous lessons
Back to courses
Next lessons

Leave a Comment