Cassandra with Kafka
Apache Kafka is a distributed event-streaming platform. Cassandra is a distributed database built for high-speed writes. These two systems complement each other naturally: Kafka captures and buffers real-time event streams, and Cassandra stores and serves that data at scale. Together they power architectures for IoT pipelines, clickstream analytics, financial transaction processing, and real-time dashboards.
Why Combine Kafka and Cassandra
Kafka Strengths Cassandra Strengths ────────────────────────────────────────────────────────────── Durable event log Fast, scalable writes High-throughput ingestion Flexible data modeling Decoupled producers/consumers Geo-distributed storage Replay events from any offset Always-on availability Buffering for backpressure Sub-millisecond read latency
Common Integration Patterns
Pattern Flow
──────────────────────────────────────────────────────────────────────
Kafka → Cassandra Kafka consumer reads events and
(stream ingestion) writes to Cassandra tables
Cassandra → Kafka Change Data Capture (CDC) sends
(change data capture) Cassandra mutations to Kafka topics
Kafka ↔ Cassandra (bidirectional) Hybrid: read from Cassandra for
enrichment, write results to Kafka
Pattern 1: Kafka Consumer Writing to Cassandra
A Kafka consumer reads messages from a topic and inserts them into Cassandra. This is the most common pattern for building real-time data stores.
IoT Device → Kafka Topic "sensor-readings" → Consumer App → Cassandra
Java Consumer + Cassandra Writer
import org.apache.kafka.clients.consumer.*;
import com.datastax.oss.driver.api.core.CqlSession;
import com.datastax.oss.driver.api.core.cql.PreparedStatement;
import java.time.Duration;
import java.util.*;
public class SensorConsumer {
public static void main(String[] args) {
// --- Kafka setup ---
Properties kafkaProps = new Properties();
kafkaProps.put("bootstrap.servers", "kafka-broker:9092");
kafkaProps.put("group.id", "sensor-cassandra-writer");
kafkaProps.put("key.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer");
kafkaProps.put("value.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer");
kafkaProps.put("auto.offset.reset", "earliest");
kafkaProps.put("enable.auto.commit", "false"); // manual commits
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(kafkaProps);
consumer.subscribe(List.of("sensor-readings"));
// --- Cassandra setup ---
CqlSession session = CqlSession.builder()
.withLocalDatacenter("us-east")
.withKeyspace("iot")
.build();
PreparedStatement insertReading = session.prepare(
"INSERT INTO sensor_readings (sensor_id, day, read_time, value) " +
"VALUES (?, ?, toTimestamp(now()), ?)"
);
// --- Consume and write loop ---
try {
while (true) {
ConsumerRecords<String, String> records =
consumer.poll(Duration.ofMillis(500));
for (ConsumerRecord<String, String> record : records) {
// Parse JSON payload: {"sensor_id":"S1","day":"2024-06-15","value":22.5}
// (Simplified: assume values are pre-parsed)
String sensorId = record.key();
String day = "2024-06-15";
double value = Double.parseDouble(record.value());
session.execute(insertReading.bind(sensorId, day, value));
}
consumer.commitSync(); // commit after successful Cassandra writes
}
} finally {
consumer.close();
session.close();
}
}
}
Pattern 2: Kafka Connect — Cassandra Sink Connector
Kafka Connect is a framework for streaming data between Kafka and external systems without writing custom consumer code. The DataStax Kafka Connector (or community alternatives) acts as a Cassandra Sink — it reads from Kafka topics and writes to Cassandra tables automatically.
Kafka Topic → Kafka Connect → DataStax Kafka Connector → Cassandra Table
Connector Configuration (JSON)
{
"name": "cassandra-sink-sensor",
"config": {
"connector.class":
"com.datastax.oss.kafka.sink.CassandraSinkConnector",
"tasks.max": "4",
"topics": "sensor-readings",
"contactPoints": "10.0.0.1,10.0.0.2",
"loadBalancing.localDc": "us-east",
"auth.username": "alice",
"auth.password": "Al1ce$ecure!",
"topic.sensor-readings.iot.sensor_readings.mapping":
"sensor_id=value.sensor_id, day=value.day, value=value.reading",
"topic.sensor-readings.iot.sensor_readings.consistencyLevel":
"LOCAL_QUORUM",
"topic.sensor-readings.iot.sensor_readings.ttl": "2592000"
}
}
Deploy the Connector
curl -X POST http://kafka-connect:8083/connectors \ -H "Content-Type: application/json" \ -d @cassandra-sink-config.json
Pattern 3: Cassandra Change Data Capture (CDC) → Kafka
Cassandra CDC captures every mutation (INSERT, UPDATE, DELETE) and writes it to a CDC log directory. A CDC reader (such as the Debezium Cassandra connector) reads from this log and publishes changes to Kafka topics, enabling downstream consumers to react to data changes.
Cassandra mutation
│ CDC log (/var/lib/cassandra/cdc_raw/)
▼
Debezium Cassandra Source Connector
│ Kafka topic "cassandra.ecommerce.orders"
▼
Downstream consumers (analytics, search index, notifications)
Enable CDC on a Table
-- Enable CDC in cassandra.yaml: -- cdc_enabled: true -- cdc_raw_directory: /var/lib/cassandra/cdc_raw -- Enable CDC on the table: ALTER TABLE ecommerce.orders WITH cdc = true;
Exactly-Once Semantics
Kafka guarantees at-least-once delivery by default. When writing to Cassandra, duplicate Kafka messages cause duplicate inserts. Because Cassandra INSERT is an upsert, duplicate inserts with the same primary key are idempotent — the second write simply overwrites the first with the same values. This makes Kafka-to-Cassandra pipelines naturally resilient to duplicates when the primary key is deterministic (derived from the event, not randomly generated).
Event: { order_id: "ORD-001", total: 99.99 }
→ Kafka delivers twice (at-least-once)
Write 1: INSERT INTO orders (order_id, total) VALUES ('ORD-001', 99.99)
Write 2: INSERT INTO orders (order_id, total) VALUES ('ORD-001', 99.99)
→ Second write is a no-op: same PK, same values
Result: exactly one row with correct data ✓
Throughput Tuning
Tuning Parameter Recommendation
──────────────────────────────────────────────────────────────
Kafka consumer tasks.max Set to number of Cassandra nodes
or topic partition count
Cassandra write consistency LOCAL_ONE for max throughput
Batch Kafka records Write multiple records per
Cassandra UNLOGGED BATCH per partition
Cassandra async writes Use execute_async; wait for futures
in groups of 100–500
Consumer poll interval 100–500 ms depending on latency needs
Typical End-to-End Architecture
Sources Kafka Cassandra ────────────────────────────────────────────────────────────── Mobile Apps ──▶ Topic: user-events ──▶ Table: user_activity IoT Devices ──▶ Topic: sensor-data ──▶ Table: sensor_readings Payment Service ──▶ Topic: payments ──▶ Table: payment_ledger Web Servers ──▶ Topic: page-views ──▶ Counter: page_views
Summary
Kafka and Cassandra work together by combining Kafka's durable, high-throughput event streaming with Cassandra's fast, distributed storage. The three integration patterns are: writing a custom Kafka consumer that inserts into Cassandra, using Kafka Connect with the DataStax Sink Connector for a no-code pipeline, and using Cassandra CDC with Debezium to publish Cassandra mutations to Kafka. Cassandra's upsert behavior makes the pipeline naturally tolerant of Kafka's at-least-once delivery. Tune throughput by using LOCAL_ONE consistency, async writes, and multiple consumer tasks aligned with partition count.