DE Lambda and Kappa Architecture
Every data engineering team eventually faces the same question: how do you serve both historical batch analytics and real-time streaming results from the same system? Two architectural patterns answer this question differently. Lambda architecture runs parallel batch and streaming systems and merges their outputs. Kappa architecture eliminates the batch layer entirely and processes everything as a stream. Both patterns appear frequently in senior-level data engineering design discussions.
The Core Problem Both Solve
Streaming systems deliver low-latency results but often produce approximate or incomplete answers — late-arriving events haven't processed yet, and the last few minutes of data may be incomplete. Batch systems deliver accurate, complete results but with hours of delay. Organizations want both: real-time dashboards showing live numbers and accurate historical reports showing final numbers. Lambda and Kappa are two different answers to this tension.
Lambda Architecture
Lambda architecture, proposed by Nathan Marz around 2011, separates processing into three distinct layers: the batch layer, the speed layer, and the serving layer.
The Batch Layer
The batch layer processes all historical data periodically — typically nightly. It recomputes views over the complete dataset and produces accurate, final results. Because it processes everything from scratch on every run, it corrects any errors the speed layer introduced from incomplete data. The batch layer uses systems like Spark, Hive, or dbt running against a data lake or warehouse.
The Speed Layer
The speed layer processes incoming events in real time. It fills the gap between the last batch run and the current moment. Because it handles only recent data and operates under time pressure, its results are approximate — based on whatever has arrived so far. The speed layer uses systems like Flink, Kafka Streams, or Spark Structured Streaming.
The Serving Layer
The serving layer merges batch and speed layer outputs to answer queries. For historical time periods already covered by the batch layer, it serves the accurate batch result. For the most recent period not yet covered by a batch run, it serves the speed layer's approximate result.
Lambda Architecture Diagram:
[All Data Sources]
|
+-----------------> [Batch Layer]
| (Spark/dbt on data lake)
| Runs nightly; produces
| accurate historical views
| |
+-> [Speed Layer] |
(Flink/Kafka) |
Processes real-time |
last few hours |
| |
v v
[Serving Layer] <---+
(Merges batch + speed results)
|
v
[Queries / Dashboards]
User query for "revenue today":
- Historical hours (midnight to 2 AM): Batch layer result (accurate)
- Most recent hours (2 AM to now): Speed layer result (approximate)
- Combined: Full day view with real-time tail
Lambda Architecture Advantages
Lambda provides strong fault tolerance. If the speed layer produces wrong results due to a bug, the next batch run overwrites them with correct values. The batch layer acts as the source of truth that always corrects mistakes. This design also allows teams to use best-fit tools for each layer — optimizing batch separately from streaming.
Lambda Architecture Disadvantages
Running two parallel systems doubles complexity, cost, and maintenance burden. The same business logic — revenue calculation, customer segmentation — must be implemented twice: once in the batch layer and once in the speed layer. When business logic changes, both implementations must stay synchronized. Inconsistencies between layers create confusing situations where real-time and historical dashboards show different numbers for the same time period. This "two codebases" problem is Lambda's most serious operational challenge.
Kappa Architecture
Kappa architecture, proposed by Jay Kreps (co-creator of Kafka) in 2014, eliminates the batch layer entirely. It processes everything — both historical and real-time data — through a single streaming pipeline. Historical data replays through the same streaming system that handles live data.
Kappa Architecture Diagram:
[All Data Sources]
|
v
[Event Log / Kafka]
(Stores ALL historical events with long retention)
|
v
[Single Streaming Pipeline]
(Flink or Kafka Streams)
Processes live events AND replays historical events
through the same code path
|
v
[Serving Layer / Data Store]
|
v
[Queries / Dashboards]
Historical reprocessing:
Start a NEW instance of the same streaming job
Point it at offset 0 (beginning of Kafka log)
It replays all history at high speed
Output overwrites old results
How Historical Reprocessing Works in Kappa
Kappa stores all events in Kafka with long retention — days, weeks, or indefinitely using Kafka's log compaction or tiered storage. When business logic changes and historical data needs recomputing, engineers start a new streaming job instance pointing at offset 0 of the relevant Kafka topic. This job replays history at maximum speed (Kafka delivers stored events much faster than they originally arrived), computes new results, and writes them to a new output table. Once reprocessing completes, the serving layer switches to the new table. The process treats batch recomputation as simply a fast replay of the event stream.
Kappa Architecture Advantages
One codebase handles both historical and real-time processing. Business logic changes require updating one system, not two. Operational complexity is lower — one pipeline to monitor, one set of tools to master, one infrastructure to maintain. Results from real-time and historical processing are definitionally consistent because they use the same code.
Kappa Architecture Disadvantages
Kappa requires storing all events in the event log long enough to support historical reprocessing — potentially years of events. Kafka with tiered storage or an event lake in S3 makes this feasible but adds storage cost. Reprocessing years of history through a streaming pipeline can take hours to days depending on event volume and pipeline complexity. Complex historical joins across multiple streams are also harder to express in streaming code than in batch SQL.
Lambda vs Kappa: Side-by-Side
Factor | Lambda Architecture | Kappa Architecture
----------------------|------------------------|-------------------------
Layers | Batch + Speed + Serving| Stream + Serving only
Code duplication | High (two systems) | None (one system)
Operational complexity| High | Lower
Historical accuracy | Batch layer guarantees | Replay guarantees
Reprocessing method | Rerun batch job | Replay from event log
Late data handling | Batch corrects speed | Reprocess from Kafka
Storage requirement | Data lake + stream | Long-retention event log
Best for | Teams with existing | Greenfield streaming-first
| batch investments | architectures
The Modern Lakehouse Alternative
Modern lakehouse architectures using Delta Lake or Apache Iceberg have blurred the Lambda/Kappa distinction for many teams. A streaming pipeline writes events continuously to a Delta table in a data lake. When reprocessing is needed, a Spark batch job rewrites affected partitions in the same Delta table. The serving layer reads from one table regardless of whether data arrived via stream or batch reprocessing. This hybrid approach retains the simplicity of Kappa's single output store while accommodating the occasional need for batch recomputation.
Modern Lakehouse Hybrid: Real-time path: Kafka --> Spark Structured Streaming --> Delta Lake table (streaming write) Reprocessing path: S3 raw files --> Spark batch job --> Delta Lake table (overwrite partitions) Serving path: Delta Lake table --> BI Tools / Analysts (always one consistent source)
Choosing Between Lambda and Kappa
Choose Lambda when: - Team already has mature batch infrastructure - Streaming results need guaranteed correction by batch - Business cannot tolerate approximate real-time numbers - Budget for running two parallel systems exists Choose Kappa when: - Building a new streaming-first architecture - Team has strong streaming expertise (Flink/Kafka) - Kafka retention budget is available for full history - Minimizing code duplication is a priority Consider Lakehouse hybrid when: - Already using Delta Lake or Iceberg - Need occasional batch reprocessing alongside streaming - Want one consistent output table for all consumers
Summary
Lambda architecture runs batch and streaming layers in parallel — the batch layer delivers accurate historical results, the speed layer delivers real-time approximations, and the serving layer merges both. The cost is duplicate business logic and double operational complexity. Kappa architecture eliminates the batch layer and processes everything through a single streaming pipeline, using event log replay for historical reprocessing. Modern lakehouse architectures offer a pragmatic hybrid. Data engineers choose between these patterns based on existing infrastructure, team expertise, data freshness requirements, and tolerance for operational complexity.