DE Stream Processing with Flink
Apache Flink is the leading framework for stateful, fault-tolerant stream processing at scale. While Spark Streaming processes data in micro-batches, Flink processes each event individually as it arrives — delivering true event-at-a-time processing with millisecond latencies and exactly-once guarantees. Organizations that need real-time fraud detection, live dashboards, and instant anomaly alerts rely on Flink.
What Makes Flink Different
Three characteristics distinguish Flink from other stream processing systems: true event-at-a-time processing (not micro-batches), rich stateful computation where operators maintain and update state across millions of keys simultaneously, and exactly-once fault tolerance without sacrificing throughput. Flink was built from the ground up as a streaming system — batch processing is a special case of streaming in Flink's model, not the reverse.
The Air Traffic Control Analogy
An air traffic controller tracks every aircraft in real time. Each new radar blip triggers immediate evaluation: is this aircraft on course? Is it too close to another? Controllers maintain a mental model of every aircraft's state — altitude, speed, trajectory — updating it continuously as new signals arrive. Flink's stateful stream processing works identically: each event triggers evaluation against the current state for that key (aircraft), the state updates, and an output may or may not be emitted depending on the result.
Flink Architecture
JobManager
The JobManager is the master process. It receives job submissions, creates the execution plan, schedules tasks across the cluster, coordinates checkpoints, and handles failure recovery. Each Flink job has one JobManager.
TaskManagers
TaskManagers are the worker processes. Each TaskManager runs multiple task slots — processing threads that execute the actual data transformations. Flink scales by adding more TaskManagers to the cluster.
Checkpoints
Flink periodically saves the state of all operators to durable storage — a process called checkpointing. If a TaskManager fails, Flink restores all operators from the last successful checkpoint and replays events from the source since that checkpoint. This mechanism enables exactly-once fault tolerance without requiring engineers to write explicit recovery logic.
Flink Checkpoint Flow: Every 30 seconds (configurable): 1. Flink inserts a checkpoint barrier into the event stream 2. Each operator saves its current state to S3/HDFS when barrier passes 3. Source records the Kafka offset at the checkpoint moment On failure: 1. Flink detects failure 2. Restores all operator states from last checkpoint 3. Resets Kafka consumer offsets to the checkpoint moment 4. Replays events since checkpoint 5. Processing continues -- no data lost, no duplicates
Flink DataStream API
The DataStream API is Flink's primary programming interface for stream processing. Engineers define a pipeline of transformations on a stream, and Flink executes it continuously as events arrive.
Basic Flink DataStream Pipeline (Java-style Python API):
from pyflink.datastream import StreamExecutionEnvironment
from pyflink.datastream.connectors.kafka import KafkaSource
env = StreamExecutionEnvironment.get_execution_environment()
# Source: read from Kafka
kafka_source = KafkaSource.builder() \
.set_bootstrap_servers("kafka:9092") \
.set_topics("orders.created") \
.set_group_id("flink-order-processor") \
.build()
stream = env.from_source(kafka_source, ...)
# Transformations
result = stream \
.filter(lambda order: order['amount'] > 0) \
.map(lambda order: enrich_with_customer_data(order)) \
.key_by(lambda order: order['region']) \
.window(TumblingEventTimeWindows.of(Time.minutes(5))) \
.sum('amount')
# Sink: write to database
result.add_sink(jdbc_sink)
env.execute("Order Revenue Pipeline")
Event Time vs Processing Time
Two time concepts matter in stream processing. Processing time is when the event arrives at Flink — easy to use but produces inconsistent results when events arrive late or out of order. Event time is when the event actually occurred, embedded in the event data — more accurate but requires handling late-arriving events explicitly. Flink supports both, and most production pipelines use event time.
Time Concepts Illustrated:
Event occurred at: 14:00:01
Network delay: 4 minutes
Arrived at Flink: 14:04:01
Processing time windowing: puts this event in the 14:00-14:05 window
(based on when Flink received it)
Event time windowing: puts this event in the 14:00-14:05 window
(based on when it actually occurred -- same result here)
But if the event arrived at 14:06:01 (6-minute delay):
Processing time: puts it in the 14:05-14:10 window (WRONG window)
Event time: puts it in the 14:00-14:05 window (CORRECT window)
(if the watermark still allows late data for that window)
Watermarks
Watermarks signal to Flink how far behind real time the event stream is. A watermark of "current_time - 2 minutes" tells Flink to assume all events up to 2 minutes ago have arrived. When the watermark passes the end of a window, Flink closes that window and emits results. Events arriving after the watermark has passed their window are late and handled by a configurable allowed lateness policy.
Stateful Stream Processing
State in Flink means an operator can remember information across events. A fraud detection operator maintains a count of transactions per customer per minute — state keyed by customer_id. When a new transaction event arrives for customer C001, Flink looks up the state for C001, increments the count, and emits a fraud alert if the count exceeds the threshold.
Stateful Fraud Detection Example:
State: { customer_id: transaction_count_last_60_sec }
Event arrives: { customer_id: "C001", amount: 50 }
Lookup state for C001: count = 4
Increment: count = 5
Check: 5 < 10 (threshold) --> No alert
Event arrives: { customer_id: "C001", amount: 30 }
Lookup state for C001: count = 5
Increment: count = 6
Check: 6 < 10 --> No alert
... 4 more events for C001 in the same minute ...
Event arrives: { customer_id: "C001", amount: 20 }
Lookup state for C001: count = 10
Increment: count = 11
Check: 11 > 10 --> EMIT FRAUD ALERT for C001
Flink SQL
Flink SQL lets engineers write standard SQL queries that execute as continuous streaming computations. A SQL SELECT statement on a streaming table produces a continuously updated result set — not a one-time snapshot. This brings the accessibility of SQL to real-time streaming without writing Java or Python operator code.
Flink SQL: Real-time revenue aggregation
CREATE TABLE orders (
order_id STRING,
region STRING,
amount DECIMAL,
event_time TIMESTAMP(3),
WATERMARK FOR event_time AS event_time - INTERVAL '10' SECOND
) WITH ('connector' = 'kafka', 'topic' = 'orders.created', ...);
-- Continuous query: 5-minute tumbling window revenue per region
SELECT
region,
TUMBLE_START(event_time, INTERVAL '5' MINUTE) AS window_start,
SUM(amount) AS revenue,
COUNT(*) AS order_count
FROM orders
GROUP BY region, TUMBLE(event_time, INTERVAL '5' MINUTE);
-- This query runs forever, emitting a new result row every 5 minutes per region
Flink Use Cases in Data Engineering
Use Case | Why Flink --------------------------|--------------------------------------------- Real-time fraud detection | Stateful per-user event counting; low latency Live dashboards | Sub-second aggregations from Kafka to DB ETL with transformations | Enrich, filter, route events in real time Data lake ingestion | Write Kafka events to S3/GCS in Parquet format Change Data Capture sink | Apply CDC events to downstream tables Feature store updates | Compute ML features from live event streams
Flink vs Spark Streaming
Feature | Apache Flink | Spark Structured Streaming ---------------------|-----------------------|--------------------------- Processing model | True event-at-a-time | Micro-batch (seconds) Latency | Milliseconds | Seconds State management | Native, rich | Limited Exactly-once | Yes (native) | Yes (with idempotent sinks) SQL support | Flink SQL | Spark SQL Ecosystem maturity | Growing rapidly | Very mature; wide adoption Best for | Low-latency, stateful | Batch + stream in same code
Summary
Apache Flink delivers true event-at-a-time stream processing with rich stateful computation and exactly-once fault tolerance through periodic checkpointing. Event time processing with watermarks handles late-arriving data correctly. The DataStream API and Flink SQL provide flexible programming models for engineers at different levels of comfort with code. Flink excels at low-latency stateful workloads — fraud detection, live aggregations, real-time feature computation — where micro-batch systems like Spark Streaming introduce unacceptable latency.