DE Stream Processing
Some data problems cannot wait for a batch job to run at midnight. A bank needs to block a fraudulent transaction before it completes. A ride-hailing app needs to match a driver to a passenger within seconds. A factory floor needs to detect a machine failure before damage spreads. These problems require stream processing — the ability to act on data the moment it arrives.
What Is Stream Processing
Stream processing handles data as a continuous, never-ending flow of individual events. Each event is processed as soon as it arrives — there is no waiting period, no accumulation phase. The system reads, processes, and produces results for each event in near real time.
The River Analogy
Batch processing is like collecting rainwater in a bucket, then pouring it through a filter once the bucket is full. Stream processing is like standing in a river with a filter and cleaning the water as it flows past you — one drop at a time, continuously. The stream never stops, and neither does the processing.
Batch vs Stream: Key Difference
Batch Processing:
Events --> [Accumulate for 24 hours] --> [Process all at once] --> Results
(runs once, finishes)
Stream Processing:
Event 1 --> [Process] --> Result 1
Event 2 --> [Process] --> Result 2 (runs forever, continuously)
Event 3 --> [Process] --> Result 3
How Stream Processing Works
Events and Event Streams
An event is a single occurrence: a user clicked a button, a sensor read a temperature, a payment was submitted. Events flow through a message queue or event streaming platform (like Apache Kafka). The stream processing system reads from that queue, processes each event, and writes results to an output destination.
Processing Steps
Stream processing systems apply operations to each event or to small groups of recent events. Common operations include filtering (keep only events matching a condition), enriching (adding data from a lookup table), aggregating (counting events per minute), and joining (combining events from two streams).
Windowing: Grouping Events Over Time
Many stream operations need to consider multiple events together — for example, counting clicks per minute or detecting a drop in sensor readings over the last 10 seconds. Windowing groups events by time so the processing system can compute across a set of related events.
Tumbling Window
Fixed, non-overlapping time periods. A 1-minute tumbling window groups all events from 12:00 to 12:01 together, then all events from 12:01 to 12:02 separately.
Sliding Window
Overlapping windows that slide forward by a step smaller than the window size. A 5-minute window that slides every 1 minute gives a new result every minute, each including the previous 5 minutes of events.
Session Window
Groups events by activity sessions. A session window opens when a user becomes active and closes after a defined period of inactivity. Useful for tracking user sessions on a website.
Tumbling (1-min windows):
|--12:00-12:01--|--12:01-12:02--|--12:02-12:03--|
Sliding (5-min window, 1-min slide):
|------12:00-12:05------|
|------12:01-12:06------|
|------12:02-12:07------|
Stream Processing Frameworks
Framework | Language | Key Feature -----------------------|----------|------------------------------------- Apache Kafka Streams | Java | Built into Kafka; lightweight Apache Flink | Java | Low latency; true event-time processing Apache Spark Streaming | Python | Micro-batch; integrates with Spark Google Dataflow | Python | Serverless; unified batch and stream Apache Storm | Java | Oldest streaming framework; very fast
Real-World Stream Processing Use Cases
Fraud Detection
A bank processes each credit card transaction as a stream event. The processing system checks each transaction against rules: Is the location unusual? Has this card made more than five transactions in one minute? Does the amount exceed the user's typical spending pattern? If suspicious, the system flags or blocks the transaction in under a second.
Real-Time Inventory
An e-commerce platform processes each sale as a stream event. When a product sells, the stream processing system immediately decrements the inventory count in the live database. The website shows accurate stock levels in real time, preventing overselling.
Live Dashboards
A call center tracks the number of active calls, average wait time, and agent availability in real time. A stream processing pipeline aggregates call events as they happen and pushes updates to the dashboard every few seconds.
Challenges in Stream Processing
Late-Arriving Data
Events do not always arrive in the order they occurred. A mobile app event created at 12:01 might arrive at the processing system at 12:05 due to a network delay. Stream processing systems must decide whether to wait for late events or process what has arrived and handle corrections later.
Exactly-Once Processing
If a processing system crashes and restarts, it might reprocess events it already handled. This can create duplicate results — for example, counting the same transaction twice in a fraud check. Modern frameworks like Kafka Streams and Flink support exactly-once semantics to prevent this, but at a performance cost.
Stateful Processing
Some operations need to remember previous events. Counting clicks per user per hour requires the system to maintain a running count for each user across all events in the hour. Managing this state reliably across restarts and failures is one of the hardest challenges in stream processing.
When to Choose Streaming Over Batch
Use Batch When: Use Streaming When: - Data freshness > 1 hour - Need results within seconds - Complex historical joins - Fraud, anomaly detection - Reporting and analytics - Real-time dashboards - Simpler to build/maintain - Instant inventory updates
Summary
Stream processing handles data as a continuous flow of events, producing results in near real time. Windowing groups events over time for aggregate operations. Frameworks like Flink and Kafka Streams power production streaming systems. Stream processing is more complex than batch processing but essential when data freshness of seconds or milliseconds matters to the business.