ADE Event Hubs and Stream Analytics

Some data cannot wait for a nightly batch. Sensors, apps, and payment systems produce events every second, and businesses want answers while the events still matter. Azure Event Hubs collects these events at massive scale. Azure Stream Analytics reads them in motion and produces results within moments.

Event Hubs in Plain Words

Event Hubs works like a multi-lane highway for data. Producers, such as apps and devices, drive events onto the highway. Consumers, such as analytics jobs, exit at their own pace. The highway never forces producers to wait for consumers.

PRODUCERS                    EVENT HUB                       CONSUMERS

Parking sensors  --->   Partition 0  [e1][e4][e7]  --->   Stream Analytics job
Mobile app       --->   Partition 1  [e2][e5][e8]  --->   Databricks streaming
Payment system   --->   Partition 2  [e3][e6][e9]  --->   Capture to data lake

Core Event Hubs Ideas

• Namespace: A container that holds one or more event hubs and shares settings among them.
• Partition: One lane of the highway. Events inside a partition keep their order, and more partitions allow more parallel readers.
• Consumer group: A separate view of the stream. Each group tracks its own reading position, so several applications read the same events independently.
• Retention: The time that Event Hubs keeps events for replay. Choose a period that matches how long consumers might lag behind.
• Capture: A built-in feature that writes events to Azure Data Lake Storage Gen2 or Blob Storage in Avro format.

Event Hubs also exposes a Kafka-compatible endpoint, so many Kafka applications connect with only a configuration change.

Stream Analytics in Plain Words

Stream Analytics is a fully managed service that runs SQL-like queries over live events. A job has three parts: one or more inputs, a query, and one or more outputs.

INPUT                    QUERY                          OUTPUT

Event Hubs   --->   SELECT ... FROM input   --->   Azure SQL Database
IoT Hub              GROUP BY window                 Data Lake Storage
Blob Storage                                         Power BI dashboard

Windows: Counting Inside Time Slices

A stream never ends, so an average over "all events" has no meaning. A window cuts the stream into time slices and calculates inside each slice.

Window	Behavior	Typical Question
Tumbling	Fixed slices that never overlap	How many cars entered each 5-minute period?
Hopping	Fixed slices that overlap	What is the 10-minute average, refreshed every 2 minutes?
Sliding	Slice produced whenever an event enters or leaves	Did more than 100 failed logins happen within any 1 minute?
Session	Slice that grows until a quiet gap appears	How long did each user visit last?

Scenario: Smart Parking Lots

CityPark, a fictional operator, installs entry sensors in 40 parking lots. Each sensor sends an event for every arriving car. The control room wants a live count of arrivals per lot every five minutes.

SELECT
    LotId,
    COUNT(*) AS ArrivalCount,
    System.Timestamp() AS WindowEnd
INTO
    LiveArrivalsOutput
FROM
    SensorInput TIMESTAMP BY EventTime
GROUP BY
    LotId,
    TumblingWindow(minute, 5)

The TIMESTAMP BY clause tells the job to use the time the car arrived, which may differ from the time the event reached Azure. System.Timestamp() returns the end of each window.

Late and Out-of-Order Events

Networks delay events, and a sensor may report an arrival from two minutes ago. Stream Analytics offers a late arrival tolerance and an out-of-order tolerance. Each setting tells the job how long to wait before closing a window. Longer waits improve accuracy and add delay to the results.

Scaling with Streaming Units

A streaming unit bundles compute and memory for a job. Add units to handle more events. Partition the query by the same key as the input partitions, and the job runs in parallel across the lanes of the highway.

Stream Analytics or Spark Streaming

Need	Better Fit
Quick SQL-only logic with little code	Stream Analytics
Complex Python logic or machine learning	Spark Structured Streaming
Raw archive of every event	Event Hubs Capture

Key Points

• Event Hubs ingests millions of events per second through partitions.
• Consumer groups let several applications read the same stream independently.
• Stream Analytics applies SQL-like queries and time windows to live data.
• Tumbling, hopping, sliding, and session windows answer different time questions.