DE Pipeline Monitoring and Alerting
A data pipeline that runs silently without observation is a liability. Failures go undetected, data arrives late, records are silently dropped, and downstream teams discover problems only when a report shows wrong numbers hours after the fact. Pipeline monitoring gives data engineers visibility into what every pipeline is doing at every moment. Alerting ensures problems surface immediately, before they reach data consumers.
What Pipeline Monitoring Covers
Pipeline monitoring tracks three dimensions: availability (is the pipeline running?), timeliness (did it finish on time?), and correctness (did it produce the right data?). A pipeline can succeed on availability while failing on timeliness or correctness — all three dimensions require independent monitoring.
The Power Grid Analogy
An electrical grid operator does not assume power is flowing simply because generators are running. Sensors throughout the grid monitor current, voltage, and frequency at every node. Control rooms display real-time dashboards. Automated systems trigger alerts and circuit breakers the moment a reading falls outside normal range — before customers experience an outage. Data pipeline monitoring works identically: instrument every stage, display real-time status, and alert automatically when something deviates from expected behavior.
Key Metrics to Monitor
Pipeline Run Metrics
Metric | Description | Alert Condition ------------------------|-----------------------------------|--------------------------- Pipeline run status | Success, failed, or still running | Status = FAILED Run duration | How long the pipeline took | Duration > 2x baseline Start time | When the pipeline actually started| Start delayed > 30 min SLA compliance | Did it finish before the deadline | Finish time > SLA time Retry count | How many retries before success | Retries > threshold
Data Volume Metrics
Metric | Description | Alert Condition ------------------------|----------------------------------|--------------------------- Rows ingested | Records pulled from source | < 50% or > 200% of average Rows loaded | Records written to destination | Mismatch with ingested rows Rows rejected | Records failing validation | Rejection rate > 1% Files processed | Count of source files handled | Count = 0 (no data arrived)
Data Freshness Metrics
Metric | Description | Alert Condition ------------------------|----------------------------------|--------------------------- Max event timestamp | Latest data timestamp in table | > 2 hours old for daily data Last successful load | When data last updated | Not updated in expected window Lag from source | Gap between source and dest time | Lag > defined threshold
Logging: The Foundation of Observability
Every pipeline step should emit structured log messages that record what happened and when. Structured logs (JSON format) allow log aggregation tools to filter, search, and alert on specific conditions automatically.
Structured Log Entry Example:
{
"timestamp": "2024-05-15T03:14:22Z",
"pipeline": "daily_orders_etl",
"run_id": "run_20240515_0300",
"step": "extract_orders",
"status": "SUCCESS",
"rows_extracted": 84302,
"duration_seconds": 47,
"source": "mysql://orders_db/orders",
"watermark_from": "2024-05-14T23:59:59Z",
"watermark_to": "2024-05-15T23:59:59Z"
}
Log aggregation platforms — Datadog, Splunk, AWS CloudWatch, Google Cloud Logging — ingest these structured logs and provide search, dashboards, and alert rules based on any field.
Metrics Dashboards
A pipeline monitoring dashboard gives the on-call engineer an at-a-glance view of every pipeline's current health. Effective dashboards show current run status, last successful run time, SLA countdown, row count trends, and error counts — all on one screen.
Pipeline Health Dashboard (example layout): Pipeline Name | Last Run | Status | Duration | SLA | Rows ---------------------|--------------|----------|----------|----------|---------- daily_orders_etl | 03:14 | SUCCESS | 14 min | ✓ Met | 84,302 customer_enrichment | 04:01 | FAILED | 3 min | ✗ Missed | -- product_catalog_sync | 05:00 | SUCCESS | 8 min | ✓ Met | 12,451 revenue_rollup | 05:45 | RUNNING | 22 min | ⚠ At Risk| --
SLA Monitoring
SLAs (Service Level Agreements) define when a pipeline's output must be available. A business might require that sales data be in the warehouse by 6 AM so executives can review it over their morning coffee. SLA monitoring tracks whether each pipeline completed before its deadline and calculates SLA compliance rates over time.
SLA Configuration:
Pipeline: daily_orders_etl
SLA deadline: 05:00 UTC
Alert at: 04:30 UTC if pipeline has not completed
(30-minute warning before SLA breach)
SLA Breach Response:
04:30: Warning alert -- "Pipeline still running, SLA risk"
05:00: SLA breach alert -- "Pipeline missed SLA deadline"
Pages on-call engineer immediately
Alerting Channels and Escalation
Alerts must reach the right person through the right channel at the right urgency level. Different severity levels warrant different responses.
Alert Severity Levels:
INFO: Informational only; no action required
Channel: Log file, dashboard
WARNING: Investigate but not urgent
Channel: Slack notification
CRITICAL: Immediate action required; SLA at risk
Channel: PagerDuty (pages on-call engineer's phone)
EMERGENCY: Major data loss or corruption detected
Channel: PagerDuty + escalate to manager
Anomaly Detection
Static thresholds alert when a metric crosses a fixed boundary. Anomaly detection alerts when a metric deviates significantly from its own historical pattern — useful when "normal" varies by day of week or time of month.
Static Threshold: Alert if rows < 50,000
Problem: Monday might normally produce 30,000 rows (no alert fired)
Friday might normally produce 150,000 rows (threshold too low)
Anomaly Detection: Alert if rows deviate > 3 standard deviations
from the historical average for this day of week
Monday avg: 30,000 ± 2,000 -- alert if < 24,000 or > 36,000
Friday avg: 150,000 ± 8,000 -- alert if < 126,000 or > 174,000
Monitoring Tools
Tool | Best For -------------------|------------------------------------------ Apache Airflow UI | DAG run status, task logs Datadog | Metrics, logs, traces, anomaly detection Monte Carlo | Data quality anomalies; table freshness Grafana + Prometheus| Custom metrics dashboards PagerDuty | On-call alerting and escalation AWS CloudWatch | AWS-native pipeline monitoring
Summary
Pipeline monitoring tracks availability, timeliness, and data correctness through structured logs, row count metrics, freshness checks, and SLA tracking. Dashboards give engineers at-a-glance visibility across all pipelines simultaneously. Alerting — from Slack warnings to PagerDuty pages — ensures problems surface immediately and reach the right person at the right urgency level. Anomaly detection extends static thresholds to handle naturally variable workloads. A well-monitored pipeline catches problems in minutes; an unmonitored pipeline lets problems fester for hours before anyone notices.