Databricks Monitoring

A pipeline that works today can fail tomorrow. A source file arrives late, a query slows down, or a cluster runs out of memory. Monitoring turns surprises into early warnings. Teams watch jobs, queries, data quality, and costs, and they act before users notice a problem.

The Hospital Monitor Analogy

A hospital monitor tracks heart rate and oxygen level all day. A nurse does not stare at the screen constantly, because alarms ring when a number crosses a safe limit. Data platforms work best with the same design. Dashboards show the normal state, and alerts call for attention when something drifts.

MONITORING LAYERS
──────────────────────────────────────────────────────────
JOBS         Did the run succeed? How long did it take?
QUERIES      Which queries run slowly?
DATA         Are values complete and fresh?
COST         Is spending in line with the plan?
SECURITY     Who accessed what?

Each layer ──► dashboard for the normal view
           ──► alert for the abnormal view

Monitor Jobs

Every job run records its status, start time, duration, and error message. The Jobs page shows the history of runs for each job. Teams add notifications so the right people hear about failures.

Notifications

Configure notifications for start, success, failure, and long duration. Destinations include email, Slack, Microsoft Teams, and other webhook tools.

Retries and Timeouts

A retry setting reruns a failed task a few times, which fixes short network problems. A timeout stops a task that runs far longer than normal, which prevents runaway costs.

JOB ALERT FLOW
──────────────────────────────────────────────────────────
Task fails ──► retry once ──► still failing
                                   │
                    notify the on-call engineer
                                   │
                  engineer reads logs ──► fixes ──► reruns

Monitor Queries and Compute

Query history lists each SQL statement with its duration, user, and rows read. Slow queries stand out in the list. The Spark UI shows stages, tasks, and memory use for a single run, which helps engineers find bottlenecks such as skewed data or too many small files. Cluster metrics display CPU, memory, and disk activity over time.

SQL Alerts

A SQL alert runs a query on a schedule and compares the result with a threshold. When the condition holds, the alert sends a notification.

-- Count orders from the last hour
SELECT COUNT(*) AS recent_orders
FROM main.shop.silver_orders
WHERE order_time >= CURRENT_TIMESTAMP() - INTERVAL 1 HOUR;

ALERT RULE
──────────────────────────────────────────────────────────
Check every 15 minutes
If recent_orders < 10  ──► send a message to the data team
Meaning: the order feed may have stopped.

Monitor Data Quality

Healthy pipelines deliver data that is fresh, complete, and correct. Teams track three simple signals.

• Freshness: the latest timestamp in the table stays within the expected delay.
• Volume: the daily row count stays near its normal range.
• Validity: key columns contain no unexpected blanks or impossible values.

Delta Live Tables expectations count rows that break a rule and record the numbers in the pipeline event log. Lakehouse Monitoring profiles a table over time, calculates statistics, and detects drift in the data.

FRESHNESS CHECK
──────────────────────────────────────────────────────────
SELECT MAX(order_time) AS latest_order
FROM main.shop.silver_orders;

Expected: within the last 30 minutes
Alert:    if older than 2 hours

System Tables

System tables expose platform activity as ordinary Unity Catalog tables. Analysts query them with SQL and build dashboards. Billing records and audit logs are two examples.

-- Recent sign-in and permission activity
SELECT event_time, user_identity.email, service_name, action_name
FROM system.access.audit
ORDER BY event_time DESC
LIMIT 50;

Billing data in system.billing.usage supports cost dashboards. Administrators need the right permissions to read system tables.

Build a Monitoring Dashboard

Panel	Source	Purpose
Job success rate	Job run history	Spot unstable pipelines
Average job duration	Job run history	Notice slowdowns
Table freshness	SQL query on key tables	Confirm data arrives on time
Daily row counts	SQL query	Detect missing or doubled loads
Spending by tag	Billing system table	Track budgets

Example: Order Pipeline Watch

An online store runs an hourly order pipeline. The team sets a job failure notification, a freshness alert on the silver table, and a daily row-count alert. One morning the freshness alert fires because a source export stopped. The team contacts the source owner, restarts the export, and reruns the pipeline within the hour.

ORDER PIPELINE WATCH
──────────────────────────────────────────────────────────
Hourly job ──► silver table ──► dashboard
     │              │
 failure alert  freshness alert
     └──────┬───────┘
            ▼
     Data team channel

Good Monitoring Habits

• Alert on problems that need action, and avoid noisy alerts that people ignore.
• Name an owner for every alert.
• Write a short runbook that lists the first steps to take.
• Review alert history monthly and tune thresholds.

Key Points

• Monitoring turns hidden failures into timely notifications.
• Job notifications, retries, and timeouts protect scheduled work.
• Query history, the Spark UI, and cluster metrics reveal performance problems.
• SQL alerts check data freshness and volume on a schedule.
• System tables expose billing and audit activity for analysis with SQL.
• Clear owners and runbooks make alerts useful.