DE Data Quality Fundamentals

A pipeline that reliably delivers bad data is worse than no pipeline at all. Business decisions made from incorrect data cost money, damage customer trust, and sometimes have regulatory consequences. Data quality is not a separate concern bolted onto the side of a pipeline — it is a responsibility embedded at every stage of data engineering work.

What Is Data Quality

Data quality measures how well data serves its intended purpose. High-quality data is accurate, complete, consistent, timely, and correctly formatted. Poor-quality data contains errors, gaps, contradictions, or values that look valid but represent something different than expected.

The Navigation Analogy

A GPS navigation system with a map updated five years ago has high technical quality — the map is complete, consistent, and formatted correctly. But the data is outdated. New roads are missing. Closed roads still appear. The navigation takes you the wrong way with complete confidence. Data quality is not just about format — freshness, accuracy, and fitness for purpose all matter equally.

The Six Dimensions of Data Quality

Accuracy

Data accurately reflects the real-world entity it represents. An order record that shows a customer paid $500 when they actually paid $50 has an accuracy problem. Inaccurate data comes from manual entry errors, broken integrations that mismap fields, and unit mismatches (dollars entered as cents).

Completeness

All required data is present. A customer record missing a phone number is incomplete. An order record with no customer ID is incomplete. Completeness issues arise from optional fields that should be mandatory, source systems that allow null values in critical columns, and partial loads where a pipeline fails before writing all records.

Consistency

Data is consistent across systems and within a dataset. A customer's address should match across the CRM, the order system, and the shipping system. A product's price should not differ between the catalog database and the sales fact table. Inconsistency creates conflicting answers to the same business question depending on which system an analyst queries.

Timeliness

Data is available when it is needed. Yesterday's sales data arriving at 11 AM for a 9 AM executive meeting is a timeliness failure. Real-time fraud detection data that arrives two minutes after a transaction completes is a timeliness failure. Pipelines must define SLAs — service level agreements — that specify the maximum acceptable data latency for each dataset.

Uniqueness

Each entity appears exactly once in the dataset. Duplicate customer records cause inflated customer counts. Duplicate order records cause double-counted revenue. Duplicates arise from multiple ingestion runs without deduplication, sources that publish the same event more than once, and join logic that creates fan-out (multiplying rows unintentionally).

Validity

Data values conform to defined business rules and formats. An email address without an "@" symbol is invalid. A date of "2024-13-45" is invalid. A product quantity of -500 is invalid. Validity problems originate in source systems with weak validation or from transformation errors that corrupt values.

Common Data Quality Issues in Practice

Issue Type        | Example                           | Root Cause
------------------|-----------------------------------|----------------------------
Null in key field | order.customer_id = NULL          | Source system allows nulls
Duplicate rows    | Order ORD001 appears twice        | Pipeline ran twice; no dedup
Wrong data type   | amount = "N/A" in numeric column  | CSV parsing error
Stale data        | Yesterday's revenue in today's    | Pipeline SLA missed
                  | report shows as today's           |
Out-of-range value| age = 250 in customer table       | Entry error; no validation
Encoding issue    | Customer name = "Pr??a"           | UTF-8 vs Latin-1 mismatch
Referential error | order.product_id not in products  | Missing product record

Data Quality Checks in Pipelines

Data engineers embed quality checks at multiple stages in the pipeline. Checks run automatically and fail the pipeline or quarantine bad records when violations are detected.

Row Count Checks

Verify that the number of rows loaded matches the expected count. A source that normally produces 100,000 rows per day should alert if today's extract contains only 50 rows — this likely indicates a broken connection, not a genuine drop in activity.

Null Checks

Verify that required columns contain no null values. A check on order_id, customer_id, and amount catches records that would cause downstream analytics to fail or produce misleading aggregations.

Range Checks

Verify that numeric values fall within expected boundaries. Revenue should be non-negative. Age should be between 0 and 120. Percentages should be between 0 and 100.

Referential Integrity Checks

Verify that foreign keys in the fact table match records in dimension tables. An order referencing a product_id that does not exist in the products table signals a data loading error.

Example Quality Check Implementation (dbt test):

-- Check: no null customer_ids in orders table
SELECT COUNT(*) AS null_count
FROM orders
WHERE customer_id IS NULL;
-- Fail pipeline if null_count > 0

-- Check: all order amounts are positive
SELECT COUNT(*) AS negative_count
FROM orders
WHERE amount <= 0;
-- Fail pipeline if negative_count > 0

-- Check: row count within expected range
SELECT COUNT(*) AS row_count FROM orders_today;
-- Fail if row_count < 50000 OR row_count > 500000

The Cost of Poor Data Quality

Research consistently estimates that poor data quality costs organizations between 15 and 25 percent of revenue due to bad decisions, wasted operational effort, failed analytics projects, and customer churn from errors in customer-facing systems. Fixing data quality problems after data reaches downstream consumers costs 10 times more than catching the problem at the source.

Data Quality Tools

Tool               | Approach
-------------------|--------------------------------------------
dbt tests          | SQL-based tests embedded in transformation
Great Expectations | Python library for data validation
Soda Core          | YAML-defined checks; integrates with pipelines
Monte Carlo        | ML-based anomaly detection for data
Datafold           | Diff-based comparison between pipeline runs

Summary

Data quality measures whether data is accurate, complete, consistent, timely, unique, and valid. Poor quality data produces wrong decisions and erodes trust in data systems. Data engineers embed automated quality checks into pipelines — null checks, range checks, row count checks, and referential integrity checks — to catch problems before bad data reaches consumers. Investing in data quality early costs far less than cleaning up incorrect data after business decisions have already been made.