DE Data Cataloging
An organization that has built hundreds of datasets, tables, and pipelines over several years faces a real problem: no one knows what data exists, where it lives, or what it means. Analysts duplicate work because they cannot find datasets that already answer their question. Engineers build pipelines for data that exists elsewhere. Data cataloging solves this by creating a searchable, organized inventory of every data asset in the organization.
What Is a Data Catalog
A data catalog is a centralized metadata repository that documents all data assets — their location, structure, ownership, quality, lineage, and business meaning. It functions as the "search engine" for data within an organization. An analyst types "customer purchase history" into the catalog and finds the exact table they need, who owns it, what columns it contains, how fresh the data is, and example values.
The Library Card Catalog Analogy
Before digital library systems, libraries maintained physical card catalogs — drawers of index cards, each describing one book: title, author, subject, shelf location, availability. A visitor searched the catalog to find books without wandering through every aisle. A modern data catalog does the same: it indexes every data asset and lets data consumers find what they need through search, filters, and browsing — without knowing the exact table name or storage location in advance.
What a Data Catalog Contains
Technical Metadata
Technical metadata describes the physical properties of a data asset — where it is stored, what format it uses, and what its schema looks like.
Technical Metadata for table: analytics.fact_sales
Location: Snowflake, database=ANALYTICS, schema=SALES
Format: Snowflake table (columnar)
Row count: 1,284,500,000 (as of 2024-05-15)
Size: 2.8 TB
Last updated: 2024-05-15 03:14:22 UTC
Partitioned: Yes (by sale_date)
Schema:
sale_id BIGINT PRIMARY KEY
sale_date DATE NOT NULL
customer_id VARCHAR(20) NOT NULL
product_id VARCHAR(20) NOT NULL
revenue_usd DECIMAL NOT NULL
quantity INTEGER NOT NULL
Business Metadata
Business metadata explains what the data means in plain language — without requiring technical knowledge of the underlying system.
Business Metadata for analytics.fact_sales:
Description: "Records all completed sales transactions
processed through the e-commerce platform.
Each row represents one line item in one order."
Owner: Finance Data Team (finance-data@company.com)
Steward: Priya Sharma (data engineer)
Domain: Revenue and Sales
Tags: #revenue, #transactions, #fact-table, #daily-refresh
Definition of "revenue_usd":
"Net order line item amount after discounts,
converted from local currency to USD at the
exchange rate on the order date."
Operational Metadata
Operational metadata tracks the health and history of a data asset — when it last refreshed, whether quality checks passed, and how reliably the pipeline that feeds it has run.
Operational Metadata: Pipeline: daily_sales_etl (runs 2:00 AM UTC) Last run status: SUCCESS Last 30 days: 28 successful / 2 failed (93% reliability) Data SLA: Available by 4:00 AM UTC Quality score: 98.4% (0 nulls in key fields; 1.6% minor issues)
Data Catalog Features
Search and Discovery
Full-text search across all metadata — table names, column names, descriptions, tags, and business terms. An analyst searching "monthly revenue by product" finds all tables with relevant columns and descriptions, ranked by relevance and freshness.
Data Lineage Visualization
The catalog integrates with lineage tools to show each asset's upstream sources and downstream consumers in an interactive graph. A single click on a table shows every pipeline that feeds it and every report or model that reads from it.
Data Profiling
Automated profiling runs on each table and produces a statistical summary: row count, column cardinality (number of unique values), null percentage, minimum and maximum values, and sample values. This profile helps analysts assess whether a table contains the data they need before writing a query.
Column Profile for "revenue_usd": Non-null: 100% (0 nulls) Min: 0.01 Max: 48,500.00 Average: 127.43 Median: 84.00 Std Dev: 210.80 Sample values: 24.99, 149.00, 399.95, 1200.00
Business Glossary Integration
The catalog links technical column names to business glossary definitions. The column "revenue_usd" in a table links to the business glossary entry for "Net Revenue" — making it clear to any user reading the catalog exactly what that number represents and how it is calculated.
Popular Data Catalog Tools
Tool | Type | Best For ----------------|--------------------|-------------------------------------- dbt | Lightweight | dbt model documentation; SQL transforms Apache Atlas | Open-source | Hadoop ecosystem; enterprise on-prem DataHub | Open-source | Large platform teams; LinkedIn-backed Alation | Commercial | Analyst-facing; strong collaboration Collibra | Commercial | Enterprise governance + catalog Google Dataplex | Cloud-native GCP | GCP-centric organizations AWS Glue Catalog| Cloud-native AWS | AWS-centric; Athena/Glue integration
Building a Data Catalog Culture
A data catalog only provides value when teams actually use it and keep it current. Data engineers contribute by documenting new tables and pipelines in the catalog as they build them. Data stewards review and update business descriptions regularly. Analysts contribute by tagging tables they find useful and flagging outdated documentation. Leadership reinforces catalog use by making it the standard starting point for any new data project — rather than emailing a colleague to ask where the data lives.
Summary
A data catalog documents every data asset with technical, business, and operational metadata. It enables discovery through search, displays lineage graphs, provides column-level profiling, and links to business glossary definitions. Effective catalogs reduce duplicated work, accelerate onboarding, support compliance, and make data assets trustworthy. Tools like DataHub, Alation, and dbt documentation generate and maintain catalog content automatically when integrated into the data engineering workflow.