DE Data Lakes

A data warehouse is excellent for structured, well-organized data. But organizations also generate enormous amounts of data that does not fit neatly into tables — log files, images, audio recordings, sensor streams, raw JSON from APIs, and more. The data lake was designed to hold all of it.

What Is a Data Lake

A data lake is a centralized storage system that holds data in its raw, original format. It accepts any type of data — structured, semi-structured, or unstructured — without requiring a predefined structure. Data lands in the lake as-is and gets organized or transformed later when someone needs to use it.

The Lake Analogy

Imagine a large natural lake. Rivers, streams, rainwater, and groundwater all flow into it. Each source carries different things — clear water, sediment, minerals. The lake stores all of it without sorting or filtering first. Someone who needs clean water for drinking extracts it and processes it separately. A geologist who needs sediment samples takes those. The lake holds everything; what you do with it depends on your purpose.

A data lake works the same way. Raw data from many different sources flows in. Analysts, data scientists, and engineers each extract and process the portion they need, in the format they need it.

Data Lake vs Data Warehouse

Feature           | Data Lake                   | Data Warehouse
------------------|-----------------------------|--------------------------
Data types        | Any (structured, JSON, imgs)| Structured only
Schema            | Schema-on-read              | Schema-on-write
Storage cost      | Very low (object storage)   | Higher (proprietary storage)
Data state        | Raw, unprocessed            | Cleaned, transformed
Best for          | ML, exploration, raw storage| Business reporting, SQL
Query speed       | Slower (raw format)         | Fast (optimized columns)
Typical storage   | Amazon S3, Azure ADLS, GCS  | Snowflake, BigQuery, Redshift

Schema-on-Read vs Schema-on-Write

Schema-on-Write (Data Warehouse)

In a data warehouse, you define the table structure before data enters. The data must fit the predefined shape. If a new column appears in the source data, someone must alter the table schema first. This approach ensures clean data but slows down ingestion when requirements change.

Schema-on-Read (Data Lake)

In a data lake, data lands without a predefined structure. The schema gets applied at query time, when someone reads the data and decides how to interpret it. A JSON file in the lake can be read as a flat table by one user and as a nested document by another. This flexibility accelerates ingestion but requires discipline to prevent the lake from becoming unnavigable.

The Data Swamp Problem

Without proper organization, a data lake degrades into what engineers call a "data swamp." Files arrive with no naming convention, no documentation, and no tracking of what each file contains. Users cannot find what they need, do not know if the data is current, and cannot trust its quality. A well-managed data lake requires a data catalog, clear folder naming conventions, and metadata tagging for every dataset.

Well-managed lake folder structure:
/raw/
  /orders/year=2024/month=05/orders_2024_05_01.parquet
  /customers/year=2024/month=05/customers_snapshot.json
/processed/
  /orders_cleaned/
  /customers_enriched/
/curated/
  /sales_summary/

Common File Formats in a Data Lake

Format    | Description                          | Best For
----------|--------------------------------------|---------------------------
CSV       | Comma-separated text                 | Simple tabular data
JSON      | Nested key-value format              | API responses, logs
Parquet   | Columnar binary format               | Large-scale analytics
Avro      | Row-based binary with schema         | Streaming and Kafka
ORC       | Columnar binary; optimized for Hive  | Hadoop ecosystem

Data engineers prefer Parquet for most analytical workloads because it compresses well and reads only the required columns — the same efficiency advantage columnar databases provide.

What Data Lakes Enable

Machine Learning

Data scientists need access to raw data to build machine learning models. A data lake stores years of historical raw data — customer behavior logs, sensor readings, text records — that training models require. The warehouse often does not preserve the raw granularity needed for ML.

Exploratory Analysis

Analysts sometimes need to explore data before they know what questions to ask. The lake lets them load raw data into a Spark job or a query tool and experiment without first building a structured pipeline.

Cost-Effective Archiving

Object storage costs pennies per gigabyte per month. Organizations archive years of raw data in a lake at a fraction of the cost of storing the same data in a data warehouse.

Cloud Data Lake Storage

Cloud Provider | Storage Service           | Common Name
---------------|---------------------------|----------------
Amazon AWS     | Amazon S3                 | S3
Google Cloud   | Google Cloud Storage      | GCS
Microsoft Azure| Azure Data Lake Storage   | ADLS Gen2

The Zone Architecture

Most mature data lake designs divide storage into zones based on data quality and processing stage.

Raw Zone      --> Bronze layer: data as it arrived, untouched
Processed Zone --> Silver layer: cleaned, validated, deduplicated
Curated Zone  --> Gold layer: aggregated, business-ready datasets

This Bronze-Silver-Gold pattern (popularized by Databricks) gives teams confidence about the state of data at each stage. Engineers transform data from raw to processed; analysts typically consume from the curated layer.

Summary

A data lake stores any type of data in its original format at low cost. It accepts everything without requiring a predefined structure. It enables machine learning, exploration, and cost-effective archiving that a data warehouse cannot support. Managed well with clear organization and metadata, a data lake becomes an invaluable complement to the structured analytics that a data warehouse provides.