ADE Data File Formats

A file format is the container that holds data on disk. The right container makes queries fast and storage cheap. The wrong container makes every job slow and every bill larger. Azure data lakes store files of many formats, and a data engineer must know the strengths of each one.

Row Storage vs Column Storage

Think of a school register. A row-based register lists every detail of one student on a single line. A column-based register keeps one page for all names, one page for all marks, and one page for all addresses. Finding the average mark is quick in the column-based register, because the reader opens only the marks page.

THREE RECORDS: (Asha, 82, Delhi)  (Ravi, 74, Pune)  (Meena, 91, Goa)

ROW-BASED FILE                         COLUMN-BASED FILE
[Asha | 82 | Delhi]                    Names : [Asha | Ravi | Meena]
[Ravi | 74 | Pune ]                    Marks : [82   | 74   | 91   ]
[Meena| 91 | Goa  ]                    Towns : [Delhi| Pune | Goa  ]

Good for: reading whole records         Good for: summing or averaging one column

Popular Formats

CSV

CSV stores plain text with commas between values. Every tool opens it, and humans read it easily. CSV carries no data types, no compression, and no support for nested data. Large CSV files scan slowly.

JSON

JSON stores nested objects and lists. Web APIs and application logs produce it everywhere. The field names repeat on every record, which inflates file size.

Parquet

Parquet stores data in columns, compresses it heavily, and embeds the schema inside the file. Spark, Synapse, Databricks, and Fabric read Parquet natively. Analytics workloads on Azure use it as the standard choice.

Avro

Avro stores data in rows with the schema attached. It writes quickly and handles schema changes gracefully. Azure Event Hubs Capture writes Avro files to storage, and streaming pipelines rely on it.

ORC

ORC works like Parquet with a column layout and strong compression. Hadoop and Hive environments use it most.

Delta

Delta files are Parquet files plus a transaction log. The log adds updates, deletes, and version history on top of plain Parquet.

Format Comparison

Format	Layout	Compression	Schema Inside	Best Use
CSV	Row, text	None by default	No	Simple exchange with other people
JSON	Row, text	None by default	Loose	API responses and logs
Parquet	Column	High	Yes	Analytics and reporting
Avro	Row	Medium	Yes	Streaming and messaging
ORC	Column	High	Yes	Hive and Hadoop workloads
Delta	Column plus log	High	Yes	Lakehouse tables with updates

Scenario: RoadRunner Cabs

RoadRunner Cabs, a fictional taxi company, records 50 million trips per month. Each trip row holds 40 columns, and a typical report reads only three of them: fare, distance, and city.

APP LOGS (JSON) ---> Raw zone (JSON, kept as received)
                          |
                          v
                  Spark job converts
                          |
                          v
          Curated zone (Parquet, partitioned by month)
                          |
                          v
          Report reads 3 of 40 columns = far less data scanned

The raw zone preserves the original JSON for audits. The curated zone holds Parquet, and a monthly report scans a small slice of the data. Query time drops from minutes to seconds, and storage cost falls because compression shrinks the files.

Choosing a Format

• Pick Parquet or Delta for analytics tables.
• Pick Avro for event streams that change shape over time.
• Pick JSON for raw API captures that need exact preservation.
• Pick CSV only for small exports shared with people or older systems.

Small File Problem

Thousands of tiny files slow every engine, because each file costs an extra open operation. Aim for files between 128 MB and 1 GB in analytics zones. Compaction jobs merge small files into larger ones on a schedule.

Partitioning in Folders

Folder names such as year=2026/month=10 let engines skip whole folders that a query does not need. Choose partition columns that queries filter on often, such as date or region. Avoid columns with millions of unique values, since they create millions of folders.

Key Points

• Column formats such as Parquet make analytics fast and storage cheap.
• Row formats such as Avro write fast and suit streaming.
• Delta adds updates and history on top of Parquet.
• Large files in well-chosen partitions give the best performance.