Spark Delta Lake Basics

Delta Lake adds reliability to files stored in a data lake. It layers a transaction log on top of Parquet files and gives Spark tables the safety that database users expect. Updates, deletes, and merges work correctly even when several jobs run together, and a failed job never leaves half-written data behind.

The Layman View

A plain folder of files is like a shared notebook where anyone can tear out pages at any time. Delta Lake adds a ledger beside the notebook. Every change gets a numbered ledger entry, and readers trust only the entries marked complete.

What Delta Lake Gives You

  • ACID transactions keep each write complete or absent, never half done.
  • Updates, deletes, and merges work directly on the table.
  • Time travel reads older versions of the table.
  • Schema enforcement rejects data that does not match the table.

How a Delta Table Looks on Disk

/data/sales_delta/
 |-- _delta_log/
 |     |-- 00000000000000000000.json    (version 0: table created)
 |     |-- 00000000000000000001.json    (version 1: rows added)
 |     |-- 00000000000000000002.json    (version 2: rows updated)
 |-- part-0001.snappy.parquet
 |-- part-0002.snappy.parquet
 |-- part-0003.snappy.parquet

Readers open the log first, learn which Parquet files belong to the current version, and read only those files.

Starting Spark with Delta

pip install delta-spark

from delta import configure_spark_with_delta_pip
from pyspark.sql import SparkSession

builder = (SparkSession.builder.appName("delta-demo")
    .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
    .config("spark.sql.catalog.spark_catalog",
            "org.apache.spark.sql.delta.catalog.DeltaCatalog"))

spark = configure_spark_with_delta_pip(builder).getOrCreate()

Creating and Reading a Delta Table

data = [(1, "Asha", 500), (2, "Ben", 300)]
df = spark.createDataFrame(data, ["id", "name", "balance"])

df.write.format("delta").mode("overwrite").save("/data/accounts")

spark.read.format("delta").load("/data/accounts").show()

Updating and Deleting Rows

Plain Parquet folders cannot change single rows. Delta tables can.

from delta.tables import DeltaTable

accounts = DeltaTable.forPath(spark, "/data/accounts")

accounts.update("id = 2", {"balance": "balance + 100"})
accounts.delete("id = 1")

Upserts with merge

A merge compares new records with existing rows. Matching rows update. Missing rows insert.

updates = spark.createDataFrame([(2, "Ben", 450), (3, "Chitra", 700)],
                                ["id", "name", "balance"])

(accounts.alias("t")
    .merge(updates.alias("s"), "t.id = s.id")
    .whenMatchedUpdateAll()
    .whenNotMatchedInsertAll()
    .execute())

Existing table        Updates            Result after merge
id 2  Ben    400      id 2  Ben    450   id 2  Ben    450  (updated)
                      id 3  Chitra 700   id 3  Chitra 700  (inserted)

Time Travel

Every write creates a new version. Read an old version by number or by timestamp.

spark.read.format("delta").option("versionAsOf", 0).load("/data/accounts").show()
spark.read.format("delta").option("timestampAsOf", "2026-01-01").load("/data/accounts").show()

accounts.history().select("version", "timestamp", "operation").show()

Schema Enforcement

A write with an unexpected column or type fails with a clear error. The table stays clean. Add the option mergeSchema when you intend to add a column on purpose.

new_df.write.format("delta").mode("append").option("mergeSchema", "true").save("/data/accounts")

Cleaning Up

Old files stay on disk to support time travel. The vacuum command removes files older than a retention period, which defaults to seven days. The optimize command combines many small files into larger ones for faster reads.

spark.sql("OPTIMIZE delta.`/data/accounts`")
spark.sql("VACUUM delta.`/data/accounts` RETAIN 168 HOURS")

When to Choose Delta Lake

  • Several jobs write to the same table.
  • Data needs corrections, such as customer updates or deletions.
  • Teams need an audit trail of table changes.
  • Streaming jobs and batch jobs share one table.

Delta Tables in SQL

SQL users create and query Delta tables with familiar commands.

spark.sql("CREATE TABLE sales_delta (id INT, item STRING, amount DOUBLE) USING DELTA")
spark.sql("INSERT INTO sales_delta VALUES (1, 'Pen', 12.5)")
spark.sql("SELECT * FROM sales_delta").show()
spark.sql("DESCRIBE HISTORY sales_delta").show()

Rolling Back a Mistake

A wrong delete or update does not end in disaster. Restore the table to an earlier version with a single command.

accounts.restore(1)

The restore creates a new version that matches version 1. The history keeps the full record of what happened.

Previous lesson
Back to course
Next lesson