DE Spark DataFrames

The DataFrame API is the primary way data engineers interact with Apache Spark. It provides a structured, tabular view of distributed data — similar to a SQL table or a pandas DataFrame — but backed by Spark's distributed processing engine. Engineers write familiar operations like filter, group, join, and sort, and Spark executes them in parallel across a cluster.

What Is a Spark DataFrame

A Spark DataFrame is a distributed collection of rows organized into named, typed columns. It represents a large dataset split across many machines in the cluster. From the engineer's perspective, it looks like a single table. Under the hood, Spark manages the partitioning, parallel execution, and fault tolerance automatically.

The Spreadsheet Across Many Computers Analogy

Imagine a spreadsheet with one billion rows. No laptop can open it — it is too large for RAM. Now imagine splitting that spreadsheet into 1,000 equal chunks and loading each chunk onto a different computer. All 1,000 computers can apply a filter simultaneously. When done, the results combine. A Spark DataFrame is that distributed spreadsheet — you interact with it as if it were one object, but it lives across many machines at once.

Creating a DataFrame

from pyspark.sql import SparkSession

# Start a Spark session (the entry point to Spark)
spark = SparkSession.builder \
    .appName("SalesAnalysis") \
    .getOrCreate()

# Read a CSV file from cloud storage into a DataFrame
df = spark.read \
    .option("header", "true") \
    .option("inferSchema", "true") \
    .csv("s3://my-bucket/sales/orders.csv")

# See the schema (column names and data types)
df.printSchema()

# Preview first 5 rows
df.show(5)

Common DataFrame Operations

select(): Choose Columns

# Select specific columns
df.select("order_id", "customer_id", "amount").show()

filter() / where(): Filter Rows

# Keep only orders above 1000
filtered = df.filter(df.amount > 1000)

# Multiple conditions
filtered = df.filter((df.amount > 1000) & (df.status == "completed"))

groupBy() and agg(): Aggregate

from pyspark.sql.functions import sum, count, avg

# Total revenue and order count per city
city_summary = df.groupBy("city") \
    .agg(
        sum("amount").alias("total_revenue"),
        count("order_id").alias("order_count"),
        avg("amount").alias("avg_order_value")
    )

city_summary.orderBy("total_revenue", ascending=False).show()

withColumn(): Add or Transform Columns

from pyspark.sql.functions import col, when, upper

# Add a new column based on logic
df = df.withColumn(
    "order_tier",
    when(col("amount") >= 5000, "High")
    .when(col("amount") >= 1000, "Medium")
    .otherwise("Low")
)

# Transform an existing column
df = df.withColumn("city_upper", upper(col("city")))

join(): Combine DataFrames

# Read a customers reference table
customers_df = spark.read.parquet("s3://my-bucket/customers/")

# Join orders with customers on customer_id
enriched = df.join(customers_df, on="customer_id", how="inner")

# Left join example
enriched = df.join(customers_df, on="customer_id", how="left")

dropDuplicates(): Remove Duplicates

# Remove exact duplicate rows
df_deduped = df.dropDuplicates()

# Remove duplicates based on specific columns (keep first occurrence)
df_deduped = df.dropDuplicates(["order_id"])

Using SQL with DataFrames

Data engineers who prefer SQL can register a DataFrame as a temporary view and query it with standard SQL syntax. Spark compiles the SQL into the same distributed execution plan as the DataFrame API.

# Register the DataFrame as a SQL view
df.createOrReplaceTempView("orders")

# Run SQL against the view
result = spark.sql("""
    SELECT
        city,
        SUM(amount)   AS total_revenue,
        COUNT(*)      AS order_count
    FROM orders
    WHERE status = 'completed'
    GROUP BY city
    ORDER BY total_revenue DESC
""")

result.show()

Writing DataFrames to Storage

After processing, DataFrames write to cloud storage or a database. Choosing the right format and partition strategy significantly impacts how fast downstream queries run.

# Write as Parquet (columnar, compressed — recommended for analytics)
df.write \
    .mode("overwrite") \
    .partitionBy("sale_year", "sale_month") \
    .parquet("s3://my-bucket/processed/orders/")

# Write as Delta Lake table (adds ACID transactions)
df.write \
    .format("delta") \
    .mode("append") \
    .save("s3://my-bucket/delta/orders/")

# Write to a database table
df.write \
    .format("jdbc") \
    .option("url", "jdbc:postgresql://host/db") \
    .option("dbtable", "processed_orders") \
    .mode("append") \
    .save()

Partitions and Parallelism

Spark divides a DataFrame into partitions. Each partition is a chunk of rows that one executor processes independently. More partitions mean more parallelism — up to the number of available CPU cores in the cluster. Too few partitions underutilize the cluster. Too many partitions create overhead from task scheduling. The recommended target is 128MB to 256MB of data per partition.

# Check current partition count
print(df.rdd.getNumPartitions())   # e.g., returns 200

# Repartition to a specific number
df = df.repartition(50)

# Coalesce (reduce partitions without full shuffle — faster)
df = df.coalesce(10)

Built-in Functions

Spark provides hundreds of built-in functions covering date arithmetic, string manipulation, mathematical operations, array handling, and type casting. These functions run natively in Spark's distributed execution engine — far faster than equivalent Python logic applied row by row.

from pyspark.sql.functions import (
    to_date, date_trunc, datediff,
    trim, lower, regexp_replace,
    round, abs, log,
    year, month, dayofweek
)

df = df.withColumn("order_date", to_date(col("order_date_str"), "yyyy-MM-dd"))
df = df.withColumn("order_month", date_trunc("month", col("order_date")))
df = df.withColumn("clean_name", trim(lower(col("customer_name"))))
df = df.withColumn("revenue_rounded", round(col("revenue"), 2))

Summary

Spark DataFrames provide a structured, distributed table abstraction that data engineers use to process massive datasets. Operations like select, filter, groupBy, join, and withColumn execute in parallel across the cluster. Engineers can use either the Python DataFrame API or SQL syntax interchangeably. Writing results in columnar formats like Parquet with smart partitioning ensures fast downstream access. The DataFrame API is the most practical entry point to building production Spark pipelines.