ADE PySpark Essentials

PySpark is the Python interface to Apache Spark, the engine that powers Azure Databricks, Synapse Spark pools, and Fabric notebooks. Spark splits large data across many machines and processes the pieces in parallel. A few core ideas unlock most day-to-day pipeline work.

The Team of Counters

Picture a mountain of 10 million paper receipts. One person needs days to total them. A manager who hands a stack to each of 100 helpers collects the final answer in minutes. Spark plays the manager, and the helpers are executors that run on separate machines.

                 +------------------+
                 |  DRIVER (manager)|
                 |  plans the job   |
                 +--------+---------+
                          |
        +-----------------+-----------------+
        |                 |                 |
+-------v------+  +-------v------+  +-------v------+
| EXECUTOR 1   |  | EXECUTOR 2   |  | EXECUTOR 3   |
| partition A  |  | partition B  |  | partition C  |
+--------------+  +--------------+  +--------------+

A partition is one slice of the data. Each executor works on its own slices at the same time.

DataFrames

A DataFrame is a table with named columns spread across the cluster. Databricks and Fabric notebooks provide a ready spark session, so reading data takes one line.

orders = spark.read.format("parquet").load("/mnt/lake/raw/orders")
orders.printSchema()
orders.show(5)

Everyday Transformations

The fictional PizzaPlanet chain stores orders with columns for city, pizza, quantity, and price.

from pyspark.sql import functions as F

clean = (
    orders
    .filter(F.col("quantity") > 0)
    .withColumn("revenue", F.col("quantity") * F.col("price"))
    .select("order_id", "city", "pizza", "revenue")
)

by_city = (
    clean.groupBy("city")
         .agg(F.sum("revenue").alias("total_revenue"),
              F.count("order_id").alias("order_count"))
         .orderBy(F.desc("total_revenue"))
)

Method	Job
select	Pick columns
filter	Keep rows that match a condition
withColumn	Add or change a column
groupBy and agg	Summarize rows by group
join	Combine two DataFrames on a key
dropDuplicates	Remove repeated rows

Joining DataFrames

stores = spark.read.format("parquet").load("/mnt/lake/raw/stores")
enriched = clean.join(stores, on="city", how="left")

Join keys with many repeated values create uneven partitions, and one slow executor drags the whole job. Small lookup tables work best with a broadcast join, which copies the small table to every executor.

from pyspark.sql.functions import broadcast
enriched = clean.join(broadcast(stores), on="city", how="left")

Lazy Evaluation

Spark does not run a transformation the moment it appears in the code. Spark records each step in a plan and waits for an action such as show, count, or write. At that point, Spark optimizes the entire plan and runs it once.

Kind	Examples	Behavior
Transformation	select, filter, join, groupBy	Adds a step to the plan
Action	show, count, collect, write	Triggers the real work

Lazy evaluation lets Spark combine steps, skip unused columns, and push filters close to the data.

Writing Results

(by_city.write
        .mode("overwrite")
        .format("delta")
        .save("/mnt/lake/gold/revenue_by_city"))

Write modes include overwrite, append, and ignore. Choose append for daily increments and overwrite for full rebuilds.

Reading Messy Files Safely

Source files rarely arrive in perfect shape. A schema tells Spark the exact column names and types, so the engine skips the slow guessing step and catches surprises early.

from pyspark.sql.types import StructType, StructField, StringType, IntegerType, DoubleType

schema = StructType([
    StructField("order_id", StringType(), False),
    StructField("city", StringType(), True),
    StructField("pizza", StringType(), True),
    StructField("quantity", IntegerType(), True),
    StructField("price", DoubleType(), True)
])

orders = (spark.read
          .option("header", True)
          .schema(schema)
          .csv("/mnt/lake/raw/orders_csv"))

A fixed schema also protects the pipeline from silent type changes. A price column that suddenly holds text triggers a clear problem instead of corrupt totals.

Handling Missing Values

Blank cells appear in almost every dataset. Spark offers simple tools to fill or drop them.

filled = orders.fillna({"quantity": 1, "city": "Unknown"})
complete_only = orders.dropna(subset=["order_id", "price"])

Choose a fill value that matches the business meaning. A missing quantity of one makes sense for a single pizza order, while a missing price needs investigation instead of a guess.

Mixing SQL and Python

Spark lets you register a DataFrame as a temporary view and query it with SQL. Analysts who know SQL contribute to the same notebook without learning new syntax.

clean.createOrReplaceTempView("clean_orders")
spark.sql("SELECT city, SUM(revenue) AS total FROM clean_orders GROUP BY city").show()

Habits for Healthy Jobs

• Avoid collect on large data, since it pulls everything into the driver memory.
• Filter and select early so Spark moves fewer bytes.
• Prefer built-in functions over custom Python functions, which run slowly.
• Keep partitions of moderate size, roughly 100 MB to 200 MB each.

Key Points

• Spark splits data into partitions and processes them in parallel.
• DataFrames offer a table view with select, filter, groupBy, and join.
• Lazy evaluation builds a plan first and runs it at an action.
• Broadcast joins and early filters keep jobs fast.