Spark Built-in Functions

Spark ships with hundreds of ready-made functions for text, numbers, dates, and missing values. Each function runs inside the Spark engine, so it runs fast and the optimizer understands it. Learn the common ones first and you will rarely need custom code.

Where Functions Live

from pyspark.sql import functions as F

pyspark.sql.functions
 |
 |-- Text      : upper, lower, trim, concat, substring, split, regexp_replace
 |-- Numbers   : round, ceil, floor, abs, sqrt
 |-- Dates     : current_date, to_date, datediff, date_add, date_format
 |-- Logic     : when, otherwise, coalesce
 |-- Summaries : count, sum, avg, min, max, collect_list

Import the module with the short alias F. Every example below uses that alias.

Sample Data

df = spark.createDataFrame(
    [(" Asha ", "sales", 52000.456, "2025-01-15", None),
     ("Ben",    "it",    48000.0,   "2024-11-03", "Pune"),
     ("Chitra", "hr",    None,      "2023-07-21", "Delhi")],
    ["name", "dept", "salary", "joined", "city"])

Text Functions

df.select(
    F.trim("name").alias("clean_name"),
    F.upper("dept").alias("dept_upper"),
    F.concat_ws(" - ", F.trim("name"), F.upper("dept")).alias("label"),
    F.substring("joined", 1, 4).alias("year")
).show()

Number Functions

df.select("name", F.round("salary", 1).alias("rounded"),
          F.ceil("salary").alias("rounded_up")).show()

Date Functions

Convert text to a real date first. Date arithmetic needs the date type.

dated = df.withColumn("joined_date", F.to_date("joined"))

dated.select(
    "name",
    F.datediff(F.current_date(), "joined_date").alias("days_employed"),
    F.date_format("joined_date", "MMM yyyy").alias("joined_month"),
    F.date_add("joined_date", 90).alias("review_date")
).show()

Conditional Logic with when

The when function works like an if-else chain. Spark checks each condition from top to bottom.

df.select("name", "salary",
    F.when(F.col("salary") >= 50000, "High")
     .when(F.col("salary") >= 45000, "Medium")
     .otherwise("Low").alias("band")
).show()

salary = 52000 --> check >= 50000 --> yes --> "High"
salary = 48000 --> check >= 50000 --> no
                  check >= 45000 --> yes --> "Medium"
salary = null  --> no check passes   --> "Low"

Handling Missing Values

Real data contains empty cells. Spark represents them as null, and null values break arithmetic quietly. A sum that includes null skips it, and a comparison with null returns null.

df.select("name", F.coalesce("city", F.lit("Unknown")).alias("city")).show()

df.filter(F.col("city").isNull()).show()
df.fillna({"salary": 0, "city": "Unknown"}).show()
df.dropna(subset=["salary"]).show()

• coalesce returns the first value that is not null.
• fillna replaces nulls with a default.
• dropna removes rows that contain nulls.

Regular Expressions

phones = spark.createDataFrame([("(91) 98765-43210",)], ["phone"])
phones.select(F.regexp_replace("phone", "[^0-9]", "").alias("digits")).show()
# 919876543210

Finding the Right Function

• Search the PySpark API reference by category before you write custom logic.
• Use F.expr("...") to run any SQL function inside the DataFrame API.
• Test a function on three or four rows with show() before you apply it to the full table.

Combining Functions

Functions nest inside each other like building blocks. The inner function runs first and passes its result outward. A cleaned, formatted label often needs three functions in one expression.

df.select(
    F.initcap(F.lower(F.trim("name"))).alias("tidy_name"),
    F.when(F.col("city").isNull(), "Remote").otherwise(F.upper("city")).alias("location")
).show()

Step order for the value " ASHA ":
trim     --> "ASHA"
lower    --> "asha"
initcap  --> "Asha"

Aggregate Functions

Summary functions such as count, sum, avg, min, and max work inside agg. The collect_list and collect_set functions gather values of a group into an array.

df.groupBy("dept").agg(
    F.count("*").alias("people"),
    F.collect_list("name").alias("names")
).show(truncate=False)

Why Built-ins Beat Custom Code

• The Catalyst optimizer reads built-in functions and rearranges them for speed.
• Built-in functions run inside the JVM and avoid the cost of moving rows to Python.
• Thousands of users test them, so edge cases such as null values already have defined behaviour.