Spark Schema Definition

A schema describes the shape of a DataFrame. It lists every column name, its data type, and whether it accepts null values. A clear schema protects your pipeline from wrong types, speeds up reading, and documents the data for the next person who opens your code.

The Layman View

A schema works like the header of a form. The form says the first box holds a name, the second holds an age, and the third holds a date. Anyone who reads the form knows what to expect. Spark reads data the same way once you hand it the form.

Inspecting a Schema

df = spark.read.csv("people.csv", header=True, inferSchema=True)
df.printSchema()

root
 |-- name: string (nullable = true)
 |-- age: integer (nullable = true)
 |-- joined: date (nullable = true)

Common Data Types

Type	Holds	Example
StringType	Text	"Asha"
IntegerType	Whole numbers	42
LongType	Large whole numbers	9000000000
DoubleType	Decimal numbers	3.14
DecimalType	Exact money values	199.99
BooleanType	True or false	True
DateType	Calendar date	2026-01-15
TimestampType	Date and time	2026-01-15 10:30:00
ArrayType, MapType, StructType	Nested values	["a", "b"]

Why Inference Has a Cost

The inferSchema option makes Spark read the file once to guess the types and a second time to load the data. A large file pays for two passes. Guessing also goes wrong. A zip code such as 00501 becomes the number 501 and loses its leading zeros.

With inferSchema:    read file --> guess types --> read file again --> DataFrame
With explicit schema: read file with known types --> DataFrame

Defining a Schema with StructType

from pyspark.sql.types import (StructType, StructField, StringType,
                               IntegerType, DateType, DecimalType)

schema = StructType([
    StructField("name",   StringType(),      nullable=False),
    StructField("age",    IntegerType(),     nullable=True),
    StructField("joined", DateType(),        nullable=True),
    StructField("salary", DecimalType(10,2), nullable=True),
])

df = spark.read.csv("people.csv", header=True, schema=schema)

Defining a Schema with a DDL String

A DDL string gives a shorter way to write the same schema.

ddl = "name STRING NOT NULL, age INT, joined DATE, salary DECIMAL(10,2)"
df = spark.read.csv("people.csv", header=True, schema=ddl)

Handling Bad Records

Real files contain broken rows. The mode option tells Spark how to react when a row does not match the schema.

Mode           Behaviour
-------------  ----------------------------------------------
PERMISSIVE     Keep the row and set bad fields to null (default)
DROPMALFORMED  Drop the broken row
FAILFAST       Stop the job at the first broken row

df = (spark.read
        .schema(schema)
        .option("mode", "PERMISSIVE")
        .option("columnNameOfCorruptRecord", "_bad_row")
        .csv("people.csv", header=True))

Add a string column named _bad_row to the schema when you use that option. Spark stores the raw text of each broken line there for later review.

Changing Types After Loading

from pyspark.sql.functions import col

df2 = df.withColumn("age", col("age").cast("long"))
df2 = df.withColumn("joined", col("joined").cast("string"))

Good Habits

Define schemas by hand for production pipelines.
Choose DecimalType for money. Floating point types round values.
Store the schema in one shared module so every job reads files the same way.
Parquet files carry their own schema, so reads need no inference.

Nested Schemas

A schema can hold structs and arrays. Declare the inner shape the same way you declare the outer shape.

from pyspark.sql.types import ArrayType

nested = StructType([
    StructField("id", IntegerType()),
    StructField("address", StructType([
        StructField("city", StringType()),
        StructField("zip",  StringType())
    ])),
    StructField("skills", ArrayType(StringType()))
])

Nested schemas keep related fields together and make JSON and Parquet files easy to read.

Comparing Two Schemas

Pipelines break when a source file adds or renames a column. Compare the expected schema with the incoming one before the job starts.

if df.schema != schema:
    raise ValueError("Incoming data does not match the expected schema")

Previous lesson

Back to course

Next lesson