Spark Production Best Practices

A job that works on a laptop can fail at scale. Production Spark needs predictable results, controlled costs, and quick recovery from problems. A set of proven habits turns a one-time script into a dependable data service.

The Layman View

A home cook makes dinner once. A restaurant serves hundreds of plates every night. The restaurant needs written recipes, labelled shelves, regular cleaning, and a plan for broken ovens. Production Spark brings the same discipline to data.

From Development to Production

Develop (local, sample data)
      |
Test (unit tests, small cluster)
      |
Stage (full-size data copy)
      |
Production (scheduled, monitored)
      |
Review (metrics, cost, failures) --> back to Develop

Data Design

  • Store data in columnar formats such as Parquet or Delta tables.
  • Define schemas by hand instead of using inference.
  • Partition output folders by a column with moderate variety, such as date.
  • Avoid many tiny files. Aim for files between 128 MB and 1 GB.
Too small:   1,000,000 files x 100 KB   --> slow listing, slow reads
Better:      100 files x 1 GB           --> fast scans

Code Habits

  • Filter rows and select columns early.
  • Prefer DataFrame functions over UDFs and RDD code.
  • Broadcast small lookup tables.
  • Avoid collect and toPandas on large data.
  • Cache a DataFrame only when several actions reuse it, and unpersist it afterward.

Safe Writes

Design every job so a rerun gives the same answer. This property is called idempotence. A failed job can run again with no duplicate rows.

Not safe:  append today's rows every run   --> rerun doubles the data
Safe:      overwrite only today's partition --> rerun gives identical data

spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")
result.write.mode("overwrite").partitionBy("run_date").parquet("/data/report")

Configuration

  • Keep settings in files or job parameters, never hard-coded in scripts.
  • Leave Adaptive Query Execution enabled.
  • Enable dynamic allocation on shared clusters.
  • Change one setting at a time and record the result.

Security

  • Store passwords and keys in a secrets manager.
  • Give each job the minimum access that it needs.
  • Encrypt data in storage and in transit.
  • Mask or remove personal data before wide sharing.
  • Keep an audit record of who runs and changes jobs.

Monitoring and Alerts

Track job duration, input rows, output rows, failed tasks, and cost. Compare each run with the recent average. Send an alert when a number moves far from normal.

Run history (minutes):  22   21   23   22   58   <-- alert: run took far longer
Output rows (millions): 4.1  4.0  4.2  4.1  0.0  <-- alert: empty output

Data Quality Checks

from pyspark.sql import functions as F

total      = result.count()
null_keys  = result.filter(F.col("customer_id").isNull()).count()
duplicates = total - result.dropDuplicates(["order_id"]).count()

assert total > 0, "Output is empty"
assert null_keys == 0, "Null customer ids found"
assert duplicates == 0, "Duplicate orders found"

Stop the pipeline when a check fails so bad data never reaches reports.

Cost Control

  • Right-size executors by reading the Spark UI, not by guessing.
  • Release idle resources with dynamic allocation.
  • Skip work that did not change by processing only new partitions.
  • Delete temporary data and old versions on a schedule.

Documentation and Ownership

Write a short page for every job. List the inputs, outputs, schedule, owner, and recovery steps. A new team member should restart a failed job without guessing.

Release Checklist

  • Unit tests pass on sample data.
  • A full-size staging run completes in the expected time.
  • Schemas, quality checks, and alerts exist.
  • Credentials come from a secrets service.
  • A rerun produces identical output.
  • The job page lists an owner and recovery steps.
Previous lesson
Back to course
Next lesson