Spark Submit and Deployment
The spark-submit command sends your finished application to a cluster. It packages options, files, and settings and starts the driver. Every production Spark job, from a nightly report to a streaming service, launches through this command or a tool that wraps it.
The Layman View
Writing a Spark program resembles writing a recipe. Submitting it resembles handing the recipe to a restaurant kitchen with instructions: how many cooks, how much counter space, and which pantry to use. spark-submit carries those instructions.
Basic Shape of the Command
spark-submit \ --master yarn \ --deploy-mode cluster \ --name daily_sales_report \ --num-executors 10 \ --executor-cores 4 \ --executor-memory 8g \ --driver-memory 4g \ my_job.py arg1 arg2
Everything after the script name goes to the program as arguments.
Key Options
| Option | Purpose |
|---|---|
| --master | Names the cluster manager |
| --deploy-mode | Chooses client or cluster |
| --executor-memory | Sets memory per executor |
| --executor-cores | Sets cores per executor |
| --driver-memory | Sets memory for the driver |
| --conf key=value | Sets any Spark setting |
| --py-files | Ships extra Python files |
| --jars and --packages | Adds Java libraries |
| --files | Ships data or config files to every executor |
What Happens After You Press Enter
spark-submit
|
v
Cluster manager accepts the application
|
v
Driver starts (inside the cluster in cluster mode)
|
v
Driver requests executors
|
v
Executors start, load code, register with driver
|
v
Driver sends tasks, executors return results
|
v
Application finishes, resources return to the cluster
A Deployable Script
Keep the master and sizing choices outside the code. Read inputs from arguments.
# my_job.py
import sys
from pyspark.sql import SparkSession, functions as F
def main(input_path, output_path):
spark = SparkSession.builder.appName("daily_sales_report").getOrCreate()
sales = spark.read.parquet(input_path)
report = sales.groupBy("region").agg(F.sum("amount").alias("total"))
report.write.mode("overwrite").parquet(output_path)
spark.stop()
if __name__ == "__main__":
main(sys.argv[1], sys.argv[2])
spark-submit --master yarn --deploy-mode cluster my_job.py /data/sales /data/reports
Shipping Python Dependencies
Executors run on other machines, so they need your helper modules and libraries.
# Zip your own modules
zip -r helpers.zip helpers/
spark-submit --py-files helpers.zip my_job.py
# Ship a packaged Python environment
spark-submit --archives my_env.tar.gz#environment \
--conf spark.pyspark.python=./environment/bin/python my_job.py
Using a Settings File
Long commands become hard to read. Move stable settings into a properties file.
# job.conf spark.executor.memory 8g spark.executor.cores 4 spark.sql.adaptive.enabled true spark-submit --properties-file job.conf my_job.py
Command line values override the file, and the file overrides defaults.
Scheduling Runs
A scheduler such as Apache Airflow, cron, or a cloud workflow service launches spark-submit at set times. The scheduler tracks success, retries failures, and orders dependent jobs.
Scheduler --> spark-submit (ingest) --> spark-submit (transform) --> spark-submit (report)
Deployment Checklist
- Test the script in local mode with a small sample first.
- Pass paths and dates as arguments instead of editing code.
- Choose cluster mode for scheduled jobs.
- Store credentials in a secrets service, never in the script or the command.
- Send logs to central storage so you can read them after the job ends.
- Return a non-zero exit code on failure so the scheduler notices.
Handling Failures
Clusters lose machines, and networks drop connections. Spark retries failed tasks and, in cluster mode, can restart a failed driver.
spark.task.maxFailures 4 spark.yarn.maxAppAttempts 2
Write results in a way that tolerates reruns. An overwrite of a dated folder produces the same output no matter how many times the job runs.
Run 1 fails halfway --> rerun --> overwrite folder /reports/2026-01-15 --> clean result