Spark Cluster Managers

A cluster manager hands out machines, memory, and CPU cores to Spark applications. Spark itself plans the work. The cluster manager owns the hardware. Choosing the right manager decides how you start jobs, share resources among teams, and scale in production.

The Layman View

A construction company wins a large building contract. The site manager plans the tasks, but a labour office supplies the workers. Spark plays the site manager, and the cluster manager plays the labour office. Spark asks for workers, and the office assigns them.

Who Talks to Whom

Driver program
    |
    |  1. request resources
    v
Cluster manager  ----->  Worker node 1 : Executor
                   ----->  Worker node 2 : Executor
                   ----->  Worker node 3 : Executor
    ^
    |  2. executors register back with the driver
Driver sends tasks directly to executors

Available Options

Manager	Master URL	Best For
Local	local[4]	Learning and testing on one computer
Standalone	spark://host:7077	Small dedicated Spark clusters
YARN	yarn	Hadoop environments
Kubernetes	k8s://https://host:6443	Container-based platforms and cloud

Local Mode

Local mode runs the driver and executors inside one process. The number in brackets sets the thread count. The star symbol uses every core.

spark = SparkSession.builder.master("local[*]").appName("practice").getOrCreate()

Standalone Mode

Spark includes its own simple manager. One master process tracks workers, and each worker offers cores and memory.

# On the master machine
./sbin/start-master.sh

# On each worker machine
./sbin/start-worker.sh spark://master-host:7077

# Submit a job
spark-submit --master spark://master-host:7077 my_job.py

The master web page on port 8080 lists workers and running applications.

YARN

YARN comes with Hadoop and shares one cluster among many frameworks. Spark requests containers from the YARN Resource Manager. Each container runs one executor.

spark-submit --master yarn --deploy-mode cluster --num-executors 10 my_job.py

Kubernetes

Kubernetes runs each driver and executor as a pod built from a container image. Teams value its portability across clouds and its simple scaling.

spark-submit \
  --master k8s://https://k8s-api.example.com:6443 \
  --deploy-mode cluster \
  --conf spark.executor.instances=5 \
  --conf spark.kubernetes.container.image=myrepo/spark-app:1.0 \
  local:///opt/app/my_job.py

Deploy Modes

Client mode:                          Cluster mode:

Your laptop                           Your laptop
  [Driver]                              submit and disconnect
     |                                       |
     v                                       v
  Cluster [Executors]                   Cluster [Driver + Executors]

• Client mode keeps the driver on the submitting machine. Output appears in your terminal. The job stops if your connection drops.
• Cluster mode starts the driver inside the cluster. The job survives your disconnection. Production jobs use this mode.

Dynamic Allocation

Spark can add executors when work piles up and release them when idle. Teams that share a cluster save resources with this feature.

spark.dynamicAllocation.enabled=true
spark.dynamicAllocation.minExecutors=2
spark.dynamicAllocation.maxExecutors=50

How to Choose

• Pick local mode for learning.
• Pick Standalone for a small cluster that only runs Spark.
• Pick YARN when the company already runs Hadoop.
• Pick Kubernetes for container-first teams and cloud deployments.
• Keep the master URL outside your code. Pass it through spark-submit so one script runs everywhere.

Sizing Executors

An executor needs enough memory for its tasks and enough cores to run them in parallel. Very large executors suffer long garbage collection pauses. Very small executors waste memory on overhead.

Machine: 16 cores, 64 GB RAM

Option A: 1 executor  x 16 cores x 60 GB  --> long pauses
Option B: 16 executors x 1 core  x 3.5 GB --> heavy overhead
Option C: 3 executors  x 5 cores x 18 GB  --> balanced choice

A common starting point gives each executor four to five cores. Leave one core and some memory for the operating system.

Monitoring the Cluster

Each manager offers a web page. The standalone master page lists workers. The YARN Resource Manager page lists applications and containers. The Kubernetes command kubectl get pods lists drivers and executors. The Spark UI of each application shows task details on top of these views.