Spark UI and Monitoring
The Spark UI is a web page that shows what your application does while it runs. It reveals slow stages, uneven tasks, memory problems, and wasted shuffles. Reading it turns tuning from guesswork into a clear investigation.
The Layman View
A car dashboard shows speed, fuel, and engine warnings. The Spark UI works as the dashboard of your job. A glance tells you whether the job runs smoothly or needs attention.
Opening the UI
A running application serves the UI on port 4040 of the driver machine. A second application on the same machine uses port 4041. The cluster manager pages link to the UI of each application.
http://driver-host:4040
How Work Is Organized
Application | |-- Job 1 (started by one action, such as count or write) | |-- Stage 1 (work that needs no shuffle) | | |-- Task 1, Task 2, Task 3 ... (one per partition) | |-- Stage 2 (starts after a shuffle) | |-- Task 1, Task 2, Task 3 ... | |-- Job 2
Each action creates a job. Each shuffle splits a job into stages. Each partition becomes a task.
The Main Tabs
- Jobs lists every job with its duration and progress, plus a timeline of executors.
- Stages lists stages with task counts, input size, and shuffle size.
- Storage shows cached DataFrames and how much memory they use.
- Environment prints every setting and library path.
- Executors shows memory, cores, garbage collection time, and failed tasks for each executor.
- SQL draws the query plan with row counts at each step.
Reading a Stage Page
The summary metrics table shows the minimum, median, and maximum of task measurements. Healthy stages show similar numbers across the row.
Metric Min Median Max Duration 2 s 3 s 4 s healthy Shuffle Read 40 MB 45 MB 52 MB Metric Min Median Max Duration 2 s 3 s 12 min skewed: one task drags the stage Shuffle Read 40 MB 45 MB 9 GB
Warning Signs and Meanings
| What You See | What It Means |
|---|---|
| Maximum task time far above the median | Data skew |
| Large values in Spill (Memory) and Spill (Disk) | Tasks lack memory and write temporary data to disk |
| High GC time in Executors tab | Executors hold too many objects |
| Thousands of tiny tasks | Too many small partitions or files |
| Few huge tasks | Too few partitions |
| Repeated failed tasks | Bad data, memory limits, or lost executors |
Using the SQL Tab
Click a query to see its plan as a diagram. Each box shows the rows it produced and the time it took. Find the box with the largest row count or time and focus your tuning there.
Scan parquet (500M rows)
|
Filter (2M rows remain)
|
Exchange (shuffle) <-- check size here
|
HashAggregate
The History Server
The live UI vanishes when the application ends. The Spark History Server keeps finished applications available. Turn on event logging so each job writes its record.
spark.eventLog.enabled true spark.eventLog.dir hdfs:///spark-logs spark.history.fs.logDirectory hdfs:///spark-logs
./sbin/start-history-server.sh
A Simple Investigation Routine
- Open the Jobs tab and find the longest job.
- Open its slowest stage.
- Compare minimum, median, and maximum task times.
- Check spill and shuffle numbers.
- Open the SQL tab and find the heavy operator.
- Change one setting or one line of code and compare the new run.