Spark SparkSession and Configuration

SparkSession is the front door to every Spark application. You create it once at the top of your program, and it gives you access to DataFrames, SQL, file readers, and settings. Modern Spark code starts with a SparkSession, and older code that begins with SparkContext now runs through it.

The Layman View

Think of a hotel reception desk. Guests ask the desk for rooms, meals, and taxis, and the desk contacts the right department. SparkSession plays the reception desk. You ask it to read a file, run a query, or change a setting, and it calls the right part of Spark.

What Sits Behind the Session

SparkSession (spark)
   |
   |-- spark.read          load data into DataFrames
   |-- spark.sql(...)      run SQL queries
   |-- spark.conf          read and change settings
   |-- spark.catalog       list databases, tables, and views
   |-- spark.udf           register custom functions
   |-- spark.sparkContext  low-level RDD access

Creating a Session

from pyspark.sql import SparkSession

spark = (SparkSession.builder
         .appName("sales_report")
         .master("local[*]")
         .config("spark.sql.shuffle.partitions", "50")
         .getOrCreate())

• appName labels the application in the Spark UI and cluster pages.
• master names the cluster manager. Skip it in production code and pass it through spark-submit.
• config sets one option. Chain as many calls as you need.
• getOrCreate returns the running session or builds a new one. Two calls never produce two sessions in the same program.

The shells pyspark and spark-shell create a session named spark before you type anything.

Reading and Changing Settings

print(spark.conf.get("spark.sql.shuffle.partitions"))   # 50

spark.conf.set("spark.sql.shuffle.partitions", "100")
spark.conf.set("spark.sql.adaptive.enabled", "true")

Settings that affect SQL behaviour change at runtime. Settings that size the cluster, such as executor memory and the number of cores, lock in when the application starts. Changing them later has no effect.

Runtime settings (change any time)     Startup settings (fixed after launch)
spark.sql.shuffle.partitions           spark.executor.memory
spark.sql.adaptive.enabled             spark.executor.cores
spark.sql.autoBroadcastJoinThreshold   spark.driver.memory

Where Settings Come From

Spark reads settings from several places. A higher level overrides a lower level.

1. Code: SparkSession.builder.config(...)        highest priority
2. spark-submit command line options
3. Properties file: conf/spark-defaults.conf
4. Built-in defaults                              lowest priority

A team often keeps stable values in the properties file and passes job-specific values on the command line.

Useful Settings to Know

Setting	Purpose
spark.sql.shuffle.partitions	Number of partitions after a shuffle
spark.sql.adaptive.enabled	Turns on adaptive query execution
spark.sql.autoBroadcastJoinThreshold	Largest table size that Spark broadcasts
spark.executor.memory	Memory for each executor
spark.sql.session.timeZone	Time zone for timestamp handling
spark.sql.warehouse.dir	Folder for managed tables

Reaching the SparkContext

The session wraps a SparkContext. Use it when you need RDD features.

sc = spark.sparkContext
rdd = sc.parallelize([1, 2, 3, 4])
sc.setLogLevel("WARN")

Several Sessions in One Program

The newSession method creates a second session that shares the same cluster resources but keeps separate temporary views and runtime settings. Notebook tools use this to isolate users.

other = spark.newSession()
other.conf.set("spark.sql.shuffle.partitions", "10")

Stopping the Session

spark.stop()

Call stop at the end of a script to release executors and flush logs. Notebooks and shells stop the session when you close them.

Good Habits

• Create the session in one place and pass it to functions as an argument.
• Keep the master URL and memory sizes out of the code.
• Print important settings at the start of a job. The log then records the exact configuration of each run.