DE Apache Spark Basics
Apache Spark is the most widely used distributed data processing framework in data engineering today. It processes large datasets across a cluster of machines in parallel, completing in minutes work that would take hours on a single server. Spark powers batch pipelines, streaming systems, machine learning workflows, and SQL-based analytics — often within the same unified framework.
What Spark Does
Spark takes a large dataset, divides it into smaller chunks, distributes those chunks across many machines, and processes all chunks simultaneously. When the processing finishes, Spark collects the results and returns them. This parallel execution gives Spark its speed advantage over sequential processing on a single machine.
The Newspaper Printing Analogy
Printing one million newspapers on a single printing press takes a week. Run 100 presses in parallel, each printing 10,000 newspapers simultaneously, and the job completes in hours. Apache Spark works the same way — instead of one machine processing all the data, a Spark cluster uses many machines working simultaneously on different portions of the dataset.
Spark vs Hadoop MapReduce
Spark outperforms MapReduce primarily because it keeps data in memory (RAM) during processing. MapReduce writes intermediate results to disk after every step. For a computation with 10 steps, MapReduce reads and writes disk 10 times. Spark reads from disk once, processes all 10 steps in memory, and writes the final result once.
MapReduce (disk I/O at every step):
Input --> [Disk] --> Step 1 --> [Disk] --> Step 2 --> [Disk] --> Output
Spark (in-memory processing):
Input --> [Memory] --> Step 1 --> Step 2 --> Step 3 --> Output --> [Disk]
(read once) (write once)
Result: Spark is typically 10x-100x faster than MapReduce.
The Spark Architecture
Driver
The driver is the central control process. It contains the main application logic, coordinates the overall job, and communicates with the cluster manager. The driver breaks the job into tasks and assigns them to executors.
Cluster Manager
The cluster manager handles resource allocation — deciding which machines run which tasks. Common cluster managers include YARN (in Hadoop environments), Apache Mesos, Kubernetes, and Spark's own built-in standalone manager.
Executors
Executors are the worker processes. Each executor runs on a worker machine in the cluster, performs the assigned tasks, and stores intermediate data in memory or on local disk. Multiple executors run simultaneously across the cluster.
Spark Cluster Architecture:
[Driver Program]
|
| (job instructions)
v
[Cluster Manager] --> [Executor on Machine 1] --> processes partition 1
--> [Executor on Machine 2] --> processes partition 2
--> [Executor on Machine 3] --> processes partition 3
--> [Executor on Machine 4] --> processes partition 4
RDDs: Resilient Distributed Datasets
The fundamental data structure in Spark is the RDD — a Resilient Distributed Dataset. An RDD is an immutable, distributed collection of records partitioned across the cluster. "Resilient" means if a partition is lost due to a machine failure, Spark can recompute it from the original data using the recorded lineage of transformations. RDDs are the low-level API; most modern Spark code uses DataFrames instead.
Transformations and Actions
Spark operations divide into two categories with fundamentally different behavior.
Transformations
Transformations define a new dataset derived from an existing one — filtering, mapping, grouping. Spark does not execute transformations immediately. Instead it records them in a logical plan — a concept called lazy evaluation. Nothing actually runs until an action triggers execution.
Actions
Actions trigger the execution of all accumulated transformations and return a result. Common actions include collect() (retrieve all rows to the driver), count() (return the number of rows), show() (display a sample), and write() (save results to storage).
Lazy Evaluation Example:
# These transformations build a plan but do NOT run yet
filtered = orders.filter(orders.amount > 1000) # Transformation
grouped = filtered.groupBy("customer_id") # Transformation
totals = grouped.sum("amount") # Transformation
# THIS action triggers execution of all three steps at once
totals.show() # Action -- Spark runs everything here
Spark APIs
Spark SQL
Spark SQL lets engineers write standard SQL queries against Spark DataFrames. The SQL runs in a distributed fashion across the cluster. This makes Spark accessible to anyone with SQL skills.
DataFrames API
The DataFrame API provides a programmatic interface to manipulate structured data using Python, Scala, Java, or R. It is the most commonly used Spark API in modern data engineering.
Structured Streaming
Structured Streaming extends the DataFrame API to real-time data streams. Engineers write the same DataFrame-style code for both batch and streaming — Spark handles the differences internally.
MLlib
MLlib provides a library of distributed machine learning algorithms that run at scale across the Spark cluster.
Where Spark Runs
Environment | How Spark Runs ---------------------|------------------------------------------ On-premise cluster | YARN on Hadoop, standalone, Kubernetes AWS | Amazon EMR, AWS Glue (serverless Spark) Google Cloud | Google Dataproc, Dataflow Azure | Azure Databricks, Azure HDInsight Databricks | Managed Spark; most popular enterprise platform Local machine | Single-machine mode for development and testing
Summary
Apache Spark is a distributed processing framework that processes large datasets across clusters of machines in parallel, with in-memory computation making it far faster than MapReduce. Its architecture consists of a driver, cluster manager, and worker executors. Transformations define a logical plan lazily; actions trigger execution. Spark supports SQL, DataFrame, streaming, and machine learning workloads through a unified API, making it the central tool for large-scale data processing in modern data engineering.