Spark Datasets

A Dataset is a typed, distributed collection of data. It combines the speed and optimization of DataFrames with the type safety of RDDs. Datasets are only available in Scala and Java — Python uses DataFrames exclusively (Python is dynamically typed so compile-time type checking does not apply).

The Three APIs — How They Relate

+------------------+------------------+------------------+
|     RDD          |   DataFrame      |   Dataset        |
+------------------+------------------+------------------+
| Untyped          | Typed columns    | Fully typed      |
| No optimization  | Optimized        | Optimized        |
| Python, Scala,   | Python, Scala,   | Scala, Java only |
|   Java, R        |   Java, R        |                  |
| JVM objects      | Row objects      | Case class /     |
|                  |                  |   Java Bean      |
+------------------+------------------+------------------+
     Raw power        Best for most       Best when you
                      Python work         want type safety
                                          in Scala/Java

Why Type Safety Matters

Type safety means the compiler catches errors before your code runs. With a DataFrame, you can accidentally reference a column that does not exist and only discover the error at runtime (after waiting for a job to fail). A Dataset catches that error during compilation.

// Scala Dataset example

// Without Dataset (DataFrame):
df.select("salary_amount")
// Error: AnalysisException — only found at RUNTIME

// With Dataset:
case class Employee(name: String, age: Int, salary: Double)
val ds: Dataset[Employee] = df.as[Employee]
ds.map(e => e.salaryAmount)   // compile error — "salaryAmount" doesn't exist
// Error caught at COMPILE TIME — before any job starts

Creating a Dataset in Scala

// Define a case class to describe the shape of one record
case class Sale(product: String, region: String, amount: Double)

// Read data as a DataFrame, then convert to Dataset
val salesDF = spark.read.option("header", "true").csv("sales.csv")
val salesDS: Dataset[Sale] = salesDF.as[Sale]

// Now you have type-safe operations
val northSales = salesDS.filter(_.region == "North")
val totalNorth = northSales.map(_.amount).reduce(_ + _)
println(totalNorth)

Dataset Operations

// All DataFrame operations work on Datasets too
salesDS.show()
salesDS.printSchema()
salesDS.groupBy("region").count().show()

// Plus typed lambda operations
salesDS
  .filter(sale => sale.amount > 5000)
  .map(sale => (sale.region, sale.amount))
  .show()

Converting Between APIs

// DataFrame to Dataset
val ds: Dataset[Sale] = df.as[Sale]

// Dataset to DataFrame
val df2: DataFrame = ds.toDF()

// Dataset to RDD
val rdd = ds.rdd

// RDD to Dataset
val ds2 = rdd.toDS()

When to Use Each API

Encoder — How Datasets Know the Type

Datasets use Encoders to convert between JVM objects (case classes) and Spark's internal binary format. Spark provides encoders automatically for common types and case classes. You rarely need to define one manually.

import org.apache.spark.sql.Encoders

// Spark automatically creates this encoder for case classes
val encoder = Encoders.product[Sale]
println(encoder.schema)
// Output: StructType(StructField(product,StringType,true),...)

Dataset Performance

Datasets perform the same as DataFrames for SQL-style operations because both use Catalyst and Tungsten under the hood. The only overhead comes when using typed lambda functions, which require serialization between the JVM object and Spark's internal format. For pure speed with SQL queries, DataFrames and Datasets are equivalent.

Key Takeaways

• Datasets add compile-time type safety on top of DataFrames
• Python users work with DataFrames — Datasets are for Scala and Java
• Converting between APIs is straightforward with .as[T]() and .toDF()
• Performance equals DataFrames for most operations