Spark Pair RDDs

A pair RDD stores every record as a key and value tuple, such as ("apple", 5). The key names the group. The value holds the number or text that Spark combines. Word counts, sales totals, and log summaries all rely on pair RDDs because they let Spark gather related records from many machines into one answer.

A Simple Way to Picture It

Imagine a school register. Each student name is a key, and each exam mark is a value. A student may have several marks. A teacher who wants the total per student groups the marks by name and adds them. A pair RDD does the same job for millions of records.

Creating a Pair RDD

Any RDD becomes a pair RDD when each element turns into a two-item tuple. The map transformation does the conversion.

sales = sc.parallelize([
    ("apple", 5), ("banana", 3), ("apple", 2),
    ("banana", 4), ("cherry", 7)
])

# Build pairs from plain text lines
lines = sc.parallelize(["spark is fast", "spark is simple"])
pairs = lines.flatMap(lambda l: l.split(" ")).map(lambda w: (w, 1))

Core Pair RDD Operations

  • reduceByKey merges the values of each key with a function you supply.
  • groupByKey collects all values of each key into one list.
  • mapValues changes the value and leaves the key untouched.
  • sortByKey orders the records by key.
  • keys and values return only one side of each pair.
  • countByKey returns the number of records for every key to the driver.

Totals per Key

totals = sales.reduceByKey(lambda a, b: a + b)
print(totals.collect())
# [('apple', 7), ('banana', 7), ('cherry', 7)]

doubled = totals.mapValues(lambda v: v * 2)
print(doubled.sortByKey().collect())
# [('apple', 14), ('banana', 14), ('cherry', 14)]

How reduceByKey Works Inside

Spark combines values inside each partition before it moves any data. The local step shrinks the data, and the shuffle then carries less across the network.

Partition 1: (apple,5) (banana,3) (apple,2)
Partition 2: (banana,4) (cherry,7)

Step 1 - Local combine on each partition
Partition 1 --> (apple,7) (banana,3)
Partition 2 --> (banana,4) (cherry,7)

Step 2 - Shuffle records with the same key to the same machine
Machine A: (apple,7)
Machine B: (banana,3) (banana,4)
Machine C: (cherry,7)

Step 3 - Final combine
(apple,7)  (banana,7)  (cherry,7)

reduceByKey Versus groupByKey

groupByKey sends every single value across the network before any addition happens. A key with millions of values can overload one executor and cause memory errors. reduceByKey adds values early and sends small partial results. Choose reduceByKey, aggregateByKey, or combineByKey whenever the task allows a combine function.

# Heavy: every value travels over the network
sales.groupByKey().mapValues(sum)

# Light: partial sums travel over the network
sales.reduceByKey(lambda a, b: a + b)

Joining Two Pair RDDs

The join operation matches records that share a key. Each match produces a key with a tuple of both values.

prices = sc.parallelize([("apple", 40), ("banana", 10)])
stock  = sc.parallelize([("apple", 100), ("banana", 250), ("kiwi", 30)])

print(prices.join(stock).collect())
# [('apple', (40, 100)), ('banana', (10, 250))]

print(prices.leftOuterJoin(stock).collect())
print(prices.rightOuterJoin(stock).collect())

The kiwi record disappears from the inner join because no price exists for it. A right outer join keeps kiwi and fills the missing price with None.

Average per Key

An average needs both a sum and a count. Pair each value with the number 1, add both parts, and divide at the end.

scores = sc.parallelize([("Ana", 80), ("Ana", 90), ("Raj", 70)])

avg = (scores.mapValues(lambda v: (v, 1))
             .reduceByKey(lambda a, b: (a[0] + b[0], a[1] + b[1]))
             .mapValues(lambda t: t[0] / t[1]))

print(avg.collect())   # [('Ana', 85.0), ('Raj', 70.0)]

Good Habits

  • Keep keys small and simple, such as strings or integers.
  • Avoid collecting a large pair RDD to the driver. Use take or save the result to storage.
  • Check key distribution. A single popular key slows one task and delays the whole stage.
  • Move to DataFrames for table-like data. Their optimizer handles many of these steps automatically.
Previous lesson
Back to course
Next lesson