# reduceByKey: How does it work internally ?

reduceByKey: How does it work internally ?

For the discrete methods and types. This is usually exposes the intricacies for new devs:

pairs.reduceByKey((a, b) => a + b)

Then it becomes

pairs.reduceByKey((a: Int, b: Int) => a + b)

During renaming the variables makes it a little more stated clearly

pairs.reduceByKey((accumulatedValue: Int, currentValue: Int) => accumulatedValue + currentValue)

Here just taking an accumulated value for the given key and summing it with the next value of that key. NOW, for the further we could understand the key part. So, the method is visualized more like this:

pairs.reduce((accumulatedValue: List[(String, Int)], currentValue: (String, Int)) => { //Turn the accumulated value into a true key->value mapping val accumAsMap = accumulatedValue.toMap //Try to get the key's current value if we've already encountered it accumAsMap.get(currentValue._1) match { //If we have encountered it, then add the new value to the existing value and overwrite the old case Some(value : Int) => (accumAsMap + (currentValue._1 -> (value + currentValue._2))).toList //If we have NOT encountered it, then simply add it to the list case None => currentValue :: accumulatedValue } })

So, Here we could notice that the** reduceByKey** takes the boilerplate of finding the key and tracking it so there will no worries for managing that part.

Here this is alternative solution

For instance:

val counts = pairs.reduceByKey((a,b) => a+b)

In the above code **a** and **b** are both Int accumulators for _2 of the tuples in pairs. reduceKey will take two tuples with the same value s and use their _2 values as a and b, producing a new Tuple[String,Int]. This operation is done again and again until there is only one tuple for each key s.

With dissimilar of non-Spark (or, really, non-parallel) reduceByKey in which the first element is always the accumulator and the second a value, reduceByKey operates in a distributed fashion, This says each node will reduce it’s set of tuples into a collection of uniquely-keyed tuples and then reduce the tuples from multiple nodes until there is a final uniquely-keyed set of tuples. As the results from nodes are reduced, **a** and **b** represent already reduced accumulators.