Avoid reduceByKey when the input and output value types are different - Apache Spark - Best Practices and Tuning

Apache Spark - Best Practices and Tuning

Search…

RDD

Don’t collect large RDDs

Don't use count() when you don't need to return the exact number of rows

Avoiding Shuffle "Less stage, run faster"

Picking the Right Operators

Avoid List of Iterators

Avoid groupByKey when performing a group of multiple items by key

Avoid groupByKey when performing an associative reductive operation

Avoid reduceByKey when the input and output value types are different

Avoid the flatMap-join-groupBy pattern

Use TreeReduce/TreeAggregate instead of Reduce/Aggregate

Hash-partition before transformation over pair RDD

Use coalesce to repartition in decrease number of partition

TreeReduce and TreeAggregate Demystified

When to use Broadcast variable

Joining a large and a small RDD

Joining a large and a medium size RDD

Dataframe

Joining a large and a small Dataset

Joining a large and a medium size Dataset

Storage

Use the Best Data Format

Cache Judiciously and use Checkpointing

Parallelism

Use the right level of parallelism

How to estimate the size of a Dataset

How to estimate the number of partitions, executor's and driver's params (YARN Cluster Mode)

Serialization and GC

Tuning Java Garbage Collection

References

Powered By GitBook

Avoid reduceByKey when the input and output value types are different

For example, consider writing a transformation that finds all the unique strings corresponding to each key. One way would be to use map to transform each element into a Set and then combine the Sets with reduceByKey:
rdd.map(kv => (kv._1, new Set[String]() + kv._2)) .reduceByKey(_ ++ _)
This code results in tons of unnecessary object creation because a new Set must be allocated for each record. It’s better to use aggregateByKey, which performs the map-side aggregation more efficiently:
val zero = new collection.mutable.Set[String]() 
rdd.aggregateByKey(zero)( (set, v) => set += v, (set1, set2) => set1 ++= set2)

Avoid groupByKey when performing an associative reductive operation

Avoid the flatMap-join-groupBy pattern

Last modified 2yr ago

Copy link