Apache Spark - Best Practices and Tuning
Search…
Introduction
RDD
Don’t collect large RDDs
Don't use count() when you don't need to return the exact number of rows
Avoiding Shuffle "Less stage, run faster"
Picking the Right Operators
TreeReduce and TreeAggregate Demystified
When to use Broadcast variable
Joining a large and a small RDD
Joining a large and a medium size RDD
Dataframe
Joining a large and a small Dataset
Joining a large and a medium size Dataset
Storage
Use the Best Data Format
Cache Judiciously and use Checkpointing
Parallelism
Use the right level of parallelism
How to estimate the size of a Dataset
How to estimate the number of partitions, executor's and driver's params (YARN Cluster Mode)
Serialization and GC
Tuning Java Garbage Collection
Serialization
References
References
Powered By GitBook
Serialization
Serialization plays an important role in the performance of any distributed application. Formats that are slow to serialize objects into, or consume a large number of bytes, will greatly slow down the computation. Often, this will be the first thing you should tune to optimize a Spark application. The Java default serializer has very mediocre performance regarding runtime as well as the size of its results. Therefore the Spark team recommends to use the Kryo serializer instead.
The following code shows an example of how you can turn on Kryo and how you can register the classes that you will be serializing:
1
val conf = new SparkConf().setAppName(...).setMaster(...)
2
.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
3
.set("spark.kryoserializer.buffer.max", "128m")
4
.set("spark.kryoserializer.buffer", "64m")
5
.registerKryoClasses(
6
Array(classOf[ArrayBuffer[String]], classOf[ListBuffer[String]])
7
)
Copied!
You can find another example here BasicAvgWithKryo ​
Serialization and GC - Previous
Tuning Java Garbage Collection
Next - References
References
Last modified 2yr ago
Copy link