Auron Support

Build Auron

Auron supports Celeborn as remote shuffle service. Below introduction is used to enable this feature.

First refer to Build From Source or Build With Docker to build Auron.

Auron Configuration

Currently, to use Auron following configurations are required in spark-defaults.conf.

spark.shuffle.manager org.apache.spark.sql.execution.auron.shuffle.celeborn.AuronCelebornShuffleManager

# celeborn master
spark.celeborn.master.endpoints clb-master:9097

spark.celeborn.client.spark.shuffle.writer hash
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.sql.adaptive.localShuffleReader.enabled false

Availability

Celeborn Version	Available in Auron?
< 0.5.0	No
>= 0.5.0	Yes