partitionを減らすとcoalesceの方が効率的です
3167 ワード
partitionを減らすとcoalesceの方が効率的です
テスト
repartition,shuffle 2.8 G,消費時間10 min 39 sec
joe: start time: Tue Jul 07 12:43:06 CST 2015
joe: end time: Tue Jul 07 12:53:45 CST 2015
coalesce、shuffleなし、6 min 22 secかかります
joe: start time: Tue Jul 07 13:39:16 CST 2015
joe: end time: Tue Jul 07 13:45:38 CST 2015
説明
repartition(numPartitions)
Reshuffle the data in the RDD randomly to create either more or fewer partitions and balance it across them. This always shuffles all data over the network.
に等しい
coalesce(numPartitions, shuffle = true)
coalesce(numPartitions)
Decrease the number of partitions in the RDD to numPartitions. Useful for running operations more efficiently after filtering down a large dataset.
If you are decreasing the number of partitions in this RDD, consider using `coalesce`,
which can avoid performing a shuffle.
However, if you're doing a drastic coalesce, e.g. to numPartitions = 1,
this may result in your computation taking place on fewer nodes than
you like (e.g. one node in the case of numPartitions = 1). To avoid this,
you can pass shuffle = true. This will add a shuffle step, but means the
current upstream partitions will be executed in parallel (per whatever
the current partitioning is).
為知ノート(Wiz)から
テスト
repartition,shuffle 2.8 G,消費時間10 min 39 sec
df.rdd.repartition(1).saveAsTextFile("/gx/gziptest", classOf[org.apache.hadoop.io.compress.GzipCodec])
joe: start time: Tue Jul 07 12:43:06 CST 2015
joe: end time: Tue Jul 07 12:53:45 CST 2015
coalesce、shuffleなし、6 min 22 secかかります
df.rdd.coalesce(1).saveAsTextFile("/gx/gziptest", classOf[org.apache.hadoop.io.compress.GzipCodec])
joe: start time: Tue Jul 07 13:39:16 CST 2015
joe: end time: Tue Jul 07 13:45:38 CST 2015
説明
repartition(numPartitions)
Reshuffle the data in the RDD randomly to create either more or fewer partitions and balance it across them. This always shuffles all data over the network.
に等しい
coalesce(numPartitions, shuffle = true)
coalesce(numPartitions)
Decrease the number of partitions in the RDD to numPartitions. Useful for running operations more efficiently after filtering down a large dataset.
If you are decreasing the number of partitions in this RDD, consider using `coalesce`,
which can avoid performing a shuffle.
However, if you're doing a drastic coalesce, e.g. to numPartitions = 1,
this may result in your computation taking place on fewer nodes than
you like (e.g. one node in the case of numPartitions = 1). To avoid this,
you can pass shuffle = true. This will add a shuffle step, but means the
current upstream partitions will be executed in parallel (per whatever
the current partitioning is).
為知ノート(Wiz)から