partitionを減らすとcoalesceの方が効率的です

3167 ワード

partitionを減らすとcoalesceの方が効率的です
テスト
repartition,shuffle 2.8 G,消費時間10 min 39 sec

    
    
    
    
  1. df.rdd.repartition(1).saveAsTextFile("/gx/gziptest", classOf[org.apache.hadoop.io.compress.GzipCodec])

joe: start time: Tue Jul 07 12:43:06 CST 2015
joe: end time: Tue Jul 07 12:53:45 CST 2015
coalesce、shuffleなし、6 min 22 secかかります

    
    
    
    
  1. df.rdd.coalesce(1).saveAsTextFile("/gx/gziptest", classOf[org.apache.hadoop.io.compress.GzipCodec])

joe: start time: Tue Jul 07 13:39:16 CST 2015
joe: end time: Tue Jul 07 13:45:38 CST 2015
説明
repartition(numPartitions) 
Reshuffle the data in the RDD randomly to create either more or fewer partitions and balance it across them. This always shuffles all data over the network. 
に等しい
coalesce(numPartitions, shuffle = true)
coalesce(numPartitions)    
Decrease the number of partitions in the RDD to numPartitions. Useful for running operations more efficiently after filtering down a large dataset.
If you are decreasing the number of partitions in this RDD, consider using `coalesce`, 
which can avoid performing a shuffle.
However, if you're doing a drastic coalesce, e.g. to numPartitions = 1,
 this may result in your computation taking place on fewer nodes than
 you like (e.g. one node in the case of numPartitions = 1). To avoid this, 
you can pass shuffle = true. This will add a shuffle step, but means the 
 current upstream partitions will be executed in parallel (per whatever
 the current partitioning is).
為知ノート(Wiz)から