Failed to execute user defined function(anonfun$2: (array) => double)

4593 ワード

エラーメッセージ:
org.apache.spark.SparkException: Failed to execute user defined function(anonfun$2: (array) => double)
Caused by: java.lang.ClassCastException: scala.collection.mutable.WrappedArray$ofRef cannot be cast to [D

コード:
	//       udf     array
    val zeroRate = udf((arr:Array[Double])=>{
      arr.count(_ == 0).doubleValue()/arr.length.doubleValue()
    })
    //        column    collect_list   
    df.groupBy("group").agg(
      collect_list("rank").alias("rank")
    ).withColumn("rankZero",zeroRate($"rank")).show()
//root
// |-- domain: string (nullable = true)
// |-- rank: array (nullable = true)
// |    |-- element: double (containsNull = true)

修正後のudf
	//     Array -> Seq
	val zeroRate = udf((arr:Seq[Double])=>{
      arr.count(_ == 0).doubleValue()/arr.length.doubleValue()
    })