Failed to execute user defined function(anonfun$2: (array) => double)
エラーメッセージ:
コード:
修正後のudf
org.apache.spark.SparkException: Failed to execute user defined function(anonfun$2: (array) => double)
Caused by: java.lang.ClassCastException: scala.collection.mutable.WrappedArray$ofRef cannot be cast to [D
コード:
// udf array
val zeroRate = udf((arr:Array[Double])=>{
arr.count(_ == 0).doubleValue()/arr.length.doubleValue()
})
// column collect_list
df.groupBy("group").agg(
collect_list("rank").alias("rank")
).withColumn("rankZero",zeroRate($"rank")).show()
//root
// |-- domain: string (nullable = true)
// |-- rank: array (nullable = true)
// | |-- element: double (containsNull = true)
修正後のudf
// Array -> Seq
val zeroRate = udf((arr:Seq[Double])=>{
arr.count(_ == 0).doubleValue()/arr.length.doubleValue()
})