在Spark DataFrame中将空值转换为空数组

我有一个Spark数据框,其中一列是整数数组.该列可以为空,因为它来自左外连接.我想将所有空值转换为空数组,因此我不必在以后处理空值.

我以为我可以这样做:

val myCol = df("myCol")
df.withColumn( "myCol", when(myCol.isNull, Array[Int]()).otherwise(myCol) )

但是,这会导致以下异常:

java.lang.RuntimeException: Unsupported literal type class [I [I@5ed25612
at org.apache.spark.sql.catalyst.expressions.Literal$.apply(literals.scala:49)
at org.apache.spark.sql.functions$.lit(functions.scala:89)
at org.apache.spark.sql.functions$.when(functions.scala:778)

显然,该when函数不支持数组类型.还有其他一些简单的方法来转换空值吗？

如果它是相关的,这是此列的架构:

|-- myCol: array (nullable = true)
|    |-- element: integer (containsNull = false)

zero323.. 19

您可以使用UDF:

import org.apache.spark.sql.functions.udf

val array_ = udf(() => Array.empty[Int])

df.withColumn("myCol", when(myCol.isNull, array_()).otherwise(myCol))
df.withColumn("myCol", coalesce(myCol, array_())).show

import org.apache.spark.sql.functions.{array, lit}

df.withColumn("myCol", when(myCol.isNull, array().cast("array")).otherwise(myCol))
df.withColumn("myCol", coalesce(myCol, array().cast("array"))).show

import org.apache.spark.sql.functions.udf

val array_ = udf(() => Array.empty[Int])

df.withColumn("myCol", when(myCol.isNull, array_()).otherwise(myCol))
df.withColumn("myCol", coalesce(myCol, array_())).show

import org.apache.spark.sql.functions.{array, lit}

df.withColumn("myCol", when(myCol.isNull, array().cast("array")).otherwise(myCol))
df.withColumn("myCol", coalesce(myCol, array().cast("array"))).show

val df = Seq("a" -> Array(1,2,3), "b" -> null, "c" -> Array(7,8,9)).toDF("id","numbers")
df.show
+---+---------+
| id|  numbers|
+---+---------+
|  a|[1, 2, 3]|
|  b|     null|
|  c|[7, 8, 9]|
+---+---------+

val df2 = df.withColumn("numbers", coalesce($"numbers", array()))
df2.show
+---+---------+
| id|  numbers|
+---+---------+
|  a|[1, 2, 3]|
|  b|       []|
|  c|[7, 8, 9]|
+---+---------+