我在Scala和Python中编写了以下代码,但是返回的DataFrame似乎并未应用我正在应用的架构中的非可空字段.italianVotes.csv是一个csv文件,带有'〜'作为分隔符和四个字段.我正在使用Spark 2.1.0.
2657~135~2~2013-11-22 00:00:00.0
2658~142~2~2013-11-22 00:00:00.0
2659~142~1~2013-11-22 00:00:00.0
2660~140~2~2013-11-22 00:00:00.0
2661~140~1~2013-11-22 00:00:00.0
2662~1354~2~2013-11-22 00:00:00.0
2663~1356~2~2013-11-22 00:00:00.0
2664~1353~2~2013-11-22 00:00:00.0
2665~1351~2~2013-11-22 00:00:00.0
2667~1357~2~2013-11-22 00:00:00.0
import org.apache.spark.sql.types._
val schema = StructType(
StructField("id", IntegerType, false) ::
StructField("postId", IntegerType, false) ::
StructField("voteType", IntegerType, true) ::
StructField("time", TimestampType, true) :: Nil)
val fileName = "italianVotes.csv"
val italianDF = spark.read.schema(schema).option("sep", "~").csv(fileName)
italianDF.printSchema()
// output
root
|-- id: integer (nullable = true)
|-- postId: integer (nullable = true)
|-- voteType: integer (nullable = true)
|-- time: timestamp (nullable = true)
from pyspark.sql.types import *
schema = StructType([
StructField("id", IntegerType(), False),
StructField("postId", IntegerType(), False),
StructField("voteType", IntegerType(), True),
StructField("time", TimestampType(), True),
])
file_name = "italianVotes.csv"
italian_df = spark.read.csv(file_name, schema = schema, sep = "~")
# print schema
italian_df.printSchema()
root
|-- id: integer (nullable = true)
|-- postId: integer (nullable = true)
|-- voteType: integer (nullable = true)
|-- time: timestamp (nullable = true)
我的主要问题是,当我在模式中将它们设置为不可为空时,为什么前两个字段可以为空？
通常,Spark Datasets要么nullable从其父项继承属性,要么根据外部数据类型进行推断.
你可以争论它是否是一种好的方法,但最终它是明智的.如果数据源的语义不支持可空性约束,那么模式的应用也不能.在一天结束null时,如果相反的假设结果不正确,那么假设事情可能比运行时失败更好.
京公网安备 11010802040832号 | 京ICP备19059560号-6 