是否可以在Pyspark中继承DataFrame？

对于Pyspark文档显示被构造DataFrames sqlContext,sqlContext.read()和各种的其他方法.

(参见https://spark.apache.org/docs/1.6.2/api/python/pyspark.sql.html)

是否有可能将Dataframe子类化并独立实例化它？我想为基本DataFrame类添加方法和功能.

这真的取决于你的目标.

从技术上讲,这是可能的.pyspark.sql.DataFrame只是一个简单的Python类.如果需要,您可以扩展它或猴子补丁.

from pyspark.sql import DataFrame

class DataFrameWithZipWithIndex(DataFrame):
     def __init__(self, df):
         super(self.__class__, self).__init__(df._jdf, df.sql_ctx)

     def zipWithIndex(self):
         return (self.rdd
             .zipWithIndex()
             .map(lambda row: (row[1], ) + row[0])
             .toDF(["_idx"] + self.columns))

用法示例:

df = sc.parallelize([("a", 1)]).toDF(["foo", "bar"])

with_zipwithindex = DataFrameWithZipWithIndex(df)

isinstance(with_zipwithindex, DataFrame)

True

with_zipwithindex.zipWithIndex().show()

+----+---+---+
|_idx|foo|bar|
+----+---+---+
|   0|  a|  1|
+----+---+---+