Pyspark:获取HDFS路径上的文件/目录列表

如标题.我知道textFile,但顾名思义,它仅适用于文本文件.我需要访问HDFS(或本地路径)上的路径内的文件/目录.我正在使用pyspark

感谢帮助

使用JVM网关可能不是那么优雅,但在某些情况下,下面的代码可能会有所帮助:

URI           = sc._gateway.jvm.java.net.URI
Path          = sc._gateway.jvm.org.apache.hadoop.fs.Path
FileSystem    = sc._gateway.jvm.org.apache.hadoop.fs.FileSystem
Configuration = sc._gateway.jvm.org.apache.hadoop.conf.Configuration


fs = FileSystem.get(URI("hdfs://somehost:8020"), Configuration())

status = fs.listStatus(Path('/some_dir/yet_another_one_dir/'))

for fileStatus in status:
    print(fileStatus.getPath())

import os
import subprocess

cmd = 'hdfs dfs -ls /user/path'
files = subprocess.check_output(cmd, shell=True).strip().split('\n')
for path in files:
  print path

import os
import subprocess

cmd = 'hdfs dfs -find {} -name *.txt'.format(source_dir)
files = subprocess.check_output(cmd, shell=True).strip().split('\n')
for path in files:
  filename = path.split(os.path.sep)[-1].split('.txt')[0]
  print path, filename

from hdfs import Config
client = Config().get_client('dev')
files = client.list('the_dir_path')