在python dask中使用分隔符读取csv

我正在尝试DataFrame通过读取由'#####'分隔的5个csv文件来创建一个哈希

代码是：

import dask.dataframe as dd
df = dd.read_csv('D:\temp.csv',sep='#####',engine='python')
res = df.compute()

错误是：

dask.async.ValueError:
Dask dataframe inspected the first 1,000 rows of your csv file to guess the
data types of your columns.  These first 1,000 rows led us to an incorrect
guess.

For example a column may have had integers in the first 1000
rows followed by a float or missing value in the 1,001-st row.

You will need to specify some dtype information explicitly using the
``dtype=`` keyword argument for the right column names and dtypes.

    df = dd.read_csv(..., dtype={'my-column': float})

Pandas has given us the following error when trying to parse the file:

  "The 'dtype' option is not supported with the 'python' engine"

Traceback
 ---------
File "/home/ec2-user/anaconda3/lib/python3.4/site-packages/dask/async.py", line 263, in execute_task
result = _execute_task(task, data)
File "/home/ec2-user/anaconda3/lib/python3.4/site-packages/dask/async.py", line 245, in _execute_task
return func(*args2)
File "/home/ec2-user/anaconda3/lib/python3.4/site-packages/dask/dataframe/io.py", line 69, in _read_csv
raise ValueError(msg)

那么如何摆脱它。

如果我遵循该错误，那么我将不得不为每列提供dtype，但是如果我有100多个列，那将毫无用处。

如果我没有分隔符地阅读，那么一切都会很好，但是到处都有#####。因此，将其计算为大熊猫后DataFrame，有没有办法摆脱它？

所以在这方面帮助我。

以as读取整个文件dtype=object，这意味着所有列都将被解释为type object。应该正确读入，摆脱#####每一行中的。从那里您可以使用compute()方法将其变成熊猫框架。数据放入pandas框架后，您可以使用pandas infer_objects方法来更新类型，而无需进行hard操作。

import dask.dataframe as dd
df = dd.read_csv('D:\temp.csv',sep='#####',dtype='object').compute()
res = df.infer_objects()