☰

如何让 csvkit 识别长 ASCII 行？

tag-icon • tag-icon linux ubuntu bash python ascii
如何让 csvkit 识别长 ASCII 行？

我正在使用 Ubuntu，我下载了这个 CSV 文件，它file告诉我编码如下：

ASCII text, with very long lines, with CRLF line terminators

然而，当我通过时csvcut -e ASCII datafile，我得到：

Your file is not "utf-8" encoded. Please specify the correct encoding with the -e flag. Use the -v flag to see the complete error.

当我通过时csvcut -e ASCII datafile，我得到：

Your file is not "ASCII" encoded. Please specify the correct encoding with the -e flag.

（大写或复制粘贴精确的file输出都无法改善这种情况。）

完整的错误（-v）如下所示：

Traceback (most recent call last):
  File "/usr/local/bin/csvcut", line 9, in <module>
    load_entry_point('csvkit==0.9.2', 'console_scripts', 'csvcut')()
  File "/usr/local/lib/python2.7/dist-packages/csvkit-0.9.2-py2.7.egg/csvkit/utilities/csvcut.py", line 64, in launch_new_instance
    utility.main()
  File "/usr/local/lib/python2.7/dist-packages/csvkit-0.9.2-py2.7.egg/csvkit/utilities/csvcut.py", line 53, in main
    for row in rows:
  File "/usr/local/lib/python2.7/dist-packages/csvkit-0.9.2-py2.7.egg/csvkit/unicsv.py", line 51, in next
    row = next(self.reader)
  File "/usr/local/lib/python2.7/dist-packages/six.py", line 535, in next
    return type(self).__next__(self)
  File "/usr/local/lib/python2.7/dist-packages/csvkit-0.9.2-py2.7.egg/csvkit/unicsv.py", line 35, in __next__
    return next(self.reader).encode('utf-8')
  File "/usr/lib/python2.7/codecs.py", line 615, in next
    line = self.readline()
  File "/usr/lib/python2.7/codecs.py", line 530, in readline
    data = self.read(readsize, firstline=True)
  File "/usr/lib/python2.7/codecs.py", line 477, in read
    newchars, decodedbytes = self.decode(data, self.errors)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 0: ordinal not in range(128)

答案1

您的有效载荷既不是 ASCII 也不是 UTF-8 编码。您可以快速找到非 ASCII 位：

awk '/[^\x00-\x7F]/{ print NR ":", $0 }' data.csv | less

您将在 UTF-8 编码的终端仿真器中看到类似的东西Briarcliffe College�??Patchogue，表明这不是 UTF-8 编码的文件。编码的第一个猜测是什么？ISO 8859-1，西欧。让我们测试一下：

# piping to /dev/null to suppress printing and speed up processing (printing to tty is slow)
csvcut -e iso-8859-1 data.csv >/dev/null

这次没有错误，瞧！

相关内容