☰

对于大文件，s3put 失败并显示“对等方重置连接”

tag-icon • tag-icon http amazon-s3 upload
对于大文件，s3put 失败并显示“对等方重置连接”

我可以使用相同的命令很好地上传小文件，但是每当我尝试将 5 GB 的文件上传到 S3 时，s3 脚本就会失败：

[backup]$ s3put --bucket john-murdoch-bucket --reduced --debug 2 --callback 100 data.tgz
/usr/bin/s3put:45: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
  err.message[len('No module named '):] + \
send: 'GET /?&max-keys=0 HTTP/1.1\r\nHost: john-murdoch-bucket.s3.amazonaws.com\r\nAccept-Encoding: identity\r\nDate: Tue, 11 Jun 2013 07:00:45 GMT\r\nContent-Length: 0\r\nAuthorization: AWS <snip>=\r\nUser-Agent: Boto/2.9.4 (linux2)\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: x-amz-id-2: <snip>
header: x-amz-request-id: 1E11AD8B3C93E6B5
header: Date: Tue, 11 Jun 2013 07:00:46 GMT
header: Content-Type: application/xml
header: Transfer-Encoding: chunked
header: Server: AmazonS3
Copying /backup/data.tgz to john-murdoch-bucket/backup/data.tgz
send: 'PUT /backup/data.tgz HTTP/1.1\r\nHost: john-murdoch-bucket.s3.amazonaws.com\r\nAccept-Encoding: identity\r\nContent-Length: 5522978068\r\nExpect: 100-Continue\r\nx-amz-storage-class: REDUCED_REDUNDANCY\r\nDate: Tue, 11 Jun 2013 07:02:15 GMT\r\nContent-MD5: idp0nQ+9y5kshw0AjL9zMw==\r\nContent-Type: application/x-tar\r\nAuthorization: AWS <snip>=\r\nUser-Agent: Boto/2.9.4 (linux2)\r\n    \r\n'
0 bytes transferred / 5522978068 bytes total
0 bytes transferred / 5522978068 bytes total
0 bytes transferred / 5522978068 bytes total
0 bytes transferred / 5522978068 bytes total
0 bytes transferred / 5522978068 bytes total
0 bytes transferred / 5522978068 bytes total
0 bytes transferred / 5522978068 bytes total
Traceback (most recent call last):
  File "/usr/bin/s3put", line 374, in <module>
    main()
  File "/usr/bin/s3put", line 371, in main
    headers=headers)
  File "/usr/bin/s3put", line 216, in singlepart_upload
    k.set_contents_from_filename(fullpath, *kargs, **kwargs)
  File "/usr/lib/python2.6/site-packages/boto/s3/key.py", line 1241, in set_contents_from_filename
    encrypt_key=encrypt_key)
  File "/usr/lib/python2.6/site-packages/boto/s3/key.py", line 1172, in set_contents_from_file
    chunked_transfer=chunked_transfer, size=size)
  File "/usr/lib/python2.6/site-packages/boto/s3/key.py", line 710, in send_file
    chunked_transfer=chunked_transfer, size=size)
  File "/usr/lib/python2.6/site-packages/boto/s3/key.py", line 882, in _send_file_internal
    query_args=query_args)
  File "/usr/lib/python2.6/site-packages/boto/s3/connection.py", line 544, in make_request
    override_num_retries=override_num_retries)
  File "/usr/lib/python2.6/site-packages/boto/connection.py", line 939, in make_request
    return self._mexe(http_request, sender, override_num_retries)
  File "/usr/lib/python2.6/site-packages/boto/connection.py", line 901, in _mexe
    raise e
socket.error: [Errno 104] Connection reset by peer

我是否缺少 s3put 的一些其他工具或命令行选项来执行多次重试，或分多个块上传等等？

PS 我刚刚执行了“sudo yum upgrade python-boto”，但没有任何改变。

更新-尝试了 s3cmd 1.0.0：

[ec2-user@ip-10-212-234-60 backup]$ s3cmd put  data.tgz s3://john-murdoch-bucket/data.tgz
data.tgz -> s3://john-murdoch-bucket/data.tgz  [1 of 1]
      36864 of 5522978068     0% in    1s    20.31 kB/s  failed
WARNING: Upload failed: /data.tgz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.00)
WARNING: Waiting 3 sec...
data.tgz -> s3://john-murdoch-bucket/data.tgz  [1 of 1]
     233472 of 5522978068     0% in    1s   144.89 kB/s  failed
WARNING: Upload failed: /data.tgz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.01)
WARNING: Waiting 6 sec...
data.tgz -> s3://john-murdoch-bucket/data.tgz  [1 of 1]
      45056 of 5522978068     0% in    1s    27.88 kB/s  failed
WARNING: Upload failed: /data.tgz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.05)
WARNING: Waiting 9 sec...
data.tgz -> s3://john-murdoch-bucket/data.tgz  [1 of 1]
      36864 of 5522978068     0% in    0s    45.52 kB/s  failed
WARNING: Upload failed: /data.tgz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.25)
WARNING: Waiting 12 sec...
data.tgz -> s3://john-murdoch-bucket/data.tgz  [1 of 1]
      20480 of 5522978068     0% in    1s    19.90 kB/s  failed
WARNING: Upload failed: /data.tgz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=1.25)
WARNING: Waiting 15 sec...
data.tgz -> s3://john-murdoch-bucket/data.tgz  [1 of 1]
      12288 of 5522978068     0% in    2s     4.63 kB/s  failed
ERROR: Upload of 'data.tgz' failed too many times. Skipping that file.

答案1

您需要pip install filechunkio，然后使用--multipart选项s3put。

相关内容