Handle MD5 from etag when upper-case are used. by leffss · Pull Request #1286 · s3tools/s3cmd

leffss · 2022-11-01T11:09:18Z

Some S3-compatible providers, like AlibabaCloud, return etag in upper-case (e.g. A256A58D6429BF79ABBC0DB3CEF2FC71)

fviard · 2022-11-05T17:30:59Z

Thank you for your PR.
It is not the first time that we get a suggestion like this, but I'm not sure that it is a good solution, because this would impact all the other s3 compatible services even if they don't need that.

To solve this problem, what I'm thinking about is to allow some "tweaking" per service.
To help me with that, would it be possible possible for you to run some commands on your Alibabacloud with the "-d" (ie debug) flag on, and to show me an example of response from the server?
I'm especially interested in the "server" string in the header of the response.
Thanks

leffss · 2022-11-06T04:24:17Z

Thank you for your PR. It is not the first time that we get a suggestion like this, but I'm not sure that it is a good solution, because this would impact all the other s3 compatible services even if they don't need that.

To solve this problem, what I'm thinking about is to allow some "tweaking" per service. To help me with that, would it be possible possible for you to run some commands on your Alibabacloud with the "-d" (ie debug) flag on, and to show me an example of response from the server? I'm especially interested in the "server" string in the header of the response. Thanks

.s3cfg setting:

[default]
access_key = access_key
secret_key = secret_key
bucket_location = oss-cn-hongkong
host_base = oss-cn-hongkong.aliyuncs.com
host_bucket = %(bucket)s.oss-cn-hongkong.aliyuncs.com
default_mime_type = binary/octet-stream
encoding = UTF-8
enable_multipart = True
multipart_max_chunks = 10000
multipart_chunk_size_mb = 128
recursive = True
recv_chunk = 65536
send_chunk = 65536
server_side_encryption = False
signature_v2 = False
socket_timeout = 300
use_mime_magic = False
verbosity = DEBUG
#verbosity = WARNING

debug log:

$ s3cmd --no-check-md5 put requirements.txt s3://leffss-gitlab-backup-upload/test/requirements.txt
DEBUG: Updating Config.Config cache_file -> 
DEBUG: Updating Config.Config follow_symlinks -> False
DEBUG: Command: put
INFO: Cache file not found or empty, creating/populating it.
DEBUG: DeUnicodising 'requirements.txt' using UTF-8
INFO: Compiling list of local files...
DEBUG: DeUnicodising 'requirements.txt' using UTF-8
DEBUG: Unicodising b'requirements.txt' using UTF-8
DEBUG: DeUnicodising 'requirements.txt' using UTF-8
DEBUG: DeUnicodising 'requirements.txt' using UTF-8
DEBUG: Unicodising b'' using UTF-8
DEBUG: DeUnicodising 'requirements.txt' using UTF-8
DEBUG: Unicodising b'requirements.txt' using UTF-8
DEBUG: DeUnicodising 'requirements.txt' using UTF-8
DEBUG: DeUnicodising 'requirements.txt' using UTF-8
DEBUG: Applying --exclude/--include
DEBUG: CHECK: 'requirements.txt'
DEBUG: PASS: 'requirements.txt'
INFO: Running stat() and reading/calculating MD5 values on 1 files, this may take some time...
DEBUG: DeUnicodising 'requirements.txt' using UTF-8
INFO: Summary: 1 local files to upload
DEBUG: String 'b'root'' encoded to 'root'
DEBUG: String 'b'root'' encoded to 'root'
DEBUG: attr_header: {'x-amz-meta-s3cmd-attrs': 'atime:1667636059/ctime:1667299379/gid:0/gname:root/mode:33188/mtime:1667299379/uid:0/uname:root'}
DEBUG: DeUnicodising 'requirements.txt' using UTF-8
DEBUG: DeUnicodising 'requirements.txt' using UTF-8
DEBUG: DeUnicodising 'requirements.txt' using UTF-8
DEBUG: CreateRequest: resource[uri]=/test/requirements.txt
upload: 'requirements.txt' -> 's3://leffss-gitlab-backup-upload/test/requirements.txt'  [1 of 1]
DEBUG: DeUnicodising 'requirements.txt' using UTF-8
DEBUG: Using signature v4
DEBUG: get_hostname(leffss-gitlab-backup-upload): leffss-gitlab-backup-upload.oss-cn-hongkong.aliyuncs.com
DEBUG: canonical_headers = content-length:13
content-type:text/plain
host:leffss-gitlab-backup-upload.oss-cn-hongkong.aliyuncs.com
x-amz-content-sha256:91bd5e257c4a6ff6d6df14ac01ba7b3d14d20bdb8feb563d131b4ac4f9815465
x-amz-date:20221106T042108Z
x-amz-meta-s3cmd-attrs:atime:1667636059/ctime:1667299379/gid:0/gname:root/mode:33188/mtime:1667299379/uid:0/uname:root
x-amz-storage-class:STANDARD

DEBUG: Canonical Request:
PUT
/test/requirements.txt

content-length:13
content-type:text/plain
host:leffss-gitlab-backup-upload.oss-cn-hongkong.aliyuncs.com
x-amz-content-sha256:91bd5e257c4a6ff6d6df14ac01ba7b3d14d20bdb8feb563d131b4ac4f9815465
x-amz-date:20221106T042108Z
x-amz-meta-s3cmd-attrs:atime:1667636059/ctime:1667299379/gid:0/gname:root/mode:33188/mtime:1667299379/uid:0/uname:root
x-amz-storage-class:STANDARD

content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-meta-s3cmd-attrs;x-amz-storage-class
91bd5e257c4a6ff6d6df14ac01ba7b3d14d20bdb8feb563d131b4ac4f9815465
----------------------
DEBUG: signature-v4 headers: {'content-length': '13', 'content-type': 'text/plain', 'x-amz-meta-s3cmd-attrs': 'atime:1667636059/ctime:1667299379/gid:0/gname:root/mode:33188/mtime:1667299379/uid:0/uname:root', 'x-amz-storage-class': 'STANDARD', 'x-amz-date': '20221106T042108Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=LTAI5tSnhs1a7aCvGwrxbfGR/20221106/oss-cn-hongkong/s3/aws4_request,SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-meta-s3cmd-attrs;x-amz-storage-class,Signature=8c254701dbd48d21548876ca8b5f05407f3ffb9b8dfef1dda31a1c005c49b06e', 'x-amz-content-sha256': '91bd5e257c4a6ff6d6df14ac01ba7b3d14d20bdb8feb563d131b4ac4f9815465'}
DEBUG: get_hostname(leffss-gitlab-backup-upload): leffss-gitlab-backup-upload.oss-cn-hongkong.aliyuncs.com
DEBUG: ConnMan.get(): creating new connection: https://leffss-gitlab-backup-upload.oss-cn-hongkong.aliyuncs.com
DEBUG: Using ca_certs_file None
DEBUG: Using ssl_client_cert_file None
DEBUG: Using ssl_client_key_file None
DEBUG: httplib.HTTPSConnection() has both context and check_hostname
DEBUG: non-proxied HTTPSConnection(leffss-gitlab-backup-upload.oss-cn-hongkong.aliyuncs.com, None)
DEBUG: format_uri(): /test/requirements.txt
 13 of 13   100% in    0s    50.96 B/sDEBUG: ConnMan.put(): connection put back to pool (https://leffss-gitlab-backup-upload.oss-cn-hongkong.aliyuncs.com#1)
DEBUG: Response:
{'data': b'',
 'headers': {'connection': 'keep-alive',
             'content-length': '0',
             'date': 'Sun, 06 Nov 2022 04:21:08 GMT',
             'etag': '"54C15BBE5108CB3CCDEF41274654127F"',
             'server': 'AliyunOSS',
             'x-amz-request-id': '636736341F85633435B2C303',
             'x-oss-hash-crc64ecma': '11264086281538792249',
             'x-oss-server-time': '50'},
 'reason': 'OK',
 'size': 13,
 'status': 200}
 13 of 13   100% in    0s    32.03 B/s  done
DEBUG: MD5 sums: computed=54c15bbe5108cb3ccdef41274654127f, received=54C15BBE5108CB3CCDEF41274654127F
WARNING: MD5 Sums don't match!
WARNING: Retrying upload of requirements.txt
upload: 'requirements.txt' -> 's3://leffss-gitlab-backup-upload/test/requirements.txt'  [1 of 1]
DEBUG: DeUnicodising 'requirements.txt' using UTF-8
DEBUG: Using signature v4
DEBUG: get_hostname(leffss-gitlab-backup-upload): leffss-gitlab-backup-upload.oss-cn-hongkong.aliyuncs.com
DEBUG: canonical_headers = content-length:13
content-type:text/plain
host:leffss-gitlab-backup-upload.oss-cn-hongkong.aliyuncs.com
x-amz-content-sha256:91bd5e257c4a6ff6d6df14ac01ba7b3d14d20bdb8feb563d131b4ac4f9815465
x-amz-date:20221106T042108Z
x-amz-meta-s3cmd-attrs:atime:1667636059/ctime:1667299379/gid:0/gname:root/mode:33188/mtime:1667299379/uid:0/uname:root
x-amz-storage-class:STANDARD

DEBUG: Canonical Request:
PUT
/test/requirements.txt

content-length:13
content-type:text/plain
host:leffss-gitlab-backup-upload.oss-cn-hongkong.aliyuncs.com
x-amz-content-sha256:91bd5e257c4a6ff6d6df14ac01ba7b3d14d20bdb8feb563d131b4ac4f9815465
x-amz-date:20221106T042108Z
x-amz-meta-s3cmd-attrs:atime:1667636059/ctime:1667299379/gid:0/gname:root/mode:33188/mtime:1667299379/uid:0/uname:root
x-amz-storage-class:STANDARD

content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-meta-s3cmd-attrs;x-amz-storage-class
91bd5e257c4a6ff6d6df14ac01ba7b3d14d20bdb8feb563d131b4ac4f9815465
----------------------
DEBUG: signature-v4 headers: {'content-length': '13', 'content-type': 'text/plain', 'x-amz-meta-s3cmd-attrs': 'atime:1667636059/ctime:1667299379/gid:0/gname:root/mode:33188/mtime:1667299379/uid:0/uname:root', 'x-amz-storage-class': 'STANDARD', 'x-amz-date': '20221106T042108Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=LTAI5tSnhs1a7aCvGwrxbfGR/20221106/oss-cn-hongkong/s3/aws4_request,SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-meta-s3cmd-attrs;x-amz-storage-class,Signature=8c254701dbd48d21548876ca8b5f05407f3ffb9b8dfef1dda31a1c005c49b06e', 'x-amz-content-sha256': '91bd5e257c4a6ff6d6df14ac01ba7b3d14d20bdb8feb563d131b4ac4f9815465'}
DEBUG: get_hostname(leffss-gitlab-backup-upload): leffss-gitlab-backup-upload.oss-cn-hongkong.aliyuncs.com
DEBUG: ConnMan.get(): re-using connection: https://leffss-gitlab-backup-upload.oss-cn-hongkong.aliyuncs.com#1
DEBUG: format_uri(): /test/requirements.txt
 13 of 13   100% in    0s    14.51 KB/sDEBUG: ConnMan.put(): connection put back to pool (https://leffss-gitlab-backup-upload.oss-cn-hongkong.aliyuncs.com#2)
DEBUG: Response:
{'data': b'',
 'headers': {'connection': 'keep-alive',
             'content-length': '0',
             'date': 'Sun, 06 Nov 2022 04:21:08 GMT',
             'etag': '"54C15BBE5108CB3CCDEF41274654127F"',
             'server': 'AliyunOSS',
             'x-amz-request-id': '636736341F8563343580C403',
             'x-oss-hash-crc64ecma': '11264086281538792249',
             'x-oss-server-time': '4'},
 'reason': 'OK',
 'size': 13,
 'status': 200}
 13 of 13   100% in    0s    89.13 B/s  done
DEBUG: MD5 sums: computed=54c15bbe5108cb3ccdef41274654127f, received=54C15BBE5108CB3CCDEF41274654127F
WARNING: MD5 Sums don't match!
WARNING: Retrying upload of requirements.txt
upload: 'requirements.txt' -> 's3://leffss-gitlab-backup-upload/test/requirements.txt'  [1 of 1]
DEBUG: DeUnicodising 'requirements.txt' using UTF-8
DEBUG: Using signature v4
DEBUG: get_hostname(leffss-gitlab-backup-upload): leffss-gitlab-backup-upload.oss-cn-hongkong.aliyuncs.com
DEBUG: canonical_headers = content-length:13
content-type:text/plain
host:leffss-gitlab-backup-upload.oss-cn-hongkong.aliyuncs.com
x-amz-content-sha256:91bd5e257c4a6ff6d6df14ac01ba7b3d14d20bdb8feb563d131b4ac4f9815465
x-amz-date:20221106T042109Z
x-amz-meta-s3cmd-attrs:atime:1667636059/ctime:1667299379/gid:0/gname:root/mode:33188/mtime:1667299379/uid:0/uname:root
x-amz-storage-class:STANDARD

DEBUG: Canonical Request:
PUT
/test/requirements.txt

content-length:13
content-type:text/plain
host:leffss-gitlab-backup-upload.oss-cn-hongkong.aliyuncs.com
x-amz-content-sha256:91bd5e257c4a6ff6d6df14ac01ba7b3d14d20bdb8feb563d131b4ac4f9815465
x-amz-date:20221106T042109Z
x-amz-meta-s3cmd-attrs:atime:1667636059/ctime:1667299379/gid:0/gname:root/mode:33188/mtime:1667299379/uid:0/uname:root
x-amz-storage-class:STANDARD

content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-meta-s3cmd-attrs;x-amz-storage-class
91bd5e257c4a6ff6d6df14ac01ba7b3d14d20bdb8feb563d131b4ac4f9815465
----------------------
DEBUG: signature-v4 headers: {'content-length': '13', 'content-type': 'text/plain', 'x-amz-meta-s3cmd-attrs': 'atime:1667636059/ctime:1667299379/gid:0/gname:root/mode:33188/mtime:1667299379/uid:0/uname:root', 'x-amz-storage-class': 'STANDARD', 'x-amz-date': '20221106T042109Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=LTAI5tSnhs1a7aCvGwrxbfGR/20221106/oss-cn-hongkong/s3/aws4_request,SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-meta-s3cmd-attrs;x-amz-storage-class,Signature=465ed95b1adc1191cdc19fccbc93817886ed9cd779964347ae90d9b2800b6c42', 'x-amz-content-sha256': '91bd5e257c4a6ff6d6df14ac01ba7b3d14d20bdb8feb563d131b4ac4f9815465'}
DEBUG: get_hostname(leffss-gitlab-backup-upload): leffss-gitlab-backup-upload.oss-cn-hongkong.aliyuncs.com
DEBUG: ConnMan.get(): re-using connection: https://leffss-gitlab-backup-upload.oss-cn-hongkong.aliyuncs.com#2
DEBUG: format_uri(): /test/requirements.txt
 13 of 13   100% in    0s    10.30 KB/sDEBUG: ConnMan.put(): connection put back to pool (https://leffss-gitlab-backup-upload.oss-cn-hongkong.aliyuncs.com#3)
DEBUG: Response:
{'data': b'',
 'headers': {'connection': 'keep-alive',
             'content-length': '0',
             'date': 'Sun, 06 Nov 2022 04:21:08 GMT',
             'etag': '"54C15BBE5108CB3CCDEF41274654127F"',
             'server': 'AliyunOSS',
             'x-amz-request-id': '636736341F8563343536C503',
             'x-oss-hash-crc64ecma': '11264086281538792249',
             'x-oss-server-time': '6'},
 'reason': 'OK',
 'size': 13,
 'status': 200}
 13 of 13   100% in    0s    89.14 B/s  done
DEBUG: MD5 sums: computed=54c15bbe5108cb3ccdef41274654127f, received=54C15BBE5108CB3CCDEF41274654127F
WARNING: MD5 Sums don't match!
WARNING: Retrying upload of requirements.txt
upload: 'requirements.txt' -> 's3://leffss-gitlab-backup-upload/test/requirements.txt'  [1 of 1]
DEBUG: DeUnicodising 'requirements.txt' using UTF-8
DEBUG: Using signature v4
DEBUG: get_hostname(leffss-gitlab-backup-upload): leffss-gitlab-backup-upload.oss-cn-hongkong.aliyuncs.com
DEBUG: canonical_headers = content-length:13
content-type:text/plain
host:leffss-gitlab-backup-upload.oss-cn-hongkong.aliyuncs.com
x-amz-content-sha256:91bd5e257c4a6ff6d6df14ac01ba7b3d14d20bdb8feb563d131b4ac4f9815465
x-amz-date:20221106T042109Z
x-amz-meta-s3cmd-attrs:atime:1667636059/ctime:1667299379/gid:0/gname:root/mode:33188/mtime:1667299379/uid:0/uname:root
x-amz-storage-class:STANDARD

DEBUG: Canonical Request:
PUT
/test/requirements.txt

content-length:13
content-type:text/plain
host:leffss-gitlab-backup-upload.oss-cn-hongkong.aliyuncs.com
x-amz-content-sha256:91bd5e257c4a6ff6d6df14ac01ba7b3d14d20bdb8feb563d131b4ac4f9815465
x-amz-date:20221106T042109Z
x-amz-meta-s3cmd-attrs:atime:1667636059/ctime:1667299379/gid:0/gname:root/mode:33188/mtime:1667299379/uid:0/uname:root
x-amz-storage-class:STANDARD

content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-meta-s3cmd-attrs;x-amz-storage-class
91bd5e257c4a6ff6d6df14ac01ba7b3d14d20bdb8feb563d131b4ac4f9815465
----------------------
DEBUG: signature-v4 headers: {'content-length': '13', 'content-type': 'text/plain', 'x-amz-meta-s3cmd-attrs': 'atime:1667636059/ctime:1667299379/gid:0/gname:root/mode:33188/mtime:1667299379/uid:0/uname:root', 'x-amz-storage-class': 'STANDARD', 'x-amz-date': '20221106T042109Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=LTAI5tSnhs1a7aCvGwrxbfGR/20221106/oss-cn-hongkong/s3/aws4_request,SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-meta-s3cmd-attrs;x-amz-storage-class,Signature=465ed95b1adc1191cdc19fccbc93817886ed9cd779964347ae90d9b2800b6c42', 'x-amz-content-sha256': '91bd5e257c4a6ff6d6df14ac01ba7b3d14d20bdb8feb563d131b4ac4f9815465'}
DEBUG: get_hostname(leffss-gitlab-backup-upload): leffss-gitlab-backup-upload.oss-cn-hongkong.aliyuncs.com
DEBUG: ConnMan.get(): re-using connection: https://leffss-gitlab-backup-upload.oss-cn-hongkong.aliyuncs.com#3
DEBUG: format_uri(): /test/requirements.txt
 13 of 13   100% in    0s    15.10 KB/sDEBUG: ConnMan.put(): connection put back to pool (https://leffss-gitlab-backup-upload.oss-cn-hongkong.aliyuncs.com#4)
DEBUG: Response:
{'data': b'',
 'headers': {'connection': 'keep-alive',
             'content-length': '0',
             'date': 'Sun, 06 Nov 2022 04:21:09 GMT',
             'etag': '"54C15BBE5108CB3CCDEF41274654127F"',
             'server': 'AliyunOSS',
             'x-amz-request-id': '636736341F85633435E3C503',
             'x-oss-hash-crc64ecma': '11264086281538792249',
             'x-oss-server-time': '4'},
 'reason': 'OK',
 'size': 13,
 'status': 200}
 13 of 13   100% in    0s    90.66 B/s  done
DEBUG: MD5 sums: computed=54c15bbe5108cb3ccdef41274654127f, received=54C15BBE5108CB3CCDEF41274654127F
WARNING: MD5 Sums don't match!
WARNING: Retrying upload of requirements.txt
upload: 'requirements.txt' -> 's3://leffss-gitlab-backup-upload/test/requirements.txt'  [1 of 1]
DEBUG: DeUnicodising 'requirements.txt' using UTF-8
DEBUG: Using signature v4
DEBUG: get_hostname(leffss-gitlab-backup-upload): leffss-gitlab-backup-upload.oss-cn-hongkong.aliyuncs.com
DEBUG: canonical_headers = content-length:13
content-type:text/plain
host:leffss-gitlab-backup-upload.oss-cn-hongkong.aliyuncs.com
x-amz-content-sha256:91bd5e257c4a6ff6d6df14ac01ba7b3d14d20bdb8feb563d131b4ac4f9815465
x-amz-date:20221106T042109Z
x-amz-meta-s3cmd-attrs:atime:1667636059/ctime:1667299379/gid:0/gname:root/mode:33188/mtime:1667299379/uid:0/uname:root
x-amz-storage-class:STANDARD

DEBUG: Canonical Request:
PUT
/test/requirements.txt

content-length:13
content-type:text/plain
host:leffss-gitlab-backup-upload.oss-cn-hongkong.aliyuncs.com
x-amz-content-sha256:91bd5e257c4a6ff6d6df14ac01ba7b3d14d20bdb8feb563d131b4ac4f9815465
x-amz-date:20221106T042109Z
x-amz-meta-s3cmd-attrs:atime:1667636059/ctime:1667299379/gid:0/gname:root/mode:33188/mtime:1667299379/uid:0/uname:root
x-amz-storage-class:STANDARD

content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-meta-s3cmd-attrs;x-amz-storage-class
91bd5e257c4a6ff6d6df14ac01ba7b3d14d20bdb8feb563d131b4ac4f9815465
----------------------
DEBUG: signature-v4 headers: {'content-length': '13', 'content-type': 'text/plain', 'x-amz-meta-s3cmd-attrs': 'atime:1667636059/ctime:1667299379/gid:0/gname:root/mode:33188/mtime:1667299379/uid:0/uname:root', 'x-amz-storage-class': 'STANDARD', 'x-amz-date': '20221106T042109Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=LTAI5tSnhs1a7aCvGwrxbfGR/20221106/oss-cn-hongkong/s3/aws4_request,SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-meta-s3cmd-attrs;x-amz-storage-class,Signature=465ed95b1adc1191cdc19fccbc93817886ed9cd779964347ae90d9b2800b6c42', 'x-amz-content-sha256': '91bd5e257c4a6ff6d6df14ac01ba7b3d14d20bdb8feb563d131b4ac4f9815465'}
DEBUG: get_hostname(leffss-gitlab-backup-upload): leffss-gitlab-backup-upload.oss-cn-hongkong.aliyuncs.com
DEBUG: ConnMan.get(): re-using connection: https://leffss-gitlab-backup-upload.oss-cn-hongkong.aliyuncs.com#4
DEBUG: format_uri(): /test/requirements.txt
 13 of 13   100% in    0s    13.34 KB/sDEBUG: ConnMan.put(): connection put back to pool (https://leffss-gitlab-backup-upload.oss-cn-hongkong.aliyuncs.com#5)
DEBUG: Response:
{'data': b'',
 'headers': {'connection': 'keep-alive',
             'content-length': '0',
             'date': 'Sun, 06 Nov 2022 04:21:09 GMT',
             'etag': '"54C15BBE5108CB3CCDEF41274654127F"',
             'server': 'AliyunOSS',
             'x-amz-request-id': '636736351F85633435ADC603',
             'x-oss-hash-crc64ecma': '11264086281538792249',
             'x-oss-server-time': '4'},
 'reason': 'OK',
 'size': 13,
 'status': 200}
 13 of 13   100% in    0s    90.39 B/s  done
DEBUG: MD5 sums: computed=54c15bbe5108cb3ccdef41274654127f, received=54C15BBE5108CB3CCDEF41274654127F
WARNING: MD5 Sums don't match!
WARNING: Retrying upload of requirements.txt
upload: 'requirements.txt' -> 's3://leffss-gitlab-backup-upload/test/requirements.txt'  [1 of 1]
DEBUG: DeUnicodising 'requirements.txt' using UTF-8
DEBUG: Using signature v4
DEBUG: get_hostname(leffss-gitlab-backup-upload): leffss-gitlab-backup-upload.oss-cn-hongkong.aliyuncs.com
DEBUG: canonical_headers = content-length:13
content-type:text/plain
host:leffss-gitlab-backup-upload.oss-cn-hongkong.aliyuncs.com
x-amz-content-sha256:91bd5e257c4a6ff6d6df14ac01ba7b3d14d20bdb8feb563d131b4ac4f9815465
x-amz-date:20221106T042109Z
x-amz-meta-s3cmd-attrs:atime:1667636059/ctime:1667299379/gid:0/gname:root/mode:33188/mtime:1667299379/uid:0/uname:root
x-amz-storage-class:STANDARD

DEBUG: Canonical Request:
PUT
/test/requirements.txt

content-length:13
content-type:text/plain
host:leffss-gitlab-backup-upload.oss-cn-hongkong.aliyuncs.com
x-amz-content-sha256:91bd5e257c4a6ff6d6df14ac01ba7b3d14d20bdb8feb563d131b4ac4f9815465
x-amz-date:20221106T042109Z
x-amz-meta-s3cmd-attrs:atime:1667636059/ctime:1667299379/gid:0/gname:root/mode:33188/mtime:1667299379/uid:0/uname:root
x-amz-storage-class:STANDARD

content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-meta-s3cmd-attrs;x-amz-storage-class
91bd5e257c4a6ff6d6df14ac01ba7b3d14d20bdb8feb563d131b4ac4f9815465
----------------------
DEBUG: signature-v4 headers: {'content-length': '13', 'content-type': 'text/plain', 'x-amz-meta-s3cmd-attrs': 'atime:1667636059/ctime:1667299379/gid:0/gname:root/mode:33188/mtime:1667299379/uid:0/uname:root', 'x-amz-storage-class': 'STANDARD', 'x-amz-date': '20221106T042109Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=LTAI5tSnhs1a7aCvGwrxbfGR/20221106/oss-cn-hongkong/s3/aws4_request,SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-meta-s3cmd-attrs;x-amz-storage-class,Signature=465ed95b1adc1191cdc19fccbc93817886ed9cd779964347ae90d9b2800b6c42', 'x-amz-content-sha256': '91bd5e257c4a6ff6d6df14ac01ba7b3d14d20bdb8feb563d131b4ac4f9815465'}
DEBUG: get_hostname(leffss-gitlab-backup-upload): leffss-gitlab-backup-upload.oss-cn-hongkong.aliyuncs.com
DEBUG: ConnMan.get(): re-using connection: https://leffss-gitlab-backup-upload.oss-cn-hongkong.aliyuncs.com#5
DEBUG: format_uri(): /test/requirements.txt
 13 of 13   100% in    0s    11.94 KB/sDEBUG: ConnMan.put(): connection put back to pool (https://leffss-gitlab-backup-upload.oss-cn-hongkong.aliyuncs.com#6)
DEBUG: Response:
{'data': b'',
 'headers': {'connection': 'keep-alive',
             'content-length': '0',
             'date': 'Sun, 06 Nov 2022 04:21:09 GMT',
             'etag': '"54C15BBE5108CB3CCDEF41274654127F"',
             'server': 'AliyunOSS',
             'x-amz-request-id': '636736351F856334357FC703',
             'x-oss-hash-crc64ecma': '11264086281538792249',
             'x-oss-server-time': '4'},
 'reason': 'OK',
 'size': 13,
 'status': 200}
 13 of 13   100% in    0s    90.43 B/s  done
DEBUG: MD5 sums: computed=54c15bbe5108cb3ccdef41274654127f, received=54C15BBE5108CB3CCDEF41274654127F
WARNING: MD5 Sums don't match!
WARNING: Too many failures. Giving up on 'requirements.txt'
ERROR: Upload of 'requirements.txt' failed too many times (Last reason: Too many failures. Giving up on 'requirements.txt')

Handle MD5 from etag when upper-case are used.

d38caa2

Some S3-compatible providers, like AlibabaCloud, return etag in upper-case (e.g. A256A58D6429BF79ABBC0DB3CEF2FC71)

fviard added the investigation-needed label Nov 5, 2022

fviard force-pushed the master branch from eab7f17 to 10aea09 Compare September 11, 2024 00:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Handle MD5 from etag when upper-case are used.#1286

Handle MD5 from etag when upper-case are used.#1286
leffss wants to merge 1 commit intos3tools:masterfrom
leffss:alioss

leffss commented Nov 1, 2022

Uh oh!

fviard commented Nov 5, 2022

Uh oh!

leffss commented Nov 6, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

leffss commented Nov 1, 2022

Uh oh!

fviard commented Nov 5, 2022

Uh oh!

leffss commented Nov 6, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants