Failed to upload files to Azure blob using s3cmd through s3proxy

[HotDog215]

When I used s3cmd to upload files to the blob through s3proxy, the upload failed, displaying: HttpResponseException without HttpResponse:Org.jclouds.http.HttpResponseException: Server rejected operation connecting to PUT https://sascenariodb.blob.core.chinacloudapi.cn/scenariodb/ads_list.sql HTTP/1.1, but it's easy for me to list files using s3cmd

Error message：

[s3proxy] D 07-24 12:12:23.369 S3Proxy-Jetty-18 o.j.h.i.JavaUrlHttpCommandExecutorService:56 |::] Caught a protocol exception on a 100-continue PUT request. Attempting to retry.
[s3proxy] W 07-24 12:12:23.370 S3Proxy-Jetty-18 o.j.a.s.h.AzureStorageClientErrorRetryHandler:74 |::] Cannot retry after server error, command is not replayable: [method=org.jclouds.azureblob.AzureBlobClient.public abstract java.lang.String org.jclouds.azureblob.AzureBlobClient.putBlob(java.lang.String,org.jclouds.azureblob.domain.AzureBlob)[scenariodb, [properties=[name=ads_list.sql, container=null, url=null, contentMetadata=[cacheControl=null, contentDisposition=null, contentEncoding=null, contentLanguage=null, contentLength=13, contentMD5=null, contentType=text/plain, expires=null], eTag=null, lastModified=null, leaseStatus=Unlocked, metadata={s3cmd-attrs=atime:1690198605/ctime:1690198596/gid:0/gname:root/md5:01bcb1fe182a23a65c5efe8326250da8/mode:33188/mtime:1690198596/uid:0/uname:root}, type=BlockBlob]]], request=PUT https://sascenariodb.blob.core.chinacloudapi.cn/scenariodb/ads_list.sql HTTP/1.1]
[s3proxy] D 07-24 12:12:23.372 S3Proxy-Jetty-18 o.g.s.S3ProxyHandlerJetty:88 |::] HttpResponseException without HttpResponse:
org.jclouds.http.HttpResponseException: Server rejected operation connecting to PUT https://sascenariodb.blob.core.chinacloudapi.cn/scenariodb/ads_list.sql HTTP/1.1
	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:120)
	at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
	at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
	at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
	at com.sun.proxy.$Proxy58.putBlob(Unknown Source)
	at org.jclouds.azureblob.blobstore.AzureBlobStore.putBlob(AzureBlobStore.java:240)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:50)
	at com.sun.proxy.$Proxy59.putBlob(Unknown Source)
	at org.gaul.s3proxy.S3ProxyHandler.handlePutBlob(S3ProxyHandler.java:1983)
	at org.gaul.s3proxy.S3ProxyHandler.doHandle(S3ProxyHandler.java:759)
	at org.gaul.s3proxy.S3ProxyHandlerJetty.handle(S3ProxyHandlerJetty.java:77)
	at org.gaul.shaded.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.gaul.shaded.org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.gaul.shaded.org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
	at org.gaul.shaded.org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
	at org.gaul.shaded.org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
	at org.gaul.shaded.org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
	at org.gaul.shaded.org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.gaul.shaded.org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.gaul.shaded.org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.gaul.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.gaul.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.net.ProtocolException: Server rejected operation

[alifirat]

Hey @HotDog215
Can you try by setting S3PROXY_IGNORE_UNKNOWN_HEADERS to true ?

[HotDog215]

@alifirat Yes, I set it to true, but I still reported the same error

[alifirat]

Is it working with the aws-cli ?
Also @HotDog215 did you check that:

• https://github.com/gaul/s3proxy/wiki/Client-compatibility-list ?
• can you enable trace logs ?

[HotDog215]

Yes，When I use aws-cli, it works normally
But my configuration is slightly different

$ aws configure
AWS Access Key ID []: name-of-your-azure-storage-account
AWS Secret Access Key []: access-key-of-your-storage-account
Default region name [None]:
Default output format [None]:

Is it related to my S3cmd configuration? Causing inability to upload when using s3cmd @alifirat

[HotDog215]

[default]
access_key = name-of-your-azure-storage-account
add_encoding_exts =
add_headers =
bucket_location = s3proxy.s3proxy # access address of s3proxy, port 80
ca_certs_file =
cache_file =
check_ssl_certificate = True
check_ssl_hostname = True
cloudfront_host = cloudfront.amazonaws.com
connection_max_age = 5
connection_pooling = True
content_disposition =
content_type =
default_mime_type = binary/octet-stream
delay_updates = False
delete_after = False
delete_after_fetch = False
delete_removed = False
dry_run = False
enable_multipart = True
encoding = UTF-8
encrypt = False
expiry_date =
expiry_days =
expiry_prefix =
follow_symlinks = False
force = False
get_continue = False
gpg_command = None
gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_passphrase =
guess_mime_type = True
host_base = s3proxy.s3proxy # access address of s3proxy, port 80
#host_bucket = %(bucket)s.s3proxy.s3proxy
host_bucket = s3proxy.s3proxy # access address of s3proxy, port 80
human_readable_sizes = False
invalidate_default_index_on_cf = False
invalidate_default_index_root_on_cf = True
invalidate_on_cf = False
kms_key =
limit = -1
limitrate = 0
list_md5 = False
log_target_prefix =
long_listing = False
max_delete = -1
mime_type =
multipart_chunk_size_mb = 15
multipart_copy_chunk_size_mb = 1024
multipart_max_chunks = 10000
preserve_attrs = True
progress_meter = True
proxy_host =
proxy_port = 0
public_url_use_https = False
put_continue = False
recursive = False
recv_chunk = 65536
reduced_redundancy = False
requester_pays = False
restore_days = 1
restore_priority = Standard
secret_key = access-key-of-your-storage-account
send_chunk = 65536
server_side_encryption = False
signature_v2 = False
signurl_use_https = False
simpledb_host = sdb.amazonaws.com
skip_existing = False
socket_timeout = 300
ssl_client_cert_file =
ssl_client_key_file =
stats = False
stop_on_error = False
storage_class =
throttle_max = 100
upload_id =
urlencoding_mode = normal
use_http_expect = False
use_https = False
use_mime_magic = True
verbosity = WARNING
website_endpoint = http://%(bucket)s.s3-website-%(location)s.amazonaws.com/
website_error =
website_index = index.html

[alifirat]

If it's working with another client it's already good news and yeah I guess something wrong with s3cmd (don't know it personally).

Maybe @timuralp or @gaul has any clue on that ?

[bdeluca-igenius]

I was having the same issue and I realised that s3Proxy was trying to store the file in a Storage container that did not exist.

[gaul]

Could you try testing with the new azureblob-sdk provider from #606?