Chunk Encoding

[vishaalt]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

-->

Tell us about your request
Add support for Chunk Encoding

[brainstorm]

Can you elaborate on what you mean by "Chunk Encoding" ?

[rcoh]

This refers to the SDK being able to support Transfer-Encoding: Chunked. It's primarily used for uploading files to S3.

[jdisanti]

Implementation is being tracked in smithy-lang/smithy-rs#1798

[rigille]

Is there anything we could do to help getting this feature implemented?

[rcoh]

The main thing here is actually prioritization. Can you tell us more about your use case and why the current SDK doesn't meet those needs?

[rigille]

Sometimes I'm processing a stream of data but don't have enough to save the result locally before uploading. Saving and then uploading would also increase latency

[mmason-evertz]

Sometimes I'm processing a stream of data but don't have enough to save the result locally before uploading. Saving and then uploading would also increase latency

+1 for low latency and close to real time use cases

[rakshith-ravi]

Can you tell us more about your use case and why the current SDK doesn't meet those needs?

I am implementing an OCI compliant registry and handling upload sizes in GiBs is very common. Naturally, buffering it to memory isn't practical. Most OCI clients (Docker, Podman, etc) sends the uploads as chunked transfer encoding instead of sending the size in the content-length header. The current SDK expects us to buffer the entire data in memory, or know the size before-hand, neither of which is possible currently. Being able to process chunked encoding is definitely something that would solve for this.

I'm happy to implement this. Please guide me on how to proceed and I'll put a PR with the changes

[rakshith-ravi]

The associated PR (smithy-lang/smithy-rs#4504) has been merged. Will this hopefully land in the next release?