Skip to content

Add full support for AWS chunked content encoding#4504

Merged
ysaito1001 merged 27 commits into
mainfrom
ysaito/support-full-aws-chunked-encoding
Feb 6, 2026
Merged

Add full support for AWS chunked content encoding#4504
ysaito1001 merged 27 commits into
mainfrom
ysaito/support-full-aws-chunked-encoding

Conversation

@ysaito1001
Copy link
Copy Markdown
Collaborator

@ysaito1001 ysaito1001 commented Jan 27, 2026

Motivation and Context

#1798
#4188

Description

This PR adds full support for AWS chunked content encoding, addressing limitations documented in these comments.

Customer impact

This PR affects S3's PutObject and UploadPart operations with streaming payloads, changing their default behavior:

Before: Streaming payloads were sent as a single chunk.

After: Streaming payloads are split into 64 KiB (the last chunk may be smaller) and sent in multiple chunks.

Customers who prefer the previous single-chunk behavior can disable chunking via the service config:

let config = aws_sdk_s3::Config::builder()
    .aws_chunked_encoding_chunk_size(None) // Use entire content as one chunk
    // --snip--
    .build();
let client = aws_sdk_s3::Client::from_conf(config);

or specify a custom chunk size

let config = aws_sdk_s3::Config::builder()
    .aws_chunked_encoding_chunk_size(Some(10240)) // 10 KiB chunks
    // --snip--
    .build();
let client = aws_sdk_s3::Client::from_conf(config);

Chunk signing

While this PR provides chunk signing capability, it only activates for non-TLS requests. For the majority of use cases where TLS provides encryption, streaming payloads continue to be sent unsigned with x-amz-content-sha256 set to STREAMING-UNSIGNED-PAYLOAD-TRAILER.

Implementation overview

The PR implements three key components:

  1. Stream chunking: Breaks input streams into chunks
  2. Deferred signer: Provides a signer specifically for aws-chunked encoding
  3. Signer integration: Threads the signer through to AwsChunkedBody

Similar to event stream signing, chunk signing occurs after the initial HTTP request signing (sign_http_request) and requires context from that process. The deferred signer enables chunk signing to be wired up later as streaming bodies are yielded.

Testing

  • Unit tests for int_log16; it had a bug for the input value 0 (should return 1 but had returned 0)
  • Unit tests for stream chunking and chunk signing
  • Integration tests for chunk signing and configurable chunk sizes
  • CI

Checklist

  • For changes to the AWS SDK, generated SDK code, or SDK runtime crates, I have created a changelog entry Markdown file in the .changelog directory, specifying "aws-sdk-rust" in the applies_to key.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@smithy-lang smithy-lang deleted a comment from github-actions Bot Jan 28, 2026
@smithy-lang smithy-lang deleted a comment from github-actions Bot Jan 28, 2026
@smithy-lang smithy-lang deleted a comment from github-actions Bot Jan 28, 2026
@ysaito1001 ysaito1001 marked this pull request as ready for review January 28, 2026 21:51
@ysaito1001 ysaito1001 requested review from a team as code owners January 28, 2026 21:51
@github-actions
Copy link
Copy Markdown

A new generated diff is ready to view.

  • AWS SDK (ignoring whitespace)
  • No codegen difference in the Client Test
  • No codegen difference in the Server Test
  • No codegen difference in the Server Test Python
  • No codegen difference in the Server Test Typescript

A new doc preview is ready to view.

@smithy-lang smithy-lang deleted a comment from github-actions Bot Jan 29, 2026
@github-actions
Copy link
Copy Markdown

A new generated diff is ready to view.

  • AWS SDK (ignoring whitespace)
  • No codegen difference in the Client Test
  • No codegen difference in the Server Test
  • No codegen difference in the Server Test Python
  • No codegen difference in the Server Test Typescript

A new doc preview is ready to view.

Copy link
Copy Markdown
Collaborator

@landonxjames landonxjames left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! One question about an edge case with user-set chunk sizes and some other small notes, nothing blocking.

Comment thread .changelog/1769635097.md Outdated
Comment thread aws/rust-runtime/aws-runtime/src/auth/sigv4.rs
@github-actions
Copy link
Copy Markdown

A new generated diff is ready to view.

  • AWS SDK (ignoring whitespace)
  • No codegen difference in the Client Test
  • No codegen difference in the Server Test
  • No codegen difference in the Server Test Python
  • No codegen difference in the Server Test Typescript

A new doc preview is ready to view.

Comment thread .changelog/1769635097.md Outdated
Comment thread .changelog/1769635097.md Outdated
Comment thread aws/rust-runtime/aws-runtime/src/content_encoding/body.rs
Comment thread aws/rust-runtime/aws-sigv4/src/sign/v4.rs Outdated
Comment thread aws/rust-runtime/aws-sigv4/src/sign/v4.rs Outdated
@ysaito1001 ysaito1001 requested a review from aajtodd February 4, 2026 00:38
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Feb 4, 2026

A new generated diff is ready to view.

  • AWS SDK (ignoring whitespace)
  • No codegen difference in the Client Test
  • No codegen difference in the Server Test
  • No codegen difference in the Server Test Python
  • No codegen difference in the Server Test Typescript

A new doc preview is ready to view.

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Feb 4, 2026

A new generated diff is ready to view.

  • AWS SDK (ignoring whitespace)
  • No codegen difference in the Client Test
  • No codegen difference in the Server Test
  • No codegen difference in the Server Test Python
  • No codegen difference in the Server Test Typescript

A new doc preview is ready to view.

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Feb 4, 2026

A new generated diff is ready to view.

  • AWS SDK (ignoring whitespace)
  • No codegen difference in the Client Test
  • No codegen difference in the Server Test
  • No codegen difference in the Server Test Python
  • No codegen difference in the Server Test Typescript

A new doc preview is ready to view.

Copy link
Copy Markdown
Contributor

@aajtodd aajtodd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good.

@ysaito1001 ysaito1001 enabled auto-merge February 6, 2026 03:46
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Feb 6, 2026

A new generated diff is ready to view.

  • AWS SDK (ignoring whitespace)
  • No codegen difference in the Client Test
  • No codegen difference in the Server Test
  • No codegen difference in the Server Test Python
  • No codegen difference in the Server Test Typescript

A new doc preview is ready to view.

@ysaito1001 ysaito1001 merged commit b442542 into main Feb 6, 2026
90 of 94 checks passed
@ysaito1001 ysaito1001 deleted the ysaito/support-full-aws-chunked-encoding branch February 6, 2026 05:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants