Skip to content

cache/s3: only request required upload checksums#6970

Open
crazy-max wants to merge 2 commits into
moby:masterfrom
crazy-max:fix-s3-cache-aws-chunked
Open

cache/s3: only request required upload checksums#6970
crazy-max wants to merge 2 commits into
moby:masterfrom
crazy-max:fix-s3-cache-aws-chunked

Conversation

@crazy-max

@crazy-max crazy-max commented Jul 20, 2026

Copy link
Copy Markdown
Member

fixes #6969

This updates the vendored AWS SDK for Go v2 modules and fixes S3 cache exports so BuildKit no longer sends optional trailing checksum uploads that force Content-Encoding: aws-chunked.

The dependency update is needed because the old transfer manager API doesn't expose RequestChecksumCalculation. The newer transfer manager release exposes that policy directly, so BuildKit can apply the same when_required behavior at the upload layer that it already applies at the S3 client layer.

The S3 cache client now passes RequestChecksumCalculationWhenRequired to the transfer manager, matching the existing S3 client configuration and allowing the SDK to avoid optional default CRC32 upload checksums while still preserving required or explicitly requested checksums.

BuildKit already configured the AWS S3 client with RequestChecksumCalculationWhenRequired:

aws_config.WithRequestChecksumCalculation(aws.RequestChecksumCalculationWhenRequired),

But the S3 cache exporter uses the higher-level transfer manager for uploads. Older transfer manager releases defaulted an empty checksum algorithm to CRC32 internally, which made upload requests explicitly request a trailing checksum. For streamed uploads, that explicit trailing checksum forces aws-chunked request encoding, and some S3-compatible backends reject that encoding with 501 NotImplemented.

Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>
Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>
@crazy-max
crazy-max force-pushed the fix-s3-cache-aws-chunked branch from 6732e81 to 5efadf0 Compare July 20, 2026 10:04
@crazy-max crazy-max added this to the v0.32.0 milestone Jul 20, 2026
@crazy-max
crazy-max requested a review from tonistiigi July 20, 2026 10:28
@crazy-max
crazy-max marked this pull request as ready for review July 20, 2026 10:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

s3 cache: AWS chunked encoding not supported on S3-compatible backends since v0.31

1 participant