s3 cache: AWS chunked encoding not supported on S3-compatible backends since v0.31

[syail]

Contributing guidelines and issue reporting guide

• I've read the contributing guidelines and wholeheartedly agree. I've also read the issue reporting guide.

Well-formed report checklist

• I have found a bug that the documentation does not mention anything about my problem
• I have found a bug that there are no open or closed issues that are related to my problem
• I have provided version/information about my environment and done my best to provide a reproducer

Bug description

Since BuildKit v0.31, exporting cache to an S3-compatible object storage that does not
support aws-chunked content encoding (e.g. Oracle Cloud OCI Object Storage S3-compat
API) fails:

#18 ERROR: error writing layer blob: operation error S3: PutObject, https response error StatusCode: 501, RequestID: yny-1:<redacted>, HostID: , api error NotImplemented: AWS chunked encoding not supported.

The same configuration works fine with BuildKit v0.30.0.

Root cause

In v0.31 the s3 cache backend was migrated from the old upload path to the new
github.com/aws/aws-sdk-go-v2/feature/s3/transfermanager (pinned at v0.2.9).

transfermanager v0.2.9 unconditionally defaults ChecksumAlgorithm to CRC32:

// feature/s3/transfermanager/options.go (v0.2.9)
func resolveChecksumAlgorithm(o *Options) {
	if o.ChecksumAlgorithm == "" {
		o.ChecksumAlgorithm = types.ChecksumAlgorithmCrc32
	}
}

and passes it explicitly to every PutObject / UploadPart. Because the checksum is
set explicitly on the request, the RequestChecksumCalculationWhenRequired client
option that cache/remotecache/s3/s3.go configures (and the
AWS_REQUEST_CHECKSUM_CALCULATION=when_required env var) has no effect — those only
suppress the SDK's implicit default checksums.

An explicit checksum on a streamed body is sent as a trailing checksum, which forces
Content-Encoding: aws-chunked — and S3-compatible backends that don't implement
aws-chunked reject the request with 501 NotImplemented.

This is fixed upstream in transfermanager v0.3.0, which respects
RequestChecksumCalculationWhenRequired and leaves the checksum algorithm empty in
that case:

// feature/s3/transfermanager/options.go (v0.3.0)
func (o Options) checksumAlgorithm() types.ChecksumAlgorithm {
	if o.ChecksumAlgorithm != "" {
		return o.ChecksumAlgorithm
	}
	if o.RequestChecksumCalculation != aws.RequestChecksumCalculationWhenRequired {
		return types.ChecksumAlgorithmCrc32
	}
	return ""
}

Proposed fix: bump feature/s3/transfermanager to >= v0.3.0 (and ensure the
client's RequestChecksumCalculation setting is propagated to the transfer manager
options).

Reproduction

1. Use any S3-compatible storage without aws-chunked support (OCI Object Storage
   S3-compat endpoint in my case)
2. Run a build with an s3 cache export:

docker buildx build . \
  --cache-to "type=s3,region=<region>,bucket=<bucket>,name=test,mode=max,use_path_style=true,endpoint_url=https://<namespace>.compat.objectstorage.<region>.oci.customer-oci.com,access_key_id=...,secret_access_key=..." \
  --cache-from "type=s3,..."

3. Cache export fails with the 501 error above.

Setting AWS_REQUEST_CHECKSUM_CALCULATION=when_required /
AWS_RESPONSE_CHECKSUM_VALIDATION=when_required on the buildkitd container does NOT
help, for the reason described above.

Version information

• BuildKit: v0.31.2
• buildx: v0.35.0
• Driver: docker-container (via docker/setup-buildx-action@v4, GitHub Actions self-hosted runner)
• Backend: Oracle Cloud Infrastructure Object Storage (S3 Compatibility API)

[syail]

the root cause is tracked upstream in aws/aws-sdk-go-v2#3469