errors due to lack of unsupported checksums are not clear

[nguyengg]

Acknowledgements

• I have searched (https://github.com/aws/aws-sdk/issues?q=is%3Aissue) for past instances of this issue
• I have verified all of my SDK modules are up-to-date (you can perform a bulk update with go get -u github.com/aws/aws-sdk-go-v2/...)

Describe the bug

My setup looks like this:

// start the multipart upload.
&s3.CreateMultipartUploadInput{
	ChecksumAlgorithm: types.ChecksumAlgorithmCrc64nvme,
	ChecksumType:      types.ChecksumTypeFullObject,
}

// upload a part.
lev := crc64.New(crc64.MakeTable(0xAD93D23594C93659))
_, _ = lev.Write(data)
input := &s3.UploadPartInput{
	ChecksumAlgorithm: s3types.ChecksumAlgorithmCrc64nvme,
	ChecksumCRC64NVME: aws.String(base64.StdEncoding.EncodeToString(lev.Sum(nil))),
}

With this setup, I run into the first error:

operation error S3: UploadPart, failed to parse algorithm, unknown checksum algorithm, CRC64NVME

No HTTP request was made, which is probably because the parsing fails due to https://github.com/aws/aws-sdk-go-v2/blob/061fd1b6d9940c6b37974e6aa64a5f9dc4ee1dc4/service/internal/checksum/algorithms.go not having support for CRC64NVME in the Parse* functions.

I understand CRC64NVME is not supported at composite level (so there's really no point in specifying it during UploadPart), so when I try this:

// start is still the same
&s3.CreateMultipartUploadInput{
	ChecksumAlgorithm: types.ChecksumAlgorithmCrc64nvme,
	ChecksumType:      types.ChecksumTypeFullObject,
}

// upload a part without an algorithm.
input := &s3.UploadPartInput{
	ChecksumAlgorithm: "",
}

This is what I get:

PUT [redacted]
Host: s3.us-west-2.amazonaws.com
User-Agent: aws-sdk-go-v2/1.36.1 ua/2.1 os/linux lang/go#1.23.6 md/GOOS#linux md/GOARCH#amd64 api/s3#1.75.4 m/E,Z
Content-Length: 233763
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 0b356ae8-eaab-4ecd-90df-f7ce9e9f76af
Amz-Sdk-Request: attempt=1; max=3
Authorization: [redacted]
Content-Encoding: aws-chunked
Content-Type: application/octet-stream
X-Amz-Content-Sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER
X-Amz-Date: 20250206T232448Z
X-Amz-Decoded-Content-Length: 233718
X-Amz-Trailer: x-amz-checksum-crc32
----------------------------------------------------------------
request failed with unretryable error https response error StatusCode: 400, RequestID: V4NHA53PZQ3WPWJG, HostID: L4laOR3k6x323g5yyiAzJqbFmmMi5n6lfB109xRQ3TmJ0YPwLFkBO5dzbtxqXI+C6v71Hf0jRkQ=, api error InvalidRequest: Checksum Type mismatch occurred, expected checksum Type: crc64nvme, actual checksum Type: crc32

So looks like if I don't specify ChecksumAlgorithm as part of UploadInput, Go SDK will attempt to attach x-amz-checksum-crc32 to it? Which then causes the server to fail the request with a 400.

Am I flat out unable to use CRC64NVME at all during the initial CreateMultipartUpload request until #2985 (S3 object integrity: support CRC64NVME #2985 ) is addressed?

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

The UploadPart request should complete successfully.

Current Behavior

The UploadPart request fails with or without specifying checksum.

Reproduction Steps

First problem:

// start the multipart upload.
&s3.CreateMultipartUploadInput{
	ChecksumAlgorithm: types.ChecksumAlgorithmCrc64nvme,
	ChecksumType:      types.ChecksumTypeFullObject,
}

// upload a part.
lev := crc64.New(crc64.MakeTable(0xAD93D23594C93659))
_, _ = lev.Write(data)
input := &s3.UploadPartInput{
	ChecksumAlgorithm: s3types.ChecksumAlgorithmCrc64nvme,
	ChecksumCRC64NVME: aws.String(base64.StdEncoding.EncodeToString(lev.Sum(nil))),
}

Second problem:

// start is still the same
&s3.CreateMultipartUploadInput{
	ChecksumAlgorithm: types.ChecksumAlgorithmCrc64nvme,
	ChecksumType:      types.ChecksumTypeFullObject,
}

// upload a part without an algorithm.
input := &s3.UploadPartInput{
	ChecksumAlgorithm: "",
}

Possible Solution

No response

Additional Information/Context

No response

AWS Go SDK V2 Module Versions Used

	github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.8 // indirect
	github.com/aws/aws-sdk-go-v2/credentials v1.17.59 // indirect
	github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.28 // indirect
	github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.32 // indirect
	github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.32 // indirect
	github.com/aws/aws-sdk-go-v2/internal/ini v1.8.2 // indirect
	github.com/aws/aws-sdk-go-v2/internal/v4a v1.3.32 // indirect
	github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.12.2 // indirect
	github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.5.6 // indirect
	github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.12.13 // indirect
	github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.18.13 // indirect
	github.com/aws/aws-sdk-go-v2/service/sso v1.24.15 // indirect
	github.com/aws/aws-sdk-go-v2/service/ssooidc v1.28.14 // indirect
	github.com/aws/aws-sdk-go-v2/service/sts v1.33.14 // indirect

Compiler and Version used

go version go1.23.6 linux/amd64

Operating System and version

5.15.167.4-microsoft-standard-WSL2

[nguyengg]

Third issue I found: if I don't specify any checksum algorithm or checksum type at all, the CompleteMultipartUpload will fail.

Setup:

// start the multipart upload without any algorithm or type.
&s3.CreateMultipartUploadInput{
	ChecksumAlgorithm: "",
	ChecksumType:      "",
}

// upload a part without any algorithm or type.
input := &s3.UploadPartInput{
	ChecksumAlgorithm: "",
	ChecksumCRC64NVME: "",
}

// complete multiupload will fai with message:
api error InvalidPart: One or more of the specified parts could not be found.  The part may not have been uploaded, or the specified entity tag may not match the part's entity tag.

HTTP log for UploadPart shows:

PUT [redacted]
Host: s3.us-west-2.amazonaws.com
User-Agent: aws-sdk-go-v2/1.36.1 ua/2.1 os/linux lang/go#1.23.6 md/GOOS#linux md/GOARCH#amd64 api/s3#1.75.4 m/E,Z
Content-Length: 233763
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 6eef4462-bbe9-4fd6-a266-c4ff79c7ebf1
Amz-Sdk-Request: attempt=1; max=3
Authorization: [redacted]
Signature=[redacted]
Content-Encoding: aws-chunked
Content-Type: application/octet-stream
X-Amz-Content-Sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER
X-Amz-Date: 20250207T005111Z
X-Amz-Decoded-Content-Length: 233718
X-Amz-Trailer: x-amz-checksum-crc32

The HTTP log is identical to the UploadPart input for Amz-Sdk-Invocation-Id: 0b356ae8-eaab-4ecd-90df-f7ce9e9f76af which makes sense because they are created identically, but the error message from CompleteMultipartUpload is different. If I didn't specify any checksum algorithm, the error message is:

api error InvalidPart: One or more of the specified parts could not be found. The part may not have been uploaded, or the specified entity tag may not match the part's entity tag.

If I specified CRC64NVME, the error message is:

api error InvalidRequest: Checksum Type mismatch occurred, expected checksum Type: crc64nvme, actual checksum Type: crc32

The former error message indicates a regression that I also called out in #2960 (comment).

[lucix-aws]

These should all be OBE when we merge #3002.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.