-
Notifications
You must be signed in to change notification settings - Fork 314
Description
Context
We currently work around a moto bug in our S3 integration tests by limiting test file sizes to 4MB to avoid triggering multipart uploads.
The Problem
Moto issue #8762 causes multipart upload tests to fail with AWS SDK v2 due to incorrect checksum handling:
- Composite checksums for multipart uploads should have a
-X
suffix to distinguish them from full object checksums - Moto doesn't add this suffix, causing AWS SDK v2 checksum validation to fail with errors like:
checksum did not match: algorithm CRC32, expect PnvBcQ==, actual KxHXkQ==
Current Workaround
In replica_client_test.go
, the TestReplicaClient_S3_UploaderConfig
test uses a 4MB file instead of a larger file to avoid triggering multipart uploads:
// Create a 4MB test file. We use 4MB instead of a larger file to avoid
// triggering multipart uploads which have a known issue with moto mock server.
// Moto issue #8762: composite checksums don't have the -X suffix, causing
// AWS SDK v2 checksum validation to fail.
// Reference: https://github.com/getmoto/moto/issues/8762
Action Required Once Moto is Fixed
Once moto issue #8762 is resolved, we should:
-
Update the test to use a larger file size (e.g., 10MB or more) to properly test multipart upload functionality:
// Change from: size := 4 * 1024 * 1024 // 4MB // To: size := 10 * 1024 * 1024 // 10MB (or larger)
-
Update or remove the workaround comment to reflect that the moto issue has been fixed
-
Verify the test passes with multipart uploads:
./etc/s3_mock.py go test -v ./replica_client_test.go -integration s3 -run TestReplicaClient_S3_UploaderConfig
-
Consider adding additional multipart upload tests to ensure our S3 client properly handles large files with custom PartSize and Concurrency settings
Benefits of Fixing This
- Better test coverage for multipart upload scenarios
- Validation that our custom PartSize and Concurrency settings work correctly with large files
- More realistic testing of production scenarios where large files are uploaded