Skip to content

Update S3 multipart upload test once moto issue #8762 is resolved #700

@corylanou

Description

@corylanou

Context

We currently work around a moto bug in our S3 integration tests by limiting test file sizes to 4MB to avoid triggering multipart uploads.

The Problem

Moto issue #8762 causes multipart upload tests to fail with AWS SDK v2 due to incorrect checksum handling:

  • Composite checksums for multipart uploads should have a -X suffix to distinguish them from full object checksums
  • Moto doesn't add this suffix, causing AWS SDK v2 checksum validation to fail with errors like:
    checksum did not match: algorithm CRC32, expect PnvBcQ==, actual KxHXkQ==
    

Current Workaround

In replica_client_test.go, the TestReplicaClient_S3_UploaderConfig test uses a 4MB file instead of a larger file to avoid triggering multipart uploads:

// Create a 4MB test file. We use 4MB instead of a larger file to avoid
// triggering multipart uploads which have a known issue with moto mock server.
// Moto issue #8762: composite checksums don't have the -X suffix, causing
// AWS SDK v2 checksum validation to fail.
// Reference: https://github.com/getmoto/moto/issues/8762

Action Required Once Moto is Fixed

Once moto issue #8762 is resolved, we should:

  1. Update the test to use a larger file size (e.g., 10MB or more) to properly test multipart upload functionality:

    // Change from:
    size := 4 * 1024 * 1024  // 4MB
    
    // To:
    size := 10 * 1024 * 1024 // 10MB (or larger)
  2. Update or remove the workaround comment to reflect that the moto issue has been fixed

  3. Verify the test passes with multipart uploads:

    ./etc/s3_mock.py go test -v ./replica_client_test.go -integration s3 -run TestReplicaClient_S3_UploaderConfig
  4. Consider adding additional multipart upload tests to ensure our S3 client properly handles large files with custom PartSize and Concurrency settings

Benefits of Fixing This

  • Better test coverage for multipart upload scenarios
  • Validation that our custom PartSize and Concurrency settings work correctly with large files
  • More realistic testing of production scenarios where large files are uploaded

Tracking

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions