TestS3FileIO fails locally (on OSX with Docker Desktop) due to missing Content-MD5 header during delete

[nastra]

Apache Iceberg version

main

Query engine

None

Please describe the bug 🐞

./gradlew :iceberg-aws:test

> Task :iceberg-aws:test

TestS3FileIO > testDeleteFilesSingleBatchWithRemainder() FAILED
    org.apache.iceberg.io.BulkDeletionFailureException: Failed to delete 18 files
        at app//org.apache.iceberg.aws.s3.S3FileIO.deleteFiles(S3FileIO.java:240)
        at app//org.apache.iceberg.aws.s3.TestS3FileIO.testBatchDelete(TestS3FileIO.java:231)
        at app//org.apache.iceberg.aws.s3.TestS3FileIO.testDeleteFilesSingleBatchWithRemainder(TestS3FileIO.java:186)

TestS3FileIO > testDeleteFilesMultipleBatches() FAILED
    org.apache.iceberg.io.BulkDeletionFailureException: Failed to delete 30 files
        at app//org.apache.iceberg.aws.s3.S3FileIO.deleteFiles(S3FileIO.java:240)
        at app//org.apache.iceberg.aws.s3.TestS3FileIO.testBatchDelete(TestS3FileIO.java:231)
        at app//org.apache.iceberg.aws.s3.TestS3FileIO.testDeleteFilesMultipleBatches(TestS3FileIO.java:176)

TestS3FileIO > testDeleteFilesLessThanBatchSize() FAILED
    org.apache.iceberg.io.BulkDeletionFailureException: Failed to delete 12 files
        at app//org.apache.iceberg.aws.s3.S3FileIO.deleteFiles(S3FileIO.java:240)
        at app//org.apache.iceberg.aws.s3.TestS3FileIO.testBatchDelete(TestS3FileIO.java:231)
        at app//org.apache.iceberg.aws.s3.TestS3FileIO.testDeleteFilesLessThanBatchSize(TestS3FileIO.java:181)

229 tests completed, 3 failed, 1 skipped

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time

[ebyhr]

All tests passed with OrbStack on my laptop. I tested with the following environment:

------------------------------------------------------------
Gradle 8.12.1
------------------------------------------------------------

Build time:    2025-01-24 12:55:12 UTC
Revision:      0b1ee1ff81d1f4a26574ff4a362ac9180852b140

Kotlin:        2.0.21
Groovy:        3.0.22
Ant:           Apache Ant(TM) version 1.10.15 compiled on August 25 2024
Launcher JVM:  11.0.25 (Azul Systems, Inc. 11.0.25+9-LTS)
Daemon JVM:    /Users/yuya.ebihara/.sdkman/candidates/java/11.0.25-zulu/zulu-11.jdk/Contents/Home (no JDK specified, using current Java home)
OS:            Mac OS X 15.0 aarch64

[nastra]

I've talked to @ebyhr on Slack and he's using OrbStack while I'm using Docker Desktop. After switching to OrbStack locally the tests started to pass again. So we need to figure out here why it passes in one Docker Env while failing in the other so that we get consistent results for all people running these tests

[Fokko]

I'm also on orbstack, and works fine on my end as well:

./gradlew :iceberg-aws:test 
Configuration on demand is an incubating feature.

> Task :iceberg-api:testJar
file '/Users/fokko.driesprong/work/iceberg/build/iceberg-build.properties' will be copied to 'iceberg-build.properties', overwriting file '/Users/fokko.driesprong/work/iceberg/api/build/resources/test/iceberg-build.properties', which has already been copied there.
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended

[Incubating] Problems report is available at: file:///Users/fokko.driesprong/work/iceberg/build/reports/problems/problems-report.html

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.12.1/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD SUCCESSFUL in 35s
20 actionable tasks: 8 executed, 12 up-to-date

[nastra]

After locally upgrading the MinIO tag to the version mentioned in https://github.com/apache/iceberg/pull/12264/files#r1961355014 it also works with Docker Desktop. @ebyhr since you're already working on #12264 could you please change the MinIO tag in your PR?

[ebyhr]

I believe this is a user-facing issue, not specific to tests.
The issue title can be change to "Cannot access to older S3-compatible storages since 1.8.0" or something.

[nastra]

Worth mentioninig that this issue started to happen after we upgraded the AWS SDK to 2.30.0, which added changes to the default integrity protection (see aws/aws-sdk-java-v2#5802 and thus negatively affects S3-compliant object storage solutions.
After reverting the AWS SDK for 1.8.x in #12339 the issue was fixed. For main @ebyhr is working on a proper fix in #12264