Refactor S3 upload logic - replace AWS CLI with SDK

[Ben-El]

Closes #9512

Followup task of #9492

Changes

• Replace the external aws s3api call with AWS SDK v2 (S3Client.putObject).
• Keep atomic “create-if-absent” semantics using PutObjectRequest.ifNoneMatch("*") (no overwrites).
• Log via streams.value.log (INFO for stdout-like messages).
• On errors, surface clear messages, and map 412 to “Artifact already exists”.
• There's no --acl public-read anymore. With Bucket Owner Enforced, ACLs are ignored and access is governed by the bucket policy.
• Add meta-build deps under clients/spark/project/s3-upload-sdk.sbt:
  • software.amazon.awssdk:s3 and :regions (v2.33.x).
  • These are build-time only and do not enter the assembly jar (no impact on EMR6/EMR7 runtime).

Testings

On the actual bucket, verified that:

• New jars upload successfully (and publicly accessible).
• Re-publishing an existing version failed as expected.

[arielshaqed]

I don't understand why we cannot just use a Java S3 client - either the AWS one or the MinIO one would do. AFAIK both support "if-none-match" semantics.

The thing about running processes is that they can fail in so many interesting ways! For example, the previous code would (always) lose output. The new code can report strange exit codes (>128) if the process exits with a signal. It separates the failure notification from the error message, because the error message goes out to the process logs. It will also break easily if aws is not on the path, or if the wrong version of aws is on the path. It has an unenforced dependency when used in an action - you must install an AWS CLI. And I find it hard to predict what it will do in different circumstances.

[Ben-El]

I don't understand why we cannot just use a Java S3 client - either the AWS one or the MinIO one would do. AFAIK both support "if-none-match" semantics.

The thing about running processes is that they can fail in so many interesting ways! For example, the previous code would (always) lose output. The new code can report strange exit codes (>128) if the process exits with a signal. It separates the failure notification from the error message, because the error message goes out to the process logs. It will also break easily if aws is not on the path, or if the wrong version of aws is on the path. It has an unenforced dependency when used in an action - you must install an AWS CLI. And I find it hard to predict what it will do in different circumstances.

You're right, updated the solution !
Thx !

[arielshaqed]

Thanks! I highly recommend a fix - adding the cause - to the exceptions thrown on error. And this does not need to be re-reviewed.

Note that this also removes ACL: public-read.

[nopcoder]

we can get it from env or other supported way by the java sdk

[nopcoder]

lgtm. remove the extra import.