HADOOP-19814: Update hadoop3 Docker image to use Hadoop 3.5.0.

[cnauroth]

Description of PR

Update hadoop3 Docker image to use Hadoop 3.5.0.

How was this patch tested?

Do not commit. This is currently not working due to two problems:

1. Downloads from dlcdn.apache.org are hanging for me right now. I haven't investigated, but I'll assume it's a temporary operational problem for now. For now, I'm working around it by patching locally to use downloads.apache.org instead.
2. The base apache/hadoop-runner image has Java 1.8, so it fails to load the 3.5.0 release's Java 17 bytecode.

For code changes:

• Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

AI Tooling

If an AI tool was used:

• The PR includes the phrase "Contains content generated by "
  where is the name of the AI tool used.
• My use of AI contributions follows the ASF legal policy
  https://www.apache.org/legal/generative-tooling.html

[cnauroth]

Hello @jojochuang and @ayushtkn . Do you know what we need to do to get an updated apache/hadoop-runner base image with an upgrade to Java 17 so that we can bundle Hadoop 3.5.0?

[ayushtkn]

There is a tag: https://hub.docker.com/layers/apache/hadoop-runner/jdk17-u2204/images/sha256-888e24acb34672d5aff591a13d0292c956374ac577327de4df3900574f3fd152

#7046

[pan3793]

The hadoop-runner uses Ubuntu 22.04 as the base image, which is inconsistent with the Hadoop release dev container - Hadoop 3.5.x uses Ubuntu 24.04, Hadoop 3.4.x uses Ubuntu 20.04, should we align that?

[cnauroth]

I'm thinking what we need to do is:

1. Release a new apache/hadoop-runner:jdk17-u2204apache/hadoop-runner:jdk17-u2404 that aligns the OS with the release container like Cheng suggested.
2. Additionally, the base container needs another change to set ownership of /opt to the hadoop user so that it has permission to install the Hadoop bits. This line in apache/hadoop-runner:latest is missing from apache/hadoop-runner:jdk17-u2204. Actually these 2 images are quite different. I wonder if they should be more aligned.
3. Create a new docker-hadoop-3.5 branch that uses the new apache/hadoop-runner:jdk17-u2204 for the base image. This way, we'll be able to independently release images for new 3.4 and 3.5 releases.

I'll proceed with this approach in a little while unless others have feedback.

[pan3793]

should it be apache/hadoop-runner:jdk17-u2404? (not u2204)

[cnauroth]

Yes, thanks for the correction. The intent is to align on Ubuntu 24.04. I updated the comment.

[ajfabbri]

Hey guys. Stumbled upon this and I'm wondering if this stuff is documented? (Which containers we publish and what the process is). If not I'd be happy to write it up. Also shout if you need any help.