Stop publishing `agent` images, keep only the `inbound-agent` ones

[lemeurherve]

We're currently building and publishing agent and inbound-agent images from this repository.

The agent image contains the JDK, required tools and the remoting JAR.
The inbound-agent adds to the agent base image the jenkins-agent script and sets it as entrypoint.

Here are the current linux targets of those images:

$ make list
agent_alpine_jdk21
agent_alpine_jdk25
agent_debian_jdk17
agent_debian_jdk21
agent_debian_jdk25
agent_rhel_ubi9_jdk17
agent_rhel_ubi9_jdk21
agent_rhel_ubi9_jdk25
inbound-agent_alpine_jdk21
inbound-agent_alpine_jdk25
inbound-agent_debian_jdk17
inbound-agent_debian_jdk21
inbound-agent_debian_jdk25
inbound-agent_rhel_ubi9_jdk17
inbound-agent_rhel_ubi9_jdk21
inbound-agent_rhel_ubi9_jdk25

Just looking at the current remoting release line 3355.v388858a_47b_33, on its 17 releases so far only 3 of them successfully published linux images in their first builds.

For the others, I had to make replays to avoid publishing all those images at once.

In most of those replays, I've focused on inbound-agent images and skipped the agent ones, but no complains of missing images AFAIK.

Are there knowns uses of agent images? I've searched a bit but didn't find anything.

If not, I'm proposing to stop providing agent images and just publish the inbound-agent ones, WDYT?

[dduportal]

I endorse the idea and made a similar proposal 2-3 years ago.

Alas, the agent images are still used by the Docker plugin: https://plugins.jenkins.io/docker-plugin/#plugin-content-launch-attached. So alas we can't do this "right now".

Also, even if no one complained, we can't rule out that they were using latest tag leaving them with outdated and potentially unsafe images.

A few proposed actions points:

• Since the publication always fails with HTTP/429 (from the second DockerHub protection layer protecting them from peak of requests, not the "usual" rate limit requiring authentication as we already authenticate with no rate limit), it means we are hammering the DockerHub:

  • Let's spread the publications in the current process by batching them. We are producing too much layers for publication, it is a fact (so many distributions and CPU architectures). That would mean we could start with a "divide by 2": let's update our pipeline to publish all agent images first. Then wait 60s and publish all inbound-agent images.
    • Benefit: the agent layers shared with inbound-agent images would not be pushed twice (buildx parallelism is expected to NOT do that, but it clearly does randomly, indicating an issue in its parallelization).
    • Benefit: we should start decreasing the pressure on DockerHub from our outbound IP, letting time to their "2nd" rate limit layer to reset our quota (hence the 60s delay).
    • Con: we add 60s in the publication time
    • If it fails, we can even split in 4 batches using Linux distributions (agent Debian, agent RHEL/Alpine, and then same for inbounds).
  • Let's use 4 outbound IPs instead of 2 for the trusted.ci ephemeral agents, to spread the load from DockerHub point of view.
  • Spread the Linux build on 2 distinct agents (instead of one today). It should increase the probability to spread requests on the 2 outbound IPs. Let's say agent images on one agent and inbound-agent on the other

• Long term: our process should publish in the private ACR registry, and then automatically promote on DockerHub with retries

[dduportal]

On the proposed topic:

• Not updating the agent images and/or deprecating them are breaking change which should be advertised loudly, clearly and also fixed as soon as we can
• Since we aim at deprecating agent images, I propose to make it favor of inbound-agent as alternative to users. That would be a breaking change for them (entrypoint would be changed) but acceptable since inbound only add 2 shell scripts.