Restore Windows 2019 support for LTS 2.541.1 release

[lemeurherve]

As Windows 2019 docker controller image has been replaced by Windows 2022 image in #2170 cf #2167 and https://github.com/jenkinsci/docker/releases/tag/2.545, we need to decide if we ship this change into incoming LTS 2.541.1 release.

To publish LTS 2.541.1 next week with a Windows Server Core 2019 image in addition to the Windows Server Core 2022, we can:

• Use a stable-2.541 branch (cf Publish Docker controller LTS images from a stable-<release-line> branch #2195) to backport the proper Windows version to build and to use as agent
• Ship chore(windows): use docker bake to generate the docker compose file #2189, notably 4e7a288 temporary workaround
• Expand 353058d mechanism to also set the proper agent to use in the pipeline

Otherwise we would be dropping support for Windows 2019 in the controller image earlier than expected, as Windows 2019 is in extended support until January 9th, 2029 cf https://learn.microsoft.com/en-us/lifecycle/products/windows-server-2019

(Assigning both Damien an I as Infra team responsible of LTS packaging process and backport)

Refs:

• Provide Windows Server 2022 image #2167
• LTS 2.541.1 release jenkins-infra/helpdesk#4945
• LTS 2.541.1 release checklist jenkins-infra/release#811
• Drop Windows 2019 support jenkins-infra/helpdesk#4954

[timja]

I wonder if anyone is actually using the Windows docker images...

#1096 (comment)

Might be something that @NotMyFault 's telemetry can capture?

[dduportal]

I wonder if anyone is actually using the Windows docker images...

#1096 (comment)

Might be something that @NotMyFault 's telemetry can capture?

We have DockerHub stats to get this. Let me update them so we'll be able to check.

I have serious doubts about #10: it was with old Windows versions, but worth checking.

(I mean, we have tests in this repository which starts the container. So I expect it to be working as expected)

[timja]

3.6% (468) of pulls (12,988) in December for the Jenkins docker image were windows.

[lemeurherve]

Regardless of the usage, and since #2195 has been accepted, I suggest restoring Windows 2019 controller image in stable-2.541 backport for publication of both Windows 2019 and 2022 images at least for the incoming LTS 2.541.1, to avoid premature Windows 2019 drop.

We could discuss later what to do about them. (Keep both, only the 2022 one or none)

WDYT @timja @dduportal?

[NotMyFault]

I wonder if anyone is actually using the Windows docker images...

#1096 (comment)

Might be something that @NotMyFault 's telemetry can capture?

Out of 9975 data sets collected so far, only 300 instances report "Windows Server 2019".

[dduportal]

Thanks for the stats! So it means we have only a few users left but we still have some.

As per Herve's description, since we are going to have a stable-2.541 branch here, are there any reason NOT to wait for the normal weekly -> LTS process for dropping Windows 2019?

E.g. since 2019 has been replaced by 2022 in 2.545, then we'll wait for the next LTS line which is expected to embed all notable changes from 2.542 (including deprecation of 2019)?

[timja]

Its already out of support so I don't think we should really ship it and a new LTS line is a good time to ship.

But we can do one more if others think we should...

[dduportal]

Its already out of support

That's the thing: it is not. What @lemeurherve and I carefully checked yesterday:

• Windows Server 2019 is in "Extended" support (it was in "Main Stream" until January 2024) until 2029

  • It means it still receives Security updates but no new features (as per https://learn.microsoft.com/en-us/lifecycle/policies/fixed#lifecycle-phases-for-products-under-the-fixed-lifecycle-policy).

• The "out of support" is only the Windows Server 2019 nodes in Azure AKS for March 2026: [Retirement] Windows Server 2019 (WS2019) will be retired on 1 March 2026 Azure/AKS#4091

That's why we're reconsidering the "importance" to drop it in LTS: we are not time bound for March 2026 neither constrained by "let's remove it ASAP because it is dangerous".

I'm all in favor of pushing user to move to Server 2022 (and beyond), but there are no emergency. I would rather stick to the usual "weekly -> wait for new LTS line" to avoid bad surprises out of the lifecycle.