Jenkins plugin installation dependency confliction issue

[Zhenye-Na]

Hello Jenkins community,

I would like to reach out for a recent issue happened in our Jenkins, especially plugin installation logic. I am here to reach out for best practices as well as caveats when working with Jenkins plugins if possible

Context

Jenkins version - 2.504.1
Jenkins plugin manager version - 2.13.0

---

Problem

We deployed our Jenkins on AWS via ECS Fargate. The controller / Jenkins sever failed to start up due to some plugin issues. To be more specific, we have the following errors

....
- Update required: Pipeline: Groovy (workflow-cps 4050.v8b_a_69b_587c39) to be updated to 4080.va_15b_44a_91525 or higher
- Update required: Folders Plugin (cloudbees-folder 6.999.v42253c105443) to be updated to 6.1012.v79a_86a_1ea_c1f or higher
- Update required: Folders Plugin (cloudbees-folder 6.999.v42253c105443) to be updated to 6.1012.v79a_86a_1ea_c1f or higher

On the first glance, looks like it is just plugin version is lower than expected. However, when we investigate further, more issues come in and we are looking for some clarifications and understanding

Mitigation 1

The below script/command line is used in the Dockerfile and will be used to produce an ECR image for Jenkins

jenkins-plugin-cli --plugin-file /usr/share/jenkins/ref/plugins.txt \ --jenkins-version=$(cat ${REF}/jenkins_version) \ --latest=false

plugins.txt is like

cloudbees-folder
workflow-cps

Notice the --latest flag is set to false explicitly. The original intention is that, we dont want aggressively update the plugins without verification. Instead, we just want minimum version of the plugin to work with the Jenkins server.

Since with the command, we failed to bring up Jenkins server. which means either

Question 1

Plugin manager does recognize --latest flag, which only download min version of plugins that work with the current Jenkins version. But that min version plugin does not satisfy its dependencies/consumers.

Could you help correct me if I am wrong on this ? If so, we would like to bring up a new feature request that not only keep the min version for Jenkins server, but also its dependencies/consumers.

Mitigation 2

We then switched to

jenkins-plugin-cli --plugin-file /usr/share/jenkins/ref/plugins.txt \ --jenkins-version=$(cat ${REF}/jenkins_version) \ --latest=true \ --latest-specified

with updated plugins.txt where latest is added

cloudbees-folder:latest
workflow-cps:latest

This time, we have the same error message

....
- Update required: Pipeline: Groovy (workflow-cps 4050.v8b_a_69b_587c39) to be updated to 4080.va_15b_44a_91525 or higher
- Update required: Folders Plugin (cloudbees-folder 6.999.v42253c105443) to be updated to 6.1012.v79a_86a_1ea_c1f or higher
- Update required: Folders Plugin (cloudbees-folder 6.999.v42253c105443) to be updated to 6.1012.v79a_86a_1ea_c1f or higher

However, the installed version 6.999.v42253c105443 for example for cloudbees is not the latest which could be found in https://plugins.jenkins.io/cloudbees-folder/releases/, which is 6.1012.v79a_86a_1ea_c1f

Question 2

Given we have --latest=true --latest-specified flag but we did not download the latest version of plugins. Could you help us understand what could be wrong ? Maybe we missed the configuration somehow ?

---

Next step

Since we have multiple Jenkins sever, this issue only happened to one of those. We are wondering if we updated our plugins.txt to

workflow-cps:4080.va_15b_44a_91525
cloudbees-folder:6.1012.v79a_86a_1ea_c1f

and still use the same step to download plugins like

jenkins-plugin-cli --plugin-file /usr/share/jenkins/ref/plugins.txt \ --jenkins-version=$(cat ${REF}/jenkins_version) \ --latest=true \ --latest-specified

Will we guarantee that we are downloading the correct plugin ?

[timja]

the safest way is to specify all plugins at a specific version and then use something like renovate or the --available-updates on the jenkins-plugin-cli to update your plugins via a pull request.

Ideally you should be building your plugins into a docker image and ensuring its starting before publishing a new image so this sort of thing shouldn't end up with a Jenkins instance not starting

[Zhenye-Na]

Hello @timja , thank you for your insights

the safest way is to specify all plugins at a specific version

I believe you more recommend we do the following for all plugins that need to be installed

workflow-cps:4080.va_15b_44a_91525
cloudbees-folder:6.1012.v79a_86a_1ea_c1f

together with --available-updates flag included. Just would like to confirm from you, this argument will check if there is a newer version than provided version AND it is available in the update center. Then it is installed ?

--available-updates: (optional) Show if any requested plugins have newer versions available. If a Jenkins version-specific update center is available, the latest plugin version will be determined based on that update center's data.

Would you mind helping me understand what is update center ? Is it like a maven central repo that registered all available plugins ? But in some scenarios it is not duplicated every where ?

---

Ideally you should be building your plugins into a docker image and ensuring its starting before publishing a new image so this sort of thing shouldn't end up with a Jenkins instance not starting

I cannot agree more. We do currently build plugins into a docker image, but there is some legacy configuration that somehow does not persist plugin version in each docker image. If this is available, we can simply revert the latest tag on the docker image for a fast rollback. For this topic, I am wondering if you have any suggestion / best practices - just to save me a few hours randomly searching with Google.

Thank you.

Cheers!

[timja]

available-updates is something you can run separately to update the file with the plugins versions file

see e.g. https://github.com/jenkinsci/docker#usage-1 for building plugins in.