idleMinutes config leads to limit counter leak with ephemeral templates (and restart before idle timeout)

[apuig]

Jenkins and plugins versions report

Environment

Jenkins: 2.528.3.35200
OS: Linux - 6.14.0-37-generic
Java: 21.0.9 - Red Hat, Inc. (OpenJDK 64-Bit Server VM)
---
kubernetes:4392.v19cea_fdb_5913
kubernetes-client-api:7.3.1-256.v788a_0b_787114
kubernetes-credentials:206.vde31a_b_0f71a_c

What Operating System are you using (both controller, and any agents involved in the problem)?

OS: Linux - 6.14.0-37-generic

Reproduction steps

1. Kubernetes cloud with containerCap: 5

2. Create pipeline using podTemplate with idleMinutes: 5

podTemplate(
    cloud: 'k8',    
    idleMinutes: 5,
    containers: [
        containerTemplate(
            name: 'jnlp', image: 'jenkins/inbound-agent:latest-jdk17'
        )
    ]
) {
    node(POD_LABEL) {
        stage('Test Job') {            
            sleep 5
        }
    }
}

3. Run pipeline 5 times, wait for each to complete

4. Restart Jenkins during idleMinutes timeout (before agents are deleted)

5. Run the pipeline again after restart

Expected Results

• A new agent can be provisioned

• Agents are always removed after an idle timeout, reducing the current total and making room for new agents to be created.

Actual Results

• The new agent cannot be provisioned because current limit is reached

• After Jenkins restarts:

1. 5 nodes (jobname-uuid) exists in manage/computer/
2. limit counter is 0
3. on first pipeline run, limit counter is updated to 5
4. after idle timeout, the nodes are removed, but the limit counter is not decremented
5. new pipeline builds cannot run because the limit is set to 5, even when no computer nodes are available

Anything else?

Possible root cause: agents lose their transient template references during reload, which causes the reaper cleanup process to skip node unregistration

There may be other ways to reproduce this issue besides a manual restart. Other plugins or processes could also serialize and restore agents, resulting in the same problem.

Are you interested in contributing a fix?

No response

[Abhijeet212004]

Hi @apuig , I'd like to work on this issue. I recently submitted PR #2785 which fixes a similar cleanup bug in the Reaper, so I'm familiar with the provisioning and cleanup logic in this plugin.

I've reviewed the code you linked and I believe the issue is that PodTemplate references are transient and become null after Jenkins reload. The NodeListener.onDeleted() checks for template != null before unregistering, which causes the leak.

My proposed approach:

• Store podTemplateId and numExecutors as persistent fields in KubernetesSlave
• Modify NodeListener.onDeleted() to use these stored values for cleanup even when template is null
• Add comprehensive test coverage for the restart scenario

I'll start investigating and submit a PR with the fix. Thanks!

[Abhijeet212004]

Just submitted PR #2787 for this.

Basically the problem is that ephemeral templates use a transient field that becomes null after restart, so the cleanup code's if (template != null) check fails and counters never get decremented.

Fixed it by using cloudName and templateId instead - those fields stick around after restart.

Let me know what you think!