Wait for pod template on resume to avoid leaking agent pods after restart by RanjitCS · Pull Request #2851 · jenkinsci/kubernetes-plugin

RanjitCS · 2026-07-10T10:55:31Z

If the controller restarts while a dynamic pod template agent is being provisioned, KubernetesLauncher.launch() fails on resume because the template isnt re-registered yet, leaving the pod orphaned

Testing done

Tested practically in an jenkins cluster
Added integration test

Submitter checklist

Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
Ensure that the pull request title represents the desired changelog entry
Please describe what you did
Link to relevant issues in GitHub or Jira
Link to relevant pull requests, esp. upstream and downstream changes
Ensure you have provided tests that demonstrate the feature works or the issue is fixed

…leaking agent pods after restart

jglick

I think this is solving some bug. But it does not seem to have anything to do with what was reported in #2512.

jglick · 2026-07-10T22:20:21Z

    private static final Logger LOGGER = Logger.getLogger(KubernetesLauncher.class.getName());

+    private static final long TEMPLATE_RESOLUTION_TIMEOUT_SECONDS =
+            SystemProperties.getLong(KubernetesLauncher.class.getName() + ".templateResolutionTimeoutSeconds", 60L);


Use getDuration please, and remove the time unit from the property name, and adjust docs to clarify that you pass a unit expression.

f4ffb03, i had to bump the baseline for getDuration

jglick · 2026-07-10T22:21:52Z

+     * re-registers the dynamic template asynchronously, and this launcher may run first. Rather than failing
+     * immediately (which would leak the pod), poll for the template for a short while. Falls back to
+     * {@link KubernetesSlave#getTemplate()} (which throws a descriptive exception) if it never becomes available.
+     * See <a href="https://github.com/jenkinsci/kubernetes-plugin/issues/2512">...</a>.


Suggested change

* See <a href="https://github.com/jenkinsci/kubernetes-plugin/issues/2512">...</a>.

Anyone interested can git blame.

jglick · 2026-07-10T22:23:23Z

+     * See <a href="https://github.com/jenkinsci/kubernetes-plugin/issues/2512">...</a>.
+     */
+    @NonNull
+    private static PodTemplate waitForTemplate(KubernetesSlave node) throws InterruptedException {


Should this not be waiting only for dynamic templates? A global template would already be available, or should be, so if it is not then we may as well fail immediately.

#2851 (comment)

jglick · 2026-07-10T22:24:04Z

+            LOGGER.log(
+                    FINE,


Suggested change

LOGGER.log(

FINE,

LOGGER.fine(

jglick · 2026-07-10T22:25:29Z

+     * launch and restores it shortly after. Without the fix the pod is orphaned and with it the build resumes on the
+     * same pod and the pod is cleaned up.
+     */
+    @Issue("JENKINS-67390")


Jira is dead. Refer to GH Issues

#2851 (comment)

jglick · 2026-07-10T22:25:47Z

    }

+    /**
+     * Reproduces <a href="https://github.com/jenkinsci/kubernetes-plugin/issues/2512">...</a> if, on resume,


Suggested change

* Reproduces <a href="https://github.com/jenkinsci/kubernetes-plugin/issues/2512">...</a> if, on resume,

* Reproduces a problem in which, on resume,

#2851 (comment)

jglick · 2026-07-10T22:27:15Z

+            cloud.removeDynamicTemplate(template);
+            // Restore it shortly after, mimicking PodTemplateStepExecution#onResume re-registering it late.
+            Timer.get().schedule(() -> cloud.addDynamicTemplate(template), 5, TimeUnit.SECONDS);


This is not very convincing. If there is no way to reliably reproduce the actual problem, I would rather not have a test at all.

as its a very narrow window, the initial test which i had was more faithful but was flaky without the fix (i.e 4/10 failures without the fix, fully passes with the fix), i will remove the test itself then if this is not good enough

jglick · 2026-07-10T22:27:34Z

+    containerTemplate(name: 'busybox', image: 'busybox', command: 'cat', ttyEnabled: true)
+]) {
+    node('orphan-race') {
+        container('busybox') {


Is the container step relevant in this context?

removed in f4ffb03

jglick · 2026-07-10T22:28:11Z

@@ -0,0 +1,15 @@
+podTemplate(label: 'orphan-race', yaml: '''


do not use label, long deprecated

removed in f4ffb03

jglick · 2026-07-10T22:35:08Z

Tested practically in an jenkins cluster

Details?

RanjitCS · 2026-07-13T07:51:33Z

I think this is solving some bug. But it does not seem to have anything to do with what was reported in #2512.

thinking it through, as i found the same symptom (suspended agent, pod never cleaned up) while trying to reproduce, i assumed it was the same fix, but i was wrong, i will continue to focus on #2512, but i believe this is also a bug that can happen

edit: i have spent some time trying to reproduce #2512, but i have been unsuccessfull, i believe #1137 has solved the "agent stays suspended, pod reconnects but Jenkins does nothing" problem. this pr however is still valid

RanjitCS · 2026-07-13T08:43:45Z

Tested practically in an jenkins cluster

Details?

create a Pipeline with an inline podTemplate {} (a dynamic template) whose pod stays unschedulable i used nodeSelector: disktype=special. This keeps it pending so the agent never connects before the restart, forcing launch() to run again on resume and hit the race
run it and wait for pod creation
restart the controller while the pod is still pending
on resume, make the pod schedulable, kubectl label node <node> disktype=special
notice that the pod is orphaned

again, this is extremely timing dependent but can be reproduced consistently without any fake pause injection at least 5/10 tries i did

RanjitCS added 7 commits July 10, 2026 16:23

[JENKINS-67390][ISSUE-2512] Wait for pod template on resume to avoid …

4ded1d8

…leaking agent pods after restart

Merge branch 'master' into JENKINS-67390

c624e88

fix up test

c8b3b6c

spotless

0d3fc5b

fix comments

45c00a3

trigger ci

03a883f

fix ci test

ceba5f4

RanjitCS marked this pull request as ready for review July 10, 2026 17:31

RanjitCS requested a review from a team as a code owner July 10, 2026 17:31

jglick added the bug Bug Fixes label Jul 10, 2026

jglick changed the title ~~[JENKINS-67390][ISSUE-2512] Wait for pod template on resume to avoid leaking agent pods after restart~~ Wait for pod template on resume to avoid leaking agent pods after restart Jul 10, 2026

jglick reviewed Jul 10, 2026

View reviewed changes

review comments

f4ffb03

	* Reproduces <a href="https://github.com/jenkinsci/kubernetes-plugin/issues/2512">...</a> if, on resume,
	* Reproduces a problem in which, on resume,

Uh oh!

Conversation

RanjitCS commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing done

Submitter checklist

Uh oh!

jglick left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RanjitCS Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RanjitCS Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jglick commented Jul 10, 2026

Uh oh!

RanjitCS commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RanjitCS commented Jul 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

RanjitCS commented Jul 10, 2026 •

edited

Loading

RanjitCS Jul 13, 2026 •

edited

Loading

RanjitCS Jul 13, 2026 •

edited

Loading

RanjitCS commented Jul 13, 2026 •

edited

Loading