fix(bootstrap): make bootstrap Job reconcile idempotent (no immutable-template churn) - #85
Merged
Merged
Conversation
…-template churn) The admin bootstrap Job (spec.auth.adminUser) could have its running pod killed ~1s after start by the Job controller itself (SuccessfulDelete immediately after "Started container bootstrap"), then BackoffLimitExceeded, leaving bootstrapStatus at bootstrap_pending (issue #83). A Kubernetes Job's pod template is immutable after creation, so any reconcile that re-renders or patches spec.template makes the Job controller churn and can delete the active pod. Separately, the Job was built with explicit pod-template labels but no explicit selector, so the Job controller could adopt a leftover/orphaned pod from a previous bootstrap Job of the same name (e.g. after a manual `kubectl delete job` + operator recreate) and then reap it. This makes the bootstrap Job lifecycle deterministic and non-churning: - BuildBootstrapJob now sets an explicit, unique spec.selector + manualSelector=true keyed to the Job name (plus a matching per-Job pod label), so this Job's pods can never be adopted from a prior Job. - BuildBootstrapJob stamps a content-hash annotation over the operator-controlled inputs (image, script, admin email, password secret ref). - reconcileBootstrapJob is strictly create-if-absent. If the Job exists and the hash matches, it is left entirely untouched (spec.template is never patched). Only when the hash differs is the stale Job deleted (foreground propagation) and recreated on a later reconcile -- never an illegal in-place template update. Adds an envtest regression test that reconciles the bootstrap Job N times and asserts the Job is neither updated (same resourceVersion) nor recreated (same UID), plus a test that a config change triggers delete+recreate. Closes #83 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
stubbi
force-pushed
the
fix/bootstrap-job-immutable-template
branch
from
June 13, 2026 18:05
2ac60b4 to
b767618
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The admin bootstrap Job (
spec.auth.adminUser) was flaky: during live multi-replica verification its pod was killed ~1s after starting by the Job controller itself (SuccessfulDeleteimmediately afterStarted container bootstrap), thenBackoffLimitExceeded, leavingbootstrapStatusstuck atbootstrap_pending(#83).Root cause: a Kubernetes Job's pod template is immutable after creation. Two things made the Job controller churn/reap its own pod:
spec.selector, so the Job controller could adopt a leftover/orphaned pod from a previous bootstrap Job of the same name (e.g. after a manualkubectl delete job+ operator recreate) and then delete it.spec.templatewould force delete/recreate of the running pod.Changes
internal/resources/bootstrap.gospec.selector+manualSelector: truekeyed to the Job name, plus a matching per-Job pod label (paperclip.ai/bootstrap-job). This Job's pods can never be adopted from a prior/other Job.paperclip.ai/bootstrap-spec-hash) over the operator-controlled inputs (image, script, admin email, password secret ref).internal/controller/instance_controller.go—reconcileBootstrapJobis now strictly create-if-absent:spec.template).Tests
instance_controller_test.gothat reconciles the bootstrap Job N times and asserts the Job is neither updated (sameresourceVersion) nor recreated (sameUID) — the direct regression for Bootstrap Job (auth.adminUser) is flaky: job controller kills its own pod, backoff-limit exceeded #83. It also asserts the explicit unique selector /manualSelector.Verification (local)
make test— green (controller + resources + registry + conformance suites)make lint— 0 issueshack/sync-chart-crds.sh --checkandhack/check-helm-rbac-sync.sh— in sync (no CRD/RBAC changes)r.UpdatecallsCloses #83
🤖 Generated with Claude Code