Skip to content

fix(bootstrap): make bootstrap Job reconcile idempotent (no immutable-template churn) - #85

Merged
stubbi merged 1 commit into
mainfrom
fix/bootstrap-job-immutable-template
Jun 13, 2026
Merged

fix(bootstrap): make bootstrap Job reconcile idempotent (no immutable-template churn)#85
stubbi merged 1 commit into
mainfrom
fix/bootstrap-job-immutable-template

Conversation

@stubbi

@stubbi stubbi commented Jun 13, 2026

Copy link
Copy Markdown
Collaborator

Summary

The admin bootstrap Job (spec.auth.adminUser) was flaky: during live multi-replica verification its pod was killed ~1s after starting by the Job controller itself (SuccessfulDelete immediately after Started container bootstrap), then BackoffLimitExceeded, leaving bootstrapStatus stuck at bootstrap_pending (#83).

Root cause: a Kubernetes Job's pod template is immutable after creation. Two things made the Job controller churn/reap its own pod:

  1. The Job was built with explicit pod-template labels but no explicit spec.selector, so the Job controller could adopt a leftover/orphaned pod from a previous bootstrap Job of the same name (e.g. after a manual kubectl delete job + operator recreate) and then delete it.
  2. Any reconcile path that re-renders/patches spec.template would force delete/recreate of the running pod.

Changes

  • internal/resources/bootstrap.go
    • Set an explicit, unique spec.selector + manualSelector: true keyed to the Job name, plus a matching per-Job pod label (paperclip.ai/bootstrap-job). This Job's pods can never be adopted from a prior/other Job.
    • Stamp a content-hash annotation (paperclip.ai/bootstrap-spec-hash) over the operator-controlled inputs (image, script, admin email, password secret ref).
  • internal/controller/instance_controller.goreconcileBootstrapJob is now strictly create-if-absent:
    • If the Job exists and the hash matches → leave it entirely untouched (never patch spec.template).
    • If the hash differs → delete the stale Job (foreground propagation) and recreate on a later reconcile. Never an in-place template update.

Tests

Verification (local)

  • make test — green (controller + resources + registry + conformance suites)
  • make lint — 0 issues
  • hack/sync-chart-crds.sh --check and hack/check-helm-rbac-sync.sh — in sync (no CRD/RBAC changes)
  • gosec on changed packages — clean
  • Reconcile Guard: no new r.Update calls

Closes #83

🤖 Generated with Claude Code

…-template churn)

The admin bootstrap Job (spec.auth.adminUser) could have its running pod
killed ~1s after start by the Job controller itself (SuccessfulDelete
immediately after "Started container bootstrap"), then BackoffLimitExceeded,
leaving bootstrapStatus at bootstrap_pending (issue #83).

A Kubernetes Job's pod template is immutable after creation, so any reconcile
that re-renders or patches spec.template makes the Job controller churn and can
delete the active pod. Separately, the Job was built with explicit pod-template
labels but no explicit selector, so the Job controller could adopt a
leftover/orphaned pod from a previous bootstrap Job of the same name (e.g.
after a manual `kubectl delete job` + operator recreate) and then reap it.

This makes the bootstrap Job lifecycle deterministic and non-churning:

- BuildBootstrapJob now sets an explicit, unique spec.selector +
  manualSelector=true keyed to the Job name (plus a matching per-Job pod
  label), so this Job's pods can never be adopted from a prior Job.
- BuildBootstrapJob stamps a content-hash annotation over the
  operator-controlled inputs (image, script, admin email, password secret ref).
- reconcileBootstrapJob is strictly create-if-absent. If the Job exists and the
  hash matches, it is left entirely untouched (spec.template is never patched).
  Only when the hash differs is the stale Job deleted (foreground propagation)
  and recreated on a later reconcile -- never an illegal in-place template
  update.

Adds an envtest regression test that reconciles the bootstrap Job N times and
asserts the Job is neither updated (same resourceVersion) nor recreated (same
UID), plus a test that a config change triggers delete+recreate.

Closes #83

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@stubbi
stubbi force-pushed the fix/bootstrap-job-immutable-template branch from 2ac60b4 to b767618 Compare June 13, 2026 18:05
@stubbi
stubbi merged commit 3f1beac into main Jun 13, 2026
13 checks passed
@stubbi
stubbi deleted the fix/bootstrap-job-immutable-template branch June 13, 2026 18:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bootstrap Job (auth.adminUser) is flaky: job controller kills its own pod, backoff-limit exceeded

1 participant