Bootstrap Job never short-circuits: /api/health/details 404s, so already-bootstrapped instances fail forever in a ~65min loop

[Renard-Nicolas]

Summary

On an already-bootstrapped Instance, the operator-generated bootstrap Job can never succeed. Its idempotency pre-check queries a route that does not exist on the Paperclip server, so the "already bootstrapped, nothing to do" branch is unreachable, and the Job always falls through to auth bootstrap-ceo, which correctly refuses.

Combined with ttlSecondsAfterFinished: 3600, this produces a permanent failure loop: the failed Job is GC'd after an hour, the next reconcile recreates it, it fails again. 22 failed Jobs in 24h, one every ~65 minutes, each one firing KubeJobFailed in Prometheus.

Root cause

internal/resources/bootstrap.go builds the Job script with:

# Use /api/health/details (authenticated) which includes bootstrapStatus;
# the plain /api/health endpoint does not return this field.
HEALTH=$(curl -sS -c "$COOKIE_JAR" -b "$COOKIE_JAR" "$SERVER_URL/api/health/details" 2>/dev/null) || true
if echo "$HEALTH" | grep -q '"bootstrapStatus":"ready"'; then
  echo "Instance already bootstrapped. Nothing to do."
  ...
  exit 0
fi

That comment is inverted with respect to the shipping server. On ghcr.io/paperclipai/paperclip:sha-e55d702:

GET /api/health          -> HTTP 200
GET /api/health/details  -> HTTP 404  {"error":"API route not found"}

and bootstrapStatus is returned by the plain /api/health:

{"status":"ok","deploymentMode":"authenticated","deploymentExposure":"private",
 "bootstrapStatus":"ready","bootstrapInviteActive":false,
 "databaseBackup":{"enabled":true,"status":"ok","warnings":[]}}

So $HEALTH is the 404 error body, the grep never matches, and the script proceeds to step 3.

This looks like it was introduced by 2011328 ("use /api/health/details for bootstrap status check") — either that route never shipped, or it was renamed/removed on the server side without the operator following.

Observed behaviour

Full log of one Job pod, repeated identically for all 4 attempts (backoffLimit: 3):

Waiting for Paperclip server...
Server is ready (HTTP 200).
Creating admin account...
Sign-up returned HTTP 422, trying sign-in...
Signed in as existing admin.
Generating bootstrap invite...
> node cli/node_modules/tsx/dist/cli.mjs cli/src/index.ts "auth" "bootstrap-ceo" ...
●  Instance already has an admin user. Use --force to generate a new bootstrap invite.
Could not extract invite token.

Note Instance already bootstrapped. Nothing to do. never prints — that is the unreachable branch. bootstrap-ceo behaves correctly here; the bug is that it should never have been reached.

Job then goes:

BackoffLimitExceeded  Job has reached the specified backoff limit

Reproduction

1. Deploy an Instance in authenticated mode with spec.auth.adminUser set and let it bootstrap successfully.
2. Leave spec.auth.adminUser in place.
3. Wait. Every ~65 min (TTL 3600s + reconcile) a new <instance>-bootstrap Job is created and fails with BackoffLimitExceeded.

Confirm the cause directly:

kubectl exec -n <ns> deploy/<paperclip> -- curl -s -o /dev/null -w '%{http_code}\n' localhost:3100/api/health/details
# 404

Impact

• Any long-lived, bootstrapped Instance that keeps spec.auth.adminUser set fires KubeJobFailed (or equivalent) forever, on a ~1h flap cycle. This is very noisy for anyone alerting on failed Jobs.
• The failure is silent about its real cause — the log points at bootstrap-ceo refusing, which reads like correct behaviour rather than a broken pre-check.
• spec.auth.adminUser is documented as first-deploy-only, but nothing in the API surface says "unset this after bootstrap", and unsetting it is the only way out today.

Suggested fixes

1. Point the pre-check at /api/health (it already carries bootstrapStatus), or probe /api/health/details and fall back to /api/health on non-2xx, so the check works across server versions.
2. Treat bootstrap-ceo exiting with "Instance already has an admin user" as success, not failure — it is a correct idempotent outcome.
3. Optionally stop re-rendering the Job once bootstrapStatus is ready, or record bootstrap completion in the Instance status so reconciles skip it.

Any one of 1 or 2 fixes the loop; 1+2 together makes it robust.

Environment

• operator ghcr.io/paperclipinc/paperclip-operator:v0.18.1
• server ghcr.io/paperclipai/paperclip:sha-e55d702
• mode authenticated, exposure private, external Postgres
• Kubernetes v1.36.1

Workaround

Remove spec.auth.adminUser from the Instance after the initial bootstrap. The operator then stops rendering the Job. This does not affect the existing admin user (it lives in Postgres; the server StatefulSet reads no ADMIN_* env). Verified: Instance stays Running, server pod untouched, no Job recreated across multiple reconciles.

Possibly related but distinct from #83 (that was the Job controller reaping its own pod mid-reconcile).