Skip to content

Bootstrap Job (auth.adminUser) is flaky: job controller kills its own pod, backoff-limit exceeded #83

Description

@stubbi

Symptom

During a live multi-replica verification (Scaleway cluster, Instance with auth.adminUser set), the operator's bootstrap Job failed twice in a row:

  1. First run: pod started while the server was still crash-looping (separate issue), job ended Failed 0/1.
  2. After kubectl delete job pc-bootstrap (operator recreated it): the new pod was killed ~1s after start by the Job controller itself (SuccessfulDelete — Deleted pod: pc-bootstrap-66thw immediately after Started container bootstrap), then BackoffLimitExceeded.

The pod deletion right after start suggests the operator updated/recreated the Job spec mid-run (CreateOrUpdate on an immutable-template resource?) or a labels/ownership churn made the Job controller reap its own pod.

Impact

spec.auth.adminUser cannot be relied on to seed the admin; bootstrapStatus stays bootstrap_pending. Workaround used live: plain HTTP POST /api/auth/sign-up/email + POST /api/bootstrap/claim against the service URL — deterministic and instant.

Suggested direction

  • Reproduce with an authenticated-mode Instance and watch the Job's pod lifecycle during operator reconciles.
  • Check reconcileBootstrapJob for spec churn against an existing Job (Jobs have immutable pod templates — CreateOrUpdate that touches the template forces delete/recreate patterns).
  • Consider replacing the curl-script Job with the seed-instance-admin init-container path or the HTTP claim flow, both of which proved robust.

Found during the multi-replica verification of #81/#82.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions