feat: lease-aware scheduler gating, leader visibility, failover e2e

[stubbi]

Note

Stacked on #81 (feat/multi-replica-workload). This branch contains all of #81's commits; please review only the last 4 commits here (dd3cef6, a708e63, d615714, 9b3a860). The diff collapses to just those once #81 merges.

Motivation

The app is gaining lease-based scheduler leader election (paperclipai/paperclip#7995): every replica is a lease candidate, one wins, and the lease fails over automatically when the holder dies. The operator's existing gating (a pod-0 ordinal shell wrapper setting HEARTBEAT_SCHEDULER_ENABLED) has two gaps that this PR closes:

• No failover. If pod-0 is down, no scheduler runs until it comes back.
• StatefulSet only. Deployment pods (the new multi-replica workload profile from feat: Deployment workload profile, multi-replica preconditions, scale subresource #81) have no stable ordinals, so the wrapper cannot apply at all.

What changed (per commit)

dd3cef6 feat(operator): lease-aware scheduler gating modes (ordinal|lease|auto)

• New spec.heartbeat.schedulerGating enum: ordinal (default, unchanged behavior), lease (no env manipulation; delegates to the app's lease election), auto (resolves to ordinal today; flips to lease once the minimum supported app version ships lease leadership).
• The ordinal wrapper is now additionally guarded by the resolved mode and skipped for Deployment workloads.
• New advisory SchedulerGatingValid condition (excluded from the Ready aggregate) + Warning event for the Deployment+ordinal+replicas>1 combination; removed at replicas <= 1.
• Tests: wrapper presence matrix across mode/workload/replicas, mode-resolution unit tests, condition lifecycle envtest.

a708e63 feat(operator): scheduler leader visibility

While lease gating is active at replicas > 1, the operator polls each running server pod's unauthenticated /api/health for the scheduler block and surfaces the lease holder:

• pod label paperclip.inc/role=scheduler on the leader, web on the rest;
• status.schedulerLeader records the leader pod name (per-pod poll failures skip the pod; when every poll fails the previous value is kept to avoid thrashing);
• controller.kubernetes.io/pod-deletion-cost on the leader for Deployment workloads, so ReplicaSet scale-in prefers web replicas and avoids needless failovers;
• ordinal gating or replicas <= 1 strips all markers and clears the status field; active instances requeue every 30s.

The health probe is injectable, so envtest covers labeling, status, deletion-cost, no-thrash, and the clear path without a pod network.

d615714 test(e2e): image-gated scheduler lease failover scenario

2-replica Deployment-workload Instance against an in-cluster postgres:16 (database.mode: external), schedulerGating: lease: asserts 2 ready pods, exactly one scheduler-labeled pod matching status.schedulerLeader, then deletes the leader and asserts a different pod is labeled and recorded. Skipped by default -- it needs an app image containing lease leadership (unreleased), and runs only when an operator provides it:

PAPERCLIP_E2E_LEASE_IMAGE=<image-with-leases> make test-e2e

objectStorage is intentionally omitted in the scenario: MultiReplicaPreconditions reports an advisory False, the workload reconciles regardless, and the failover assertions touch no file state (keeps the suite free of a MinIO dependency).

9b3a860 docs: scheduler gating, version skew, migration, observability

README "Scheduler gating and failover" subsection under Scaling.

Version skew

Operator gating	App without leases	App with leases (>= the #7995 release)
ordinal (default)	pod-0 pinned, no failover	pod-0 pinned (the wrapper wins: only pod-0 is a lease candidate)
lease	ALL replicas run the scheduler -- unsafe, do not use	automatic failover

Migration recipe (order matters): upgrade the app image first, then set schedulerGating: lease, then optionally drop any manual HEARTBEAT_SCHEDULER_ENABLED pinning from spec.env.

Verification

• make test green (unit + envtest, controller coverage includes the new gating condition and leader-visibility paths)
• make lint 0 issues
• go vet ./test/... && go build ./test/... green; go test ./test/e2e/ -ginkgo.dry-run builds the spec tree (5 specs registered, including the new one)
• Skip gating confirmed: the failover context Skips in BeforeAll when PAPERCLIP_E2E_LEASE_IMAGE is unset, so a normal make test-e2e run is unaffected; the full kind suite was not run here
• The failover spec itself cannot pass until a lease-capable app image exists (by design)

Dependencies

• Stacked on feat: Deployment workload profile, multi-replica preconditions, scale subresource #81 -- must merge first.
• The e2e failover spec needs a released paperclip app image with lease leadership (feat: lease-based scheduler leader election with automatic failover paperclipai/paperclip#7995).

Mark ready for review after paperclip releases lease leadership and #81 merges.

🤖 Generated with Claude Code