| # | Risk | Mitigation |
|---|---|---|
| R1 | Controller crash / leader loss | replicas=2 + leader election; backstop retained; failure alerts |
| R2 | Window too short for all candidates | §3.2 throughput check warns; alert on sustained candidates |
| R3 | Surge capacity unavailable (AZ shortage / limits) | Pre-check headroom (§5.2); readyTimeout rollback; multi-AZ / multi-instance-type |
| R4 | Drain blocks on misconfigured PDB | terminationGracePeriod forces drain; PDB review is app owner's responsibility |
| R5 | Forgotten freeze during critical period | Manage declaratively (GitOps) rather than ad-hoc |
| R6 | Test clusters routinely turn over | Disable shutdown for a soak exceeding ageThreshold |
- R3 zonal caveat: surge is pinned to the candidate's AZ (§3.7). A same-AZ shortage cannot fall back to another zone — keep per-AZ headroom for zonal-PV NodePools.
::: tip Validation summary Validated (20+ scenarios): core surge, same-AZ zonal-PV rebind, rollback, limits gating, multi-pool confinement, PDB drain, do-not-disrupt markers, force-expiry detection, capacity-absorb, placeholder preemption, window-boundary, leader-change resume, forceful fallback, earliest-deadline ordering, operator opt-out, and a 12-hour tight-race soak.
Open: genuine same-AZ capacity shortage (ICE) driving rollback on real cloud (issue #109). :::
| Assumption | Status | Date |
|---|---|---|
Standalone NodeClaim is provisionable on Auto Mode |
Validated | 2026-05-29 |
| Placeholder-Pod surge completes make-before-break | Validated | 2026-06-22 |
| Same-AZ surge lets EBS re-attach (zonal-PV rebind) | Validated | 2026-06-22 |
readyTimeout miss rolls back cleanly |
Validated | 2026-06-22 |
NodePool limits exhaustion gates surge |
Validated | 2026-06-22 |
Required karpenter.sh/nodepool confines surge to pool |
Validated | 2026-06-22 |
Explicit NodeClaim deletion drains via voluntary path (PDBs) |
Validated | 2026-06-22 |
do-not-disrupt applied to both nodes, removed on completion |
Validated | 2026-06-22 |
Force-expiry mid-pending records expired (not success/failure) |
Validated | 2026-06-22 |
| Assumption | Status | Date |
|---|---|---|
| Capacity-absorb path (bin-pack onto spare, no new node) | Validated | 2026-06-23 |
| Leader-change resumes purely from annotations | Validated | 2026-06-23 |
| In-flight rotation completes past window boundary | Validated | 2026-06-23 |
| Placeholder is preemption victim; hostile preemption → rollback | Validated | 2026-06-23 |
do-not-disrupt honored against Drift |
Validated | 2026-06-23 |
| Assumption | Status | Date |
|---|---|---|
| Forceful fallback (12-node batch, graceful + surge-less mix) | Validated | 2026-07-04 |
| Earliest-deadline candidate ordering | Validated | 2026-07-04 |
Operator do-not-disrupt excludes from selection |
Validated | 2026-07-04 |
| Assumption | Status | Date |
|---|---|---|
| 12h tight-race soak: 71/71 graceful, 0 expired, 0 failure | Validated | 2026-07-15 |
| Forceful fallback fires deterministically for bounded claim | Validated | 2026-07-15 |
::: details Full validation evidence — click to expand
A NodeClaim with only nodeClassRef + requirements reached Ready (~30s, real EC2); admission accepted --dry-run=server; graceful finalizer-driven deletion confirmed. K8s 1.35, karpenter.sh/v1 (2026-05-29).
A low-priority placeholder induced a NodePool-owned surge NodeClaim that reached Ready (~30s) before the old was deleted; drain followed the voluntary path; workload rescheduled onto surge node; noderotation_completed_total{outcome="success"} incremented.
StatefulSet gp3 PVC in us-west-2a; matchNodeRequirements kept every surge node in us-west-2a; the same PV re-attached (not reprovisioned); sentinel data survived.
readyTimeout set below node-ready time → timeout → surge claim reaped, placeholder deleted, candidate retained + uncordoned, outcome="failure" incremented.
spec.limits.cpu with no headroom: eligible candidate not surged; logged insufficient limits headroom; cannot surge; in_progress stayed 0.
Second pool with same-AZ spare: surge stayed in the candidate's pool (karpenter.sh/nodepool=nrc-poc); other pool unchanged.
Blocking PDB (minAvailable=2, 2 replicas) stalled drain; relaxing to minAvailable=1 let migration complete one at a time.
Both old and surge nodes carried do-not-disrupt=true + do-not-disrupt-owned; unfreeze removed both on completion.
Froze pool + deleted pending candidate → state=expired, anchor cleared, no surge left behind, outcome="expired" incremented.
Young same-AZ spare with ~1970m free absorbed the 250m placeholder (no new NodeClaim induced); pool stayed at 2 claims throughout; outcome="success" incremented.
Mid-rotation leader killed; new replica took Lease and continued the same rotation from annotations — same surge-claim, same started-at, completed without restart.
Rotation started in-window; closing window did not abort it; second eligible candidate did not start (window_active=0).
Higher-priority Pod preempted placeholder; placeholder never preempts (preemptionPolicy=Never); with limits blocking re-provision, stayed Pending until readyTimeout → clean rollback.
Drifted node with do-not-disrupt=true was not replaced for >3 min; removing annotation triggered immediate drift-replace.
12 nodes, fixed 2h expireAfter, N=12 > K·C=2: first 6 gracefully, surplus 6 surge-less. rotation-mode=forceful-fallback while in flight; ForcefulFallback Warning Events; no placeholder for forceful candidates; noderotation_forceful_fallback_total climbed 0→6; PDB held throughout; zero expired.
12-node batch shared one creationTimestamp → ordering degraded to Name tiebreak: claims consumed exactly ascending (2rvd5 < 6ssql < dtkgz < ...).
Annotating a candidate's Node do-not-disrupt=true (no owned marker) dropped candidates gauge 4→3; removing restored it.
E=2h12m, leadTime=1h12m, 48 windows/day, 5-node pool. 71/71 rotations gracefully (~12m cadence). Min margin 68.3m, median 70.3m, max 71.2m. Zero expired, zero failure, forceful_fallback_total=0 (armed but never needed). Controller restartCount=0, 909 scrapes, contiguous seq. Full record: test/e2e/eks-automode/VALIDATION.md.
Separate single-node pool, released after aging past candidacy. Graceful surge no longer fit → surge-less branch: forceful_fallback_total 0→1; claim deleted 56s after release (10m04s ahead of deadline); no placeholder at any point; expired stayed 0.
:::
A genuine same-AZ capacity shortage (ICE) driving rollback — stood in for by a short readyTimeout (not deterministically inducible on demand). Tracked in issue #109.
- The standalone
NodeClaimresult de-risks the project but is not the surge mechanism (§3.3 — standalone nodes break NodePool accounting) - RBAC sufficiency and
karpenter.sh/v1CRD decode are implicitly exercised by all scenarios
- Holiday-aware scheduling — skip rotation if a window day falls on a holiday. v1 intentionally ignores holidays.
- Pre-pull image source — whether to use Karpenter NodeClass image-pulling or a dedicated Job (v2).
- Multi-cloud verification — AKS NAP, GKE testing before claiming compatibility beyond EKS Auto Mode.
::: tip Resolved
CRD-based policy migration and per-NodePool maintenance window — both delivered by the RotationPolicy CRD (issue #119, §5.4).
:::