Skip to content

05-implementation.md

Latest commit

 

History

History
505 lines (417 loc) · 29.8 KB

05-implementation.md

File metadata and controls

505 lines (417 loc) · 29.8 KB
Raw

5. Implementation

5.1 Architecture

::: tip What this section defines The controller is a controller-runtime manager (Deployment, replicas=2, leader election) that reconciles per NodePool, resolving each pool's governing RotationPolicy on every pass. :::

flowchart TD
    subgraph cluster["Cluster (Karpenter v1+)"]
        subgraph ns["Namespace: node-rotation-system (configurable)"]
            ctrl["node-rotation-controller (Deployment)<br/>controller-runtime manager, replicas=2 + leader election (1 active)<br/>NodePool reconciler; watches NodeClaim / Pod / Node + RotationPolicy<br/>1-min self-requeue, /metrics endpoint"]
        end
        rp["RotationPolicy<br/>noderotation.io/v1alpha1, cluster-scoped<br/>selects NodePools via nodePoolSelector<br/>carries maintenanceWindows / minRotationChances / surge"]
        nc["NodeClaims (karpenter.sh/v1)<br/>nc-aaa 15d (old) / nc-bbb 14d (old) / nc-ccc 08d (surge)"]
    end
    ctrl -->|"resolve governing policy per pool"| rp
    ctrl -->|"watch / create placeholder / delete old NodeClaim"| nc
Loading

Policy vs state separation

  • Policy = the RotationPolicy spec (desired configuration an operator authors)
  • State = annotations on NodeClaim/NodePool + transient Node/placeholder markers (§5.3)
  • The CRD never carries authoritative runtime state — its status is observational only

Startup preflight

Before reconciling, the controller fails fast if:

  • The cluster does not serve karpenter.sh/v1 with nodeclaims/nodepools resources
  • RBAC cannot read them

The compatibility contract is the karpenter.sh/v1 group/version, independent of the managed Karpenter minor (EKS Auto Mode does not expose it). A successful decode of the v1 types confirms wire-compatible schema. Per-field CRD introspection is not attempted.

5.2 Reconcile Loop

::: tip What this section defines Each Reconcile call performs exactly one non-blocking step and returns a Requeue. No blocking waits — all state is read from annotations and survives restarts. :::

The reconciler is keyed on NodePool and watches:

  • NodeClaim (mapped to owning NodePool)
  • Placeholder Pod reaching Running
  • Surge host Node reaching Ready

A periodic self-requeue remains the backstop for window edges, freeze releases, drain progress, and force-expiry.

Decision flow

flowchart TD
    entry(["Reconcile (NodePool)"]) --> q1{"active-rotation<br/>anchor set?"}
    q1 -->|yes| adv["advance(): drive in-flight<br/>rotation one step (§5.3)"]
    q1 -->|no| q1b{"spec.replicas set?<br/>(static capacity)"}
    q1b -->|yes| stat["warn StaticNodePool;<br/>Requeue (1m)"]
    q1b -->|no| q2{"start gates pass?<br/>in window, not frozen,<br/>cooldown + failure-pause elapsed"}
    q2 -->|no| rq["Requeue (1m)"]
    q2 -->|yes| pick["pick earliest-deadline eligible candidate"]
    pick --> q3{"candidate<br/>found?"}
    q3 -->|no| rq
    q3 -->|yes| q4{"surge_headroom?<br/>clamped footprint<br/>vs spec.limits budget"}
    q4 -->|no| warn["warn: insufficient limits;<br/>Requeue (1m)"]
    q4 -->|yes| anchor["write active-rotation anchor<br/>(conflict-checked, only-if-absent)"]
    anchor --> adv
Loading

Static capacity gate (step 1a)

A NodePool with spec.replicas set can never complete a surge (§3.3), so no rotation is started for it: the pass warns once (StaticNodePool, §4.3) and requeues.

The gate sits after the in-flight advance() and before every start gate, so an anchor written before the gate existed (by an earlier controller version — Karpenter itself rejects adding spec.replicas to a running NodePool) still drives that rotation to completion instead of stranding a cordoned node and a placeholder. advance()'s failed-retry branch is a new attempt and is closed on a static pool separately, so it releases the anchor rather than retrying the doomed attempt once per escalated backoff.

Start gates (step 2)

All of the following must pass before a new rotation can begin:

  • in_window(now) — maintenance window open
  • not frozen(np) — no freeze annotation
  • since_last_rotation(np) >= cooldownAfter — gate A: post-success settle
  • since_last_failure(np) >= failurePause — gate B: post-failure pause (§4.4, ADR-0004)

Candidate selection (step 3)

pick_earliest_deadline_eligible selects claims with:

  • No deletionTimestamp
  • state empty (fresh) or failed past escalated backoff (retryBackoff · 2^(retry-count − 1), capped 8×)
  • pending/draining never re-selected; expired is terminal

Anchor semantics

The active-rotation anchor is:

  • Written before any other side effect at start
  • Cleared last at completion/failure
  • Conflict-checked, only-if-absent write (optimistic concurrency)
  • Ticks and NodeClaim events can race on the same NodePool — the precondition makes the race harmless

Completion outcome

Decided by the NodePool-side active-rotation-state mirror:

  • draining present → success (cooldown consumed)
  • draining absent → expired (alert, no cooldown)

Force-expiry detection

Caught on two paths:

  • Early: deletionTimestamp appearing while still pending — checked first, before everything else
  • Late: old NodeClaim disappearing with no draining mirror

The early path also writes state=expired before releasing the anchor (prevents livelock under Auto Mode's tGP = 24h).

Stuck drain

A drain exceeding tGP + buffer raises noderotation_drain_stuck but keeps the serial gate held — a rotation in draining cannot be rolled back (the delete already happened), and releasing the gate would violate maxUnavailable = 1.

Cooldown anchor

last-rotation-at lives on the NodePool (not the deleted old NodeClaim). The pause is durable across the completion boundary and leader changes.

::: details Full pseudocode — click to expand

Reconcile(req):
  if req is Tick:
      for np in in_scope_nodepools():
          reconcile_nodepool(np)
      return Requeue(1m)
  return reconcile_nodepool(nodepool(req.obj))

reconcile_nodepool(np):
  # ── 1. Drive in-flight rotation first (serial: at most one per NodePool)
  if name := np[active-rotation]:
      return advance(np, name)

  # ── 1a. Static capacity gate (§3.3): surge cannot serve a fixed-replica pool.
  #        After advance(), so an in-flight rotation still completes.
  if np.spec.replicas is set:
      warn_once(np, StaticNodePool)
      return Requeue(1m)

  # ── 2. Start gates
  start_gates(np) :=
      in_window(now) and not frozen(np)
      and since_last_rotation(np) >= cooldownAfter   # gate A
      and since_last_failure(np)  >= failurePause    # gate B
  if not start_gates(np): return Requeue(1m)

  # ── 3. Pick candidate, check headroom, anchor
  cand := pick_earliest_deadline_eligible(np)
  if cand == nil: return Requeue(1m)
  surgeless := forceful_fallback(np, cand)
  if not surgeless and not surge_headroom(np, cand):
      warn("insufficient limits headroom"); return Requeue(1m)
  annotate(np, active-rotation=cand.name)    # conflict-checked, only-if-absent
  if surgeless:
      annotate(np, rotation-mode=forceful-fallback,
               active-rotation-state=draining, draining-at=now)
      annotate(cand, state=draining)
      emit_metrics(forceful_fallback); event
      delete(cand)
      return Requeue(30s)
  return advance(np, cand.name)

advance(np, name):
  cand := nodeclaim(name)
  if cand == nil:                            # old NodeClaim finalized
      delete(placeholder(name))
      for node in nodes_with(surge-for=name):
          unfreeze(node)
      # ONE conflict-checked write, only-if active-rotation == name. It reads the
      # outcome from the same fresh copy it is validated against, stamps
      # last-rotation-at when that copy says draining, clears the anchor, and
      # reports whether THIS pass released it. A pass holding a stale cached np
      # loses the race and emits nothing (§5.2).
      won, rotated := release_anchor(np, name)
      if not won:                            # an earlier pass already completed it
          return Requeue(1m)
      if rotated:
          emit_metrics(success, duration)
      else:
          emit_metrics(expired); alert
      return Requeue(1m)

  switch cand.state:
  case (none) | pending:
      if cand.deletionTimestamp != nil:      # force-expiry caught
          # ONE conflict-checked write, only-if the claim still holds THIS
          # handler's pre-state, reporting what it did. It runs BEFORE the
          # cleanup: a pass that does not own the transition must not unfreeze
          # the surge node a live drain still depends on (§5.2).
          out := mark_expired(cand, from=[none, pending],
                              clear=[started-at, surge-claim])
          if out in {gone, raced}:           # nothing written; this pass owns nothing
              return Requeue(30s)            # gone ⇒ release_anchor counts the abort
          # announce BEFORE the fallible cleanup: a cleanup error sends the next
          # reconcile to the `expired` handler, which repairs cleanup and never
          # emits, so an emission behind it would be lost, not deferred (§5.2)
          if out == claimed: emit_metrics(expired); alert
          delete(placeholder(name))
          for node in nodes_with(surge-for=name): unfreeze(node)
          clear(np, anchor)
          return Requeue(1m)
      # only from the states advance() dispatches here on: a `pending` view of a
      # claim whose durable state has moved past it must not undo the rollback —
      # re-stamping started-at would restart the readyTimeout deadline (§5.2)
      wrote := annotate_if(cand, from=[none, pending],
                           state=pending, once(started-at=now))
      if not wrote: return Requeue(30s)   # this pass owns nothing
      if elapsed(cand.started-at) > readyTimeout:
          reap_surge_claim(cand[surge-claim])
          delete(placeholder(name))
          for node in nodes_with(surge-for=name): unfreeze(node)
          wrote := annotate(cand, state=failed, failed-at=now, retry-count+=1,
                            clear=[started-at, surge-claim])
          if not wrote:                      # the claim finalized away mid-rollback
              return Requeue(30s)            # a force-expiry, not a failed attempt
          # the alert reports the retry-count this write produced, never the
          # caller's cached copy of it
          emit_metrics(failure); alert
          annotate(np, last-failure-at=now, clear=anchor)
          return Requeue(1m)
      freeze(cand.node, surge-for=name)
      cordon(cand.node)
      if c := induced_claim(name):
          annotate(cand, surge-claim=c.name)
      if frozen(np): return Requeue(1m)      # hold escalation
      if placeholder(name) is missing:
          create_placeholder(np, cand)
          return Requeue(30s)
      if surge_ready(cand):
          host := placeholder_node(name)
          freeze(host, surge-for=name)
          annotate(np, active-rotation-state=draining, draining-at=now,
                   surge-wait=now − cand.started-at)
          annotate(cand, state=draining)
          delete(cand)
          return Requeue(30s)
      return Requeue(30s)

  case draining:
      annotate(np, active-rotation-state=draining)
      if cand.deletionTimestamp == nil:      # crash recovery
          delete(cand)
          return Requeue(30s)
      if elapsed(cand.deletionTimestamp) > drain_bound(np):
          alert(stuck_drain)
      return Requeue(30s)

  case failed:
      if cand.deletionTimestamp != nil:
          out := mark_expired(cand, from=[failed])   # same conditional write
          if out in {gone, raced}: return Requeue(30s)
          if out == claimed: emit_metrics(expired); alert
          clear(np, anchor)
          return Requeue(1m)
      # A retry is a NEW attempt: it must also clear the step-1a static gate,
      # which this path sits above (the anchor entered advance() first).
      if start_gates(np) and np.spec.replicas is unset
         and elapsed(cand.failed-at) >= escalated_backoff(cand)
         and surge_headroom(np, cand):
          # only from failed. The same guard bounds the re-entry below: advance()
          # re-reads through the cache, and a read still lagging this write would
          # dispatch straight back here with every gate open (§5.2)
          wrote := annotate_if(cand, from=[failed], state=pending)
          if not wrote: return Requeue(30s)
          return advance(np, name)
      annotate(np, last-failure-at=max(np[last-failure-at], cand.failed-at),
               clear=anchor)
      return Requeue(1m)

  case expired:                              # terminal cleanup
      delete(placeholder(name))
      for node in nodes_with(surge-for=name): unfreeze(node)
      clear(np, anchor)
      return Requeue(1m)

:::

Idempotent recovery

Each state handler re-asserts its phase's desired state rather than performing one-shot actions:

  • pending re-asserts freeze, cordon, placeholder existence on every pass
  • draining re-issues idempotent delete if deletionTimestamp is missing (crash between state write and delete)
  • completion re-runs its cleanup but claims the rotation with a conditional write: the anchor's release and the success/expired outcome are both decided from the fresh read that write is validated against, so a pass that arrives on a cached NodePool whose anchor was already released does the idempotent cleanup and emits nothing
  • the four writes a cache-lagged dispatch can reach claim their transition, accepting only the states their handler is dispatched on — the two entries into expired, pending's entry assertion, and the failed → pending retry. A pass arriving on a cached claim already written terminal cleans up, releases the anchor and emits nothing; a pass whose claim has moved on to any other state writes nothing at all, touches none of the rotation's runtime objects, and leaves it to the handler that owns it
  • the reconcile's remaining claim-state writes stay unconditional, and are safe structurally rather than by veto — though not all by the same structure. The two the pending handler makes (pending → draining, pending → failed) follow its own guarded entry. The forceful fallback is started directly from candidate selection, never through that handler, so what protects its draining write is the only-if-absent NodePool anchor it has just won. All three record work the owning pass performed, and all three move the claim forward. The §5.3 startup sweep's write sits outside this dispatch altogether; it is conditional too, but on the predicate that selected the claim rather than on a handler's pre-state
  • that guard is what stops a lagging pending view from undoing a rollback — restoring pending, re-stamping started-at and so restarting the readyTimeout deadline while retry-count keeps the value the escalation was based on — and what bounds the retry branch's re-entry into the dispatcher, whose own cached read can still lag the write it has just made

Observability skews (accepted in v1)

  • Mirror-to-delete gap: a crash there followed by force-expiry records success (surge was reserved — practical outcome matches)
  • Metric emission (completion): emitted after the anchor-releasing write and only by the pass that performed it, so the counter, the histogram, the completion line and the Event fire once per released anchor. A crash between the write and the emission drops it (at-most-once)
  • Metric emission (claim-scoped): both transitions into expired — abortPendingExpiry and advanceFailed's deletion branch — claim the transition with a conditional NodeClaim write that accepts only the dispatching handler's own pre-state, so expired is announced once by the pass that made it. That matches advanceExpired, which never re-announces a claim already terminal
  • Announcement follows the write, never the attempt: the outcome of a conditional claim write is produced by the write loop itself and reset per attempt, so a first attempt that conflicts and a retry that finds the claim finalized away report gone, not success. A claim that vanishes before a terminal write is left anchored and its outcome falls to completion (expired, no cooldown) — this covers the failure rollback too, which announces an attempt and stamps the failure pause only when the write that records it landed, and reports the retry count that write produced
  • Emission sits immediately after the write, ahead of the cleanup: the cleanup is fallible, and an error there hands the next reconcile to advanceExpired, which repairs it and deliberately never emits — so an emission placed behind the cleanup would be dropped by an ordinary transient API error rather than retried. The remaining loss window is the irreducible one the completion path already accepts: a controller that dies between the write and the emission (at-most-once)

5.3 State Model

::: tip What this section defines All state lives on Kubernetes objects — no external datastore. The NodePool's active-rotation anchor records which rotation is in flight; the old NodeClaim's state records where it is. :::

Annotation reference

Key Target Value Purpose
active-rotation NodePool NodeClaim name Durable anchor + serial gate
active-rotation-state NodePool draining Phase mirror for completion outcome
draining-at NodePool RFC3339 Drain-duration anchor (§4.2)
surge-wait NodePool Go duration Surge-phase duration for completion log
rotation-mode NodePool forceful-fallback Surge-less path marker
state Old NodeClaim pending/draining/failed/expired Progress state
started-at Old NodeClaim RFC3339 readyTimeout deadline
failed-at Old NodeClaim RFC3339 Backoff anchor
retry-count Old NodeClaim integer Escalates backoff
surge-claim Old NodeClaim NodeClaim name Induced surge identification
surge-for Pod + frozen nodes NodeClaim name Rotation pairing
do-not-disrupt Old + surge nodes true Block voluntary disruption
do-not-disrupt-owned Old + surge nodes true Controller ownership marker
cordoned Old node true Controller's cordon marker
last-failure-at NodePool RFC3339 Inter-attempt pause anchor
freeze NodePool RFC3339 Suppresses rotation until time
last-rotation-at NodePool RFC3339 cooldownAfter gate anchor

All keys use the noderotation.io/ prefix except karpenter.sh/do-not-disrupt.

::: details Annotation details — click to expand

  • active-rotation: written before any side effect, cleared last. Outlives the old NodeClaim (which is deleted on success). Also the serial gate for maxUnavailable = 1
  • active-rotation-state: written immediately before delete(cand). Absence = rotation never left pending. Read by completion handler after old NodeClaim is gone
  • draining-at: write-once at pending → draining. The old NodeClaim's deletionTimestamp is gone by completion — needs this anchor
  • surge-wait: write-once at pending → draining. The old NodeClaim (started-at carrier) is deleted at that transition
  • rotation-mode: stamped on anchor at forceful-fallback start. Absent = default surge. Cleared with anchor on every end path
  • state: expired is terminal — blocks re-selection while the claim finalizes under the forceful drain
  • started-at: write-once per attempt. Cleared by the failed write (single update with state=failed). Re-stamped on retry
  • surge-claim: persisted as soon as placeholder's bind target (spec.nodeName) is observable. Cleared with the failed write
  • surge-for: on frozen nodes, attributes freeze to this rotation. On the Pod, pairs it for discovery
  • do-not-disrupt-owned: set only when the controller actually applies do-not-disrupt. An operator's pre-existing annotation (no marker) is never touched
  • cordoned: set only when the controller flips spec.unschedulable. An operator's cordon (no marker) is never adopted
  • last-failure-at: max semantics on crash-recovery branch prevents voiding the pause

:::

State transitions

stateDiagram-v2
    [*] --> pending: selected in window
    [*] --> draining: forceful fallback (surge-less, §3.6)
    pending --> draining: surge_ready
    pending --> failed: readyTimeout elapsed
    pending --> expired: old NodeClaim force-expiring
    draining --> [*]: old NodeClaim gone (success + cooldown)
    failed --> pending: backoff elapsed + start gates pass (retry)
    failed --> expired: deletionTimestamp observed (backstop)
    expired --> [*]: terminal cleanup, release gate
    draining --> draining: drain exceeds tGP+buffer (stuck, gate held)
    note right of expired
        terminal: nothing was rotated,
        no cooldown
    end note
Loading

::: details Transition side effects — click to expand

From Event To Side effects
(none) selected in window pending write anchor (first); freeze old node; cordon old node; create placeholder
(none) forceful fallback draining write anchor + rotation-mode + draining-at; write state=draining; delete old NodeClaim (surge-less)
pending each reconcile pending claim state=pending from none/pending (conditional, before anything else); re-assert freeze + cordon; persist surge-claim; recreate placeholder if missing (held during freeze)
pending surge_ready draining freeze surge target; write draining-at + surge-wait; delete old NodeClaim
pending readyTimeout failed reap surge claim; delete placeholder; unfreeze; write state=failed + last-failure-at; clear anchor. A claim that vanished mid-rollback writes nothing: no attempt is announced, no pause stamped, and the anchor is left for completion to record a force-expiry
pending force-expiring expired claim state=expired from pending (conditional, before cleanup); emit expired once; delete placeholder; unfreeze; clear anchor
draining no deletionTimestamp draining re-issue delete (crash recovery)
draining drain > tGP + buffer draining stuck-drain gauge; gate held
draining NodeClaim gone (success) unfreeze; write last-rotation-at; emit success; clear anchor
failed backoff + gates pass pending claim state=pending from failed (conditional); started-at re-stamped by the new attempt
failed deletionTimestamp expired claim state=expired from failed (conditional); emit expired once; clear anchor
expired still anchored expired idempotent cleanup; clear anchor (metric not re-emitted)

:::

Clearing the anchor

clear(np, anchor) is a single update removing the whole rotation-scoped set:

  • active-rotation, active-rotation-state, draining-at, surge-wait, rotation-mode

No companion field can outlive the rotation. The failure path additionally writes last-failure-at in the same update.

Startup sweep

Runs once, gated before the first reconcile. Cleans only markers that no anchor references:

  • Placeholder Pods whose surge-for claim is absent/not-anchored → deleted
  • Node markers (surge-for, controller's do-not-disrupt by owned marker) → removed
  • cordoned marker with no anchored rotation → uncordon and remove

Rules:

  • An anchored NodePool is not stale — step 1 resumes it normally
  • failed/expired claims keep their annotations (backoff re-entry / terminal marker)
  • A pending/draining claim with no anchor (impossible from any crash point) → claim state=failed from pending/draining (conditional) + alert, both only when that write lands. The sweep selects from a List — a cache read — and writes later; a claim finalized away in that window, or one whose durable state has already left those two, was repaired by nothing here, so nothing is written and nothing is announced. Unlike the reconcile paths there is no anchor to hand the outcome to: having none is what selected the claim
  • Every line the sweep logs names work it performed. The placeholder delete and the node reversal are no-ops when the object vanished in the same List-to-write window as above — at either end of it, the read or the write — or when its markers had already been reversed, and announce nothing then. The node leg re-applies its selection predicate to the read its write is validated against, exactly as the claim leg does, and against the anchor set captured when the sweep started: a node whose markers that read shows belong to an anchored rotation carries current markers, not orphaned ones, and is left to the rotation that owns them. What was reversed is decided from that same read, and the line names it — unfroze for a surge-frozen node, uncordoned for a cordon-only one, which was never frozen and belongs to no claim
  • An orphaned active-rotation-state without anchor → simply removed
  • Best-effort: per-item errors logged, never fatal

5.4 Configuration Schema

::: tip What this section defines The RotationPolicy CRD (cluster-scoped, v1alpha1) carries per-NodePool rotation configuration. The controller resolves each NodePool's governing policy by selector specificity. :::

RotationPolicy CRD (noderotation.io/v1alpha1)

apiVersion: noderotation.io/v1alpha1
kind: RotationPolicy
metadata:
  name: api                       # cluster-scoped; one per NodePool policy
spec:
  nodePoolSelector:               # selects governed NodePools
    matchLabels:
      workload: api
  ageThreshold: auto              # "auto" (derived, §3.2) or Go duration override
  minRotationChances: 2           # K; floor 1
  maintenanceWindows:             # per-policy; union semantics (§3.1)
    - timezone: Asia/Tokyo
      days: [Wed, Sat]
      start: "02:00"
      end:   "06:00"
  surge:
    maxUnavailable: 1             # v1 fixed at 1 (OpenAPI rejects other)
    readyTimeout: 15m             # must be > 0
    cooldownAfter: 10m            # gate A; may be 0
    # failurePause: 10m           # gate B; unset → max(10m, cooldownAfter)
    # drainEstimate: 10m          # layer-2 only; unset → min(tGP, 10m)
    # provisioningEstimate: 5m    # layer-2 only; unset → min(readyTimeout, 5m)
    retryBackoff: 30m             # must be > 0
    matchNodeRequirements:        # placeholder requirement replication (§3.7)
      required:
        - topology.kubernetes.io/zone
        - kubernetes.io/arch
        - karpenter.sh/capacity-type
      preferred: []
    forcefulFallback:             # opt-in surge-less fallback (§3.6)
      enabled: false
  prePull:                        # v2 (disabled in v1)
    enabled: false
status:
  observedGeneration: 3
  matchedNodePools: 2
  rotatingNodePools: 1
  conditions:
    - type: Ready
      status: "True"
      reason: Accepted

Status subresource

  • matchedNodePools: pools this policy wins by selector specificity
  • rotatingNodePools: of those, count with an in-flight rotation
  • Ready condition:
    • Accepted — valid and uncontested
    • Invalid — failed reconcile-time validation
    • Conflict — equal-specificity tie (§below)
  • Invalid takes precedence over Conflict
  • Status is observational only — never authoritative for rotation decisions

A dedicated RotationPolicyStatusReconciler populates this view. Optimistic-concurrency conflicts are treated as silent requeues.

Targeting and conflict resolution

Rule Behavior
Most-specific wins Specificity = label-key constraint count
Equal-specificity tie Hard error — refuses to rotate that NodePool
No policy matches Not rotated (safe no-op)
  • Specificity: matchLabels entries + matchExpressions entries. Empty (catch-all) selector scores 0 — loses to any keyed selector
  • Tie: emits PolicyConflict Warning Event + sets noderotation_policy_conflict{nodepool} = 1
  • Unmatched: no implicit default; operator writes a catch-all if blanket coverage is desired

Leaving governance mid-rotation

When a pool ceases to be governed while a rotation is anchored, the controller rolls it back, in this order:

  1. Deletes placeholder
  2. Unfreezes nodes (preserving operator's own protections)
  3. Clears the anchor — only while it still names this rotation
  4. Emits GovernanceLost Warning Event

This prevents orphaned placeholders and stale do-not-disrupt markers from silently blocking Karpenter's voluntary operations indefinitely.

The order is normative, for two reasons:

  • The rollback precedes the clear. The anchor is the only thing that brings a later reconcile back to this cleanup — the reap returns immediately on a pool without one, and no policy governs the pool any longer. Clearing it ahead of a step that then fails would orphan the artifacts permanently.
  • The conditional clear elects the announcer. The reap is entered from the anchor its caller was handed, which is a cache read that still shows an anchor an earlier pass already cleared. The write that clears it is therefore what identifies the pass that reaped the rotation: at most one pass ever earns the announcement, and the pass that earns it describes work already done. This is the claim-then-announce ordering §5.2 uses for completion, and it inherits the same at-most-once semantics — one Event per reaped rotation in ordinary operation, and none at all when the controller dies between the write and the emission.

Policy change propagation

A create/update/delete of any RotationPolicy re-enqueues every NodePool for re-resolution (one change can alter which policy wins for any pool).

Per-NodePool maintenance windows

maintenanceWindows lives on each policy, so the window is per-NodePool. The union semantics (§3.1) apply within one policy's list. This is why noderotation_window_active and noderotation_window_period_seconds carry a load-bearing nodepool label (§4.2).