fix: reclaim orphaned Hetzner servers Karpenter has lost track of - #53
Draft
pkieszcz wants to merge 4 commits into
Draft
fix: reclaim orphaned Hetzner servers Karpenter has lost track of#53pkieszcz wants to merge 4 commits into
pkieszcz wants to merge 4 commits into
Conversation
pkieszcz
force-pushed
the
fix/orphaned-server-recovery
branch
from
August 25, 2026 06:49
3e3143c to
d194bef
Compare
When the controller dies between the Hetzner create call and persisting the provider ID, the server keeps running with no owner. Every retry then collides on the server name, Karpenter eventually deletes the NodeClaim, and the server bills indefinitely. A lost leader election was enough to strand several servers this way, some of them running for days before anyone noticed. That crash window cannot be closed by error handling, so recover from it instead: on uniqueness_error, look the server up by name and adopt it. Adoption is deliberately narrow. Requiring this cluster's and this NodeClaim's labels is what makes it safe to assume the machine was built from these inputs, since userData, SSH keys and the placement group are not verifiable afterwards. Server type, location and image are checked directly, because Karpenter derives the NodeClaim's capacity, zone and image from the offering selected on THIS attempt: a mismatch would advertise capacity the machine lacks, or a zone its volumes cannot reach. An unresolved location fails closed rather than skipping the check, as that is the case where the zone is most likely wrong. A server that is not running or still starting is refused, since adoption cannot wait on create actions the way the normal path does. A create whose actions fail now deletes the server it just made rather than abandoning it, which keeps adoption's only input the crash case, where the machine is healthy. That cleanup runs on a context detached from the caller's: WaitFor returns ctx.Err() when the caller's context is cancelled, which is the likeliest way to reach the branch at all, so reusing it would fail the delete before it issued a request and leak the very server being reclaimed. Adoption outcomes are counted separately from creates -- including declines and lookup failures, since a NodeClaim retrying into a collision adoption keeps refusing is otherwise invisible while the machine bills.
Karpenter core's garbage collector runs in one direction only: it deletes NodeClaims that have no instance. Nothing terminates an instance that has no NodeClaim, so that direction is the cloud provider's responsibility. Without it an orphaned server runs and bills forever. Sweep every two minutes over the servers this installation created, terminating those Karpenter no longer claims along with the Node object left behind. Ownership is read from the karpenter.sh/nodeclaim label as well as the NodeClaim provider ID, so a server whose ID was never recorded is still recognised while its Create is in flight, and the ownership labels are re-checked here rather than trusting the caller's list filter. Grace is counted in consecutive sweeps that found nothing standing in the way of reclaiming the server -- not sweeps since its NodeClaim went missing, and not elapsed time. Machine age is the wrong quantity: a server owned for hours would lose all protection the moment its NodeClaim blipped. Counting observations also removes any dependence on comparing Hetzner's clock to this pod's. Crucially the count only advances on sweeps that cleared every guard, so a machine spared for hours never sits on a spent counter where a single briefly-NotReady kubelet would destroy it; once sparing stops it must earn a complete fresh window. The sweep declines to act whenever the state is not clearly an orphan. A node that is registered and still Ready is left alone, since a lost NodeClaim (a CRD reinstall, an etcd restore) must never become a fleet-wide termination with no eviction or drain. Registration is read from karpenter.sh/registered, which Karpenter stamps itself and treats as the definition of registered; the unregistered taint is deliberately not used, as it reaches a node only when the NodePool declares it as a startupTaint. A node the CCM has not yet stamped with a provider ID is matched by name rather than treated as absent. Provider IDs mapping to more than one Node are skipped rather than resolved by guessing, and Node deletes carry a UID precondition so a same-named replacement is never removed. Failures never cost the cadence: list errors and undeletable servers are logged, counted, and stepped over, because returning an error drops RequeueAfter and ratchets controller-runtime's backoff toward its cap. Sweeps that cannot complete are recorded as sweep_failed, so a permanently broken sweep is distinguishable from a cluster that simply has no orphans. DISABLE_INSTANCE_GARBAGE_COLLECTION pauses the sweep for maintenance that removes NodeClaims wholesale, without also stopping provisioning and disruption. An unrecognised value is rejected rather than defaulted: the moment an operator reaches for this flag is a maintenance window, and a typo failing open would reap the fleet they were protecting. The chart already grants node delete, which Karpenter core requires for termination, so no RBAC change is needed.
CLUSTER_NAME is operator-supplied and nothing enforces uniqueness. Two clusters sharing one in a single Hetzner project each read the other's servers as their own. That was harmless while the label only scoped listings; it is not now that unclaimed servers are deleted, so a copied values.yaml or a blue/green rebuild could have one cluster terminate another's fleet. Stamp the UID of the cluster's kube-system namespace on every server created. It is unique per cluster, stable for the cluster's lifetime, needs no state of our own, and is readable with the RBAC the chart already grants. Both the adoption path and the sweep refuse a server whose UID is present and different. A missing UID is treated as ours: servers created before this change carry none, and refusing them would strand every pre-existing orphan and break adoption for machines this cluster really did create. Distinct names therefore remain the thing to get right for existing fleets; new servers are protected regardless. Declining silently would not be enough. A name collision would look exactly like a cluster with no orphans, and nobody would learn two clusters share a name until something else broke. The sweep reports each colliding cluster once, naming both UIDs -- once per cluster rather than once per sweep, so it stays readable.
Three changes aimed at making this safe to adopt on a fleet nobody has audited. OBSERVE MODE. This controller deletes machines, and an operator adopting it could otherwise only learn what it would do to their fleet by letting it do it. A kill switch is reactive; observe mode is not. It runs every check, emits every signal, and changes nothing -- not the fleet, not the servers' labels, nothing outside the cluster. INSTANCE_GARBAGE_COLLECTION_MODE replaces the boolean, since three states do not fit in one, and an unrecognised value refuses to start rather than falling back to the mode that deletes. A STARTUP GRACE. Grace is counted in consecutive sweeps, which a restart or a leader handover resets -- and the instability that strands servers is exactly what causes those. A fresh process could otherwise reach the threshold having watched the cluster for only a few minutes. It must now also have been sweeping for a full window before it may reclaim anything, so operator instability delays reclamation rather than either skipping it forever or authorising it on a short history. This is what makes in-process counting safe rather than merely simple, and it follows core: nodeclaim/consistency keeps first-seen times in an in-process cache and writes nothing durable. Core's durable timeouts hang off status conditions on objects it owns, never off the cloud resource. An earlier revision of this branch stored the clock on the Hetzner server itself; it was withdrawn because it made a safety-critical invariant depend on a remote write that can fail silently, on a clock written by a process that may be gone, and on state an operator can edit with one CLI command. NODE EVENTS. Reclamations are recorded as GarbageCollected, and observe-mode candidates as WouldGarbageCollect, typed Normal -- reclaiming an orphan is this controller working, not a fault, and a cluster alerting on Warning events against Nodes should not page every time the sweep does its job. Most orphans never registered a Node, so this supplements the log and the metric rather than replacing them, and the doc comment says so. Three chart-level holes closed alongside them. The ClusterRole now grants events.k8s.io, without which the manager's recorder has every event rejected 403 while the README tells operators to validate observe mode by reading them. The mode is emitted unconditionally rather than through a "with" block that skips falsy values, so "mode: false" -- the natural typo when migrating from the boolean this replaces -- reaches the parser and stops the operator instead of silently defaulting to enabled. And a "fail" guard rejects an upgrade still carrying the removed "disabled" key, which Helm would otherwise merge while the sweep ran on a fleet the operator believed was paused. The clock is injected, matching core, so the startup grace is testable without reaching into a fake provider.
pkieszcz
force-pushed
the
fix/orphaned-server-recovery
branch
from
August 25, 2026 06:54
d194bef to
dc43935
Compare
Contributor
|
Directional answers while this is in draft, so you're not blocked on them:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A Hetzner server can outlive Karpenter's record of it. If the operator dies between the hcloud create call and writing the provider ID to the NodeClaim — a lost leader election, an evicted pod, an API-server timeout — the machine boots and runs with nothing pointing at it.
Every retry then collides on the server name (
uniqueness_error), so the launch never succeeds, Karpenter eventually deletes the NodeClaim, and the server bills indefinitely with no owner.Karpenter core does not reclaim it.
nodeclaim/garbagecollectiondeletes NodeClaims whose instance has disappeared, never the reverse — that direction is the provider's responsibility, and both the AWS and Azure providers ship their own instance GC for it.The shape of the failure, from a real occurrence:
No amount of in-process error handling closes that window: the process is gone between the API call and the write.
Two mechanisms
Adoption — on
uniqueness_error, take the existing server over instead of retrying into the same collision. After a crash the server's own name and labels are the only surviving record, which is exactly what this reads.It is narrow by design: the server must carry this cluster's and this NodeClaim's labels, be running or still starting, and match the requested type, location and image. An unresolved location fails closed. Relatedly, a create whose follow-up actions fail now deletes the server it just made — on a context detached from the caller's, since
WaitForreturnsctx.Err()on cancellation and reusing that context would fail the cleanup before it issued a request.A sweep every two minutes reclaims servers Karpenter no longer claims, along with the Node objects they left behind.
The sweep deletes machines — what bounds it
karpenter.sh/nodeclaimlabel, not just the NodeClaim provider ID, so an in-flight create whose ID was never recorded is still recognised as owned.Ready. Registration is read fromkarpenter.sh/registered, which Karpenter stamps itself — deliberately not the unregistered taint, which only reaches a node if the NodePool declares it as astartupTaint, so a cluster that does not configure it would have every orphan look registered.RequeueAfterand ratchets controller-runtime's backoff toward its cap, delaying every other orphan.Adopting this safely
INSTANCE_GARBAGE_COLLECTION_MODE=observeruns every check and reports what it would reclaim, changing nothing — not the fleet, not the servers' labels, nothing outside the cluster. On an existing fleet that is the way in: watchkarpenter_hetzner_orphaned_server_gc_total{result="would_reap"}and theWouldGarbageCollectevents, then switch toenabled.Neither AWS nor Azure offers this. They also default to on, which they have earned with years of fleet-hours that this has not.
Checked against the AWS and Azure providers
CreationTimestampCreationTimestampcloudProvider.List()Deleting the Node after the instance, and running an independent
Liston its own cadence, both match. The Ready guard is a deliberate divergence: a NodeClaim loss that is not per-instance — a CRD reinstall, an etcd restore — makes every server unclaimed at once, and without the guard the sweep becomes a fleet-wide termination with no eviction, drain or volume detachment. Both references look exposed to that. Happy to drop it and match them if you would rather not carry the difference.On grace: both references derive it from the instance's creation timestamp, which is durable but measures machine age — a server owned for hours loses all protection the moment its NodeClaim blips. Counting consecutive unowned sweeps measures the right quantity but resets when the process does, so the sweep additionally requires that this process has been watching for a full window before it may reclaim anything. Instability delays reclamation rather than skipping it forever or authorising it on a short history.
An earlier revision of this branch stored that clock on the Hetzner server as a label. It was withdrawn: it made a safety-critical invariant depend on a remote write that can fail silently, on a clock written by a process that may be gone, and on state anyone can edit with one CLI command. Core solves the same problem in memory (
nodeclaim/consistency) and reserves durable timeouts for status conditions on objects it owns.Two caveats
CLUSTER_NAMEis not guaranteed unique, and it previously decided what the provider would touch. Now that unclaimed servers are deleted, two clusters sharing a name in one Hetzner project would delete each other's fleets. Ownership therefore also stamps thekube-systemnamespace UID. A missing UID is treated as ours — servers predating this carry none, and refusing them would strand every pre-existing orphan — so until a fleet has rolled, distinct names remain the thing to get right.A rebuilt control plane mints a new UID, so the previous incarnation's servers are refused rather than reclaimed.
result="skipped_foreign_cluster"is the signal, the log names both possible causes, and the README documents relabelling as the recovery.Verification
Unit tests throughout, RED-first, with each guard mutation-tested — reverting it makes exactly its test fail. Beyond that the sweep has been exercised against a real fleet: it reclaimed precisely the orphaned servers, left every healthy worker and every control-plane node untouched, and on clusters with no orphans ran and correctly did nothing.
No RBAC change beyond
events.k8s.io, which the manager's event recorder requires;nodesdelete andnamespacesget were already granted. That one-line grant is the same change as #52 — happy to drop the hunk and depend on it if that lands first.Known gaps
Listduplicates core's on the same cadence — as it does on AWS and Azure.Split from a companion change that reports nodes Karpenter cannot price; unrelated failure, read-only, so it did not belong behind a controller that deletes machines.