fix: reclaim orphaned Hetzner servers Karpenter has lost track of

README.md

@@ -25,7 +25,7 @@ version tag in production.
 - **Talos Linux and Ubuntu images**, resolved per architecture.
 - **Placement groups** for spreading nodes across physical hosts.
 - **Cost controls** — opt out of the billed public IPv4 (and/or IPv6) per node class for private-network clusters.
-- **Multi-cluster safe** — every managed server is tagged with the cluster name, so several clusters can share one Hetzner project without touching each other's nodes.
+- **Multi-cluster safe** — every managed server is tagged with the cluster name and the cluster's `kube-system` UID, so several clusters can share one Hetzner project without touching each other's nodes. Servers created before the UID label existed are matched on name alone, so give each cluster a distinct `clusterName` until the fleet has rolled.
 
 ## How it works
 
@@ -40,6 +40,7 @@ version tag in production.
 │  • instancetype  — server types → priced InstanceTypes      │
 │  • imagefamily   — resolve Talos/Ubuntu images per arch     │
 │  • nodeclass ctrl— validate HCloudNodeClass, set Ready      │
+│  • instance GC   — reclaim servers with no NodeClaim        │
 └───────────────┬────────────────────────────────────────────┘
                 │ hcloud API
         ┌───────▼────────┐
@@ -49,6 +50,83 @@ version tag in production.
 
 A `NodePool` references an `HCloudNodeClass`. When pods are unschedulable, Karpenter asks this provider for instance types, picks the cheapest compatible offering, creates the server, and the node joins the cluster.
 
+### Reclaiming orphaned servers
+
+A server can outlive Karpenter's record of it. If the operator dies between the
+hcloud create call and writing the provider ID to the NodeClaim — a lost leader
+election, an evicted pod, an API-server timeout — the machine boots and runs with
+nothing pointing at it. Karpenter core does not reclaim it: its garbage collector
+deletes NodeClaims that have no server, never the reverse.
+
+Two mechanisms cover this:
+
+- **Adoption.** Hetzner rejects duplicate server names, so the next attempt for
+  the same NodeClaim collides. Rather than retrying into that collision forever,
+  the provider looks the server up and adopts it, provided it belongs to this
+  cluster and this NodeClaim and matches the requested type, location and image.
+- **Garbage collection.** A sweep every two minutes reclaims servers Karpenter
+  has no NodeClaim for, along with the Node objects they left behind. A server
+  must be seen unowned on several consecutive sweeps, and one whose node is
+  registered and still `Ready` is never touched — a machine carrying workloads is
+  core's to drain, not this sweep's to destroy.
+
+  Every path that declines to act resets the count, so the window always measures
+  an uninterrupted run of sweeps that found nothing in the way; a machine the
+  `Ready` guard protected never sits on a spent window waiting for its first
+  NotReady blip. The count is per-process, so a restart or leader handover starts
+  it again — and the operator must additionally have been sweeping for a full
+  window before it may reclaim anything, so instability delays reclamation rather
+  than authorising it on a short history.
+
+**`clusterName` must be unique per cluster within a Hetzner project.** Servers
+are labelled with it, and the sweep uses that label to decide what it owns. Two
+clusters sharing a name in one project would each see the other's servers as
+unclaimed. The operator therefore also stamps the UID of the cluster's
+`kube-system` namespace on every server it creates and refuses to touch a server
+carrying a different one, logging the collision once and counting it as
+`karpenter_hetzner_orphaned_server_gc_total{result="skipped_foreign_cluster"}` on
+every sweep.
+
+Two things this does not cover. It protects servers created from this version
+onward; servers predating it carry no UID and are still matched on name alone, so
+until a fleet has fully rolled, distinct names remain the thing to get right.
+And the UID identifies the *control plane*, not the servers: rebuilding a cluster
+from scratch mints a new `kube-system` UID, after which the previous
+incarnation's servers are refused forever — never reclaimed, still billing. The
+`skipped_foreign_cluster` counter is the signal for both. Recovering from a
+rebuild means relabelling those servers with the new UID
+(`hcloud server add-label <server> karpenter.sh/cluster-uid=<uid>`, where `<uid>`
+is `kubectl get ns kube-system -o jsonpath='{.metadata.uid}'`) or deleting them
+by hand.
+
+> **Upgrading an existing cluster.** This version adds a controller that
+> **deletes Hetzner servers**. On first start it reclaims every server in the
+> project that carries this cluster's labels and has no NodeClaim — which is the
+> point, but on a fleet nobody has audited it is worth seeing first.
+>
+> Set `instanceGarbageCollection.mode: observe` to run every check and report
+> what *would* be reclaimed without deleting anything. Watch
+> `karpenter_hetzner_orphaned_server_gc_total{result="would_reap"}` and the
+> `WouldGarbageCollect` events on the affected Nodes, satisfy yourself the list
+> is right, then switch to `enabled`. Reclamations are recorded as
+> `GarbageCollected` events on the Node, so `kubectl describe node` explains a
+> server that disappeared.
+
+Set `instanceGarbageCollection.mode: disabled` to pause the sweep during
+maintenance that removes NodeClaims wholesale (reinstalling the CRDs, restoring
+etcd, clearing finalizers by hand), so it does not act on a cluster that only
+looks empty. Provisioning and disruption keep working while it is off.
+
+Watch `karpenter_hetzner_orphaned_server_gc_total` and
+`karpenter_hetzner_server_adopt_total`, both labelled by `result`. Sustained
+`server_adopt_total{result="adopted"}` means creates are losing their results.
+Sustained `result="declined"` is worse: a NodeClaim keeps colliding with a server
+adoption refuses to take, so the machine bills with nothing able to claim it. On
+the sweep, a rising `orphaned_server_gc_total{result="error"}` means a server
+cannot be reclaimed and is still billing, and `result="sweep_failed"` means the
+sweep itself cannot run — it swallows list failures to protect its cadence, so
+this counter is the only place a permanently broken sweep shows up.
+
 ## Installation
 
 You need a Hetzner Cloud API token with read/write access and a Kubernetes cluster on Hetzner (the [hcloud Cloud Controller Manager](https://github.com/hetznercloud/hcloud-cloud-controller-manager) should set node provider IDs as `hcloud://<id>`).
@@ -176,7 +254,8 @@ comments explaining every field.
 | Env var | Required | Description |
 |---------|----------|-------------|
 | `HCLOUD_TOKEN` | yes | Hetzner Cloud API token |
-| `CLUSTER_NAME` | yes | Cluster identifier; scopes managed servers |
+| `CLUSTER_NAME` | yes | Cluster identifier; scopes managed servers. Must be unique per Hetzner project — two clusters sharing a value will reclaim each other's servers |
+| `INSTANCE_GARBAGE_COLLECTION_MODE` | no (`enabled`) | `enabled`, `observe` or `disabled`; an unrecognised value stops the operator starting (chart value: `instanceGarbageCollection.mode`) |
 | `METRICS_PORT` | no (8080) | Prometheus metrics port |
 | `HEALTH_PROBE_PORT` | no (8081) | Health/readiness probe port |
 

charts/karpenter-provider-hetzner/Chart.yaml

@@ -2,7 +2,7 @@ apiVersion: v2
 name: karpenter-provider-hetzner
 description: Karpenter cloud provider for Hetzner Cloud
 type: application
-version: 2.1.0
+version: 2.2.0
 appVersion: "2.1.0"
 # home points at Paperclip.inc (the project's canonical page) so listing
 # "Homepage" links drive backlinks to the domain; GitHub stays as source.
@@ -31,7 +31,5 @@ annotations:
   artifacthub.io/category: integration-delivery
   artifacthub.io/containsSecurityUpdates: "false"
   artifacthub.io/changes: |
-    - kind: fixed
-      description: "Pods bound to an hcloud CSI volume can now trigger provisioning: csi.hetzner.cloud/location is aliased to topology.kubernetes.io/zone"
     - kind: added
-      description: "Expose nodeSelector, affinity, tolerations, topologySpreadConstraints, imagePullSecrets, priorityClassName, command, args and an optional PodDisruptionBudget (minAvailable or maxUnavailable) on the Deployment via chart values"
+      description: "Reclaim Hetzner servers Karpenter no longer has a NodeClaim for, which would otherwise run and bill unowned; set instanceGarbageCollection.mode to observe to preview it, or disabled to pause it during maintenance that removes NodeClaims wholesale"

charts/karpenter-provider-hetzner/README.md

@@ -36,7 +36,8 @@ Existing `v1alpha1` objects are not migrated automatically; recreate them under
 
 | Key | Default | Description |
 |-----|---------|-------------|
-| `clusterName` | `""` (required) | Scopes which servers the controller manages |
+| `clusterName` | `""` (required) | Scopes which servers the controller manages; must be unique per Hetzner project |
+| `instanceGarbageCollection.mode` | `enabled` | `enabled`, `observe` (report what would be reclaimed, delete nothing) or `disabled` (see values.yaml) |
 | `replicas` | `1` | Controller replicas |
 | `image.repository` | `ghcr.io/paperclipinc/karpenter-provider-hetzner` | Image |
 | `image.tag` | `""` | Empty tracks the chart appVersion; pin a tag in production |
@@ -86,4 +87,12 @@ When `serviceMonitor.enabled=true` the chart creates:
 - a `Service` named `karpenter-provider-hetzner-metrics` exposing port `http-metrics`
 - a `ServiceMonitor` that selects that Service and scrapes `/metrics` at the configured interval
 
-Requires the [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator) CRDs to be present. The controller exposes provider metrics under the `karpenter_hetzner_` prefix (server creates/deletes, durations, drift reasons, instance-type cache hits/misses, and raw hcloud API call counts).
+Requires the [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator) CRDs to be present. The controller exposes provider metrics under the `karpenter_hetzner_` prefix (server creates/deletes, durations, drift reasons, instance-type cache hits/misses, orphaned-server garbage-collection outcomes, adopted servers, unpriceable nodes, and raw hcloud API call counts).
+
+Worth an alert:
+
+- `karpenter_hetzner_orphaned_server_gc_total{result="error"}` — a server cannot be reclaimed and is still billing.
+- `karpenter_hetzner_server_adopt_total{result="declined"}` — a NodeClaim keeps colliding with a server adoption refuses to take.
+- `karpenter_hetzner_orphaned_server_gc_total{result="skipped_foreign_cluster"}` — servers carry this cluster's `clusterName` but another cluster's UID. Either two clusters share a name in one Hetzner project, or this cluster's control plane was rebuilt and these servers predate it. Either way they will never be reclaimed.
+
+Note that the metrics endpoint is scraped on every replica, but the sweeps behind these metrics run only on the leader. Aggregate with `max()` rather than `avg()`/`min()`, or a standby's zero will read as a healthy cluster.

charts/karpenter-provider-hetzner/templates/deployment.yaml

@@ -68,6 +68,17 @@ spec:
                   key: {{ .Values.auth.secretRef.key }}
             - name: CLUSTER_NAME
               value: {{ .Values.clusterName | quote }}
+            {{- if hasKey .Values.instanceGarbageCollection "disabled" }}
+              {{- fail "instanceGarbageCollection.disabled has been replaced by instanceGarbageCollection.mode (enabled|observe|disabled). Your existing value is being ignored, which would leave the orphaned-server sweep RUNNING. Set mode explicitly and remove the disabled key." }}
+            {{- end }}
+            # Emitted unconditionally rather than through `with`, which skips
+            # falsy values: `mode: false` -- the natural typo when migrating from
+            # the boolean this replaced -- would omit the variable, the operator
+            # would default to enabled, and a value meant to stop the sweep would
+            # start it. Passed through so an invalid value reaches parseGCMode,
+            # which refuses to start.
+            - name: INSTANCE_GARBAGE_COLLECTION_MODE
+              value: {{ .Values.instanceGarbageCollection.mode | quote }}
             - name: METRICS_PORT
               value: {{ .Values.metrics.port | quote }}
             - name: HEALTH_PROBE_PORT

charts/karpenter-provider-hetzner/templates/rbac.yaml

@@ -15,14 +15,23 @@ rules:
   - apiGroups: [""]
     resources: ["nodes", "nodes/status"]
     verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
-  # Core resources Karpenter's scheduling simulation observes.
+  # Core resources Karpenter's scheduling simulation observes. "namespaces" is
+  # also load-bearing for startup: the operator reads the kube-system namespace's
+  # UID to identify this cluster and refuses to start without it, so narrowing
+  # this rule takes the whole controller down, not just scheduling.
   - apiGroups: [""]
     resources: ["pods", "persistentvolumes", "persistentvolumeclaims", "replicationcontrollers", "namespaces"]
     verbs: ["get", "list", "watch"]
   - apiGroups: [""]
     resources: ["pods/eviction"]
     verbs: ["create"]
-  - apiGroups: [""]
+  # Karpenter core and leader election use the deprecated core-group recorder;
+  # this provider's garbage collector uses the manager's recorder, which writes
+  # to events.k8s.io/v1. RBAC authorizes on (apiGroup, resource), so the core
+  # group alone would 403 every GarbageCollected event -- silently, since only a
+  # klog line reports it, while the README tells operators to validate observe
+  # mode by reading those very events.
+  - apiGroups: ["", "events.k8s.io"]
     resources: ["events"]
     verbs: ["create", "patch"]
   - apiGroups: [""]

charts/karpenter-provider-hetzner/values.yaml

@@ -52,6 +52,24 @@ args: []
 # Required: scopes managed servers so multiple clusters can share one Hetzner project.
 clusterName: ""
 
+# The operator reclaims servers Karpenter no longer has a NodeClaim for, which
+# would otherwise run and bill with nothing pointing at them.
+instanceGarbageCollection:
+  # enabled  - reclaim orphaned servers.
+  # observe  - run every check and report what WOULD be reclaimed, deleting
+  #            nothing. Start here on an existing fleet: watch
+  #            karpenter_hetzner_orphaned_server_gc_total{result="would_reap"},
+  #            satisfy yourself it names the right machines, then switch to
+  #            enabled. A kill switch only helps after something has gone wrong.
+  # disabled - do not sweep at all. For maintenance that removes NodeClaims
+  #            wholesale (reinstalling the CRDs, restoring etcd, clearing
+  #            finalizers by hand), so the sweep does not act on a cluster that
+  #            only looks empty. Provisioning and disruption keep working.
+  #
+  # An unrecognised value stops the operator starting rather than falling back
+  # to enabled: a typo here would reap the fleet it was meant to protect.
+  mode: enabled
+
 metrics:
   port: 8080
 healthProbe:

cmd/controller/main.go

@@ -1,11 +1,16 @@
 package main
 
 import (
+	"context"
+	"time"
+
 	"sigs.k8s.io/controller-runtime/pkg/log"
 
 	// Register karpenter core types into the default k8s scheme.
 	_ "sigs.k8s.io/karpenter/pkg/apis/v1"
 
+	"github.com/awslabs/operatorpkg/controller"
+
 	"sigs.k8s.io/karpenter/pkg/cloudprovider/overlay"
 	"sigs.k8s.io/karpenter/pkg/controllers"
 	"sigs.k8s.io/karpenter/pkg/controllers/state"
@@ -15,13 +20,18 @@ import (
 	_ "github.com/paperclipinc/karpenter-provider-hetzner/pkg/apis/v1"
 
 	hetznercp "github.com/paperclipinc/karpenter-provider-hetzner/pkg/cloudprovider"
+	instancegc "github.com/paperclipinc/karpenter-provider-hetzner/pkg/controllers/instance/garbagecollection"
 	"github.com/paperclipinc/karpenter-provider-hetzner/pkg/controllers/nodeclass"
 	hetznerop "github.com/paperclipinc/karpenter-provider-hetzner/pkg/operator"
 	"github.com/paperclipinc/karpenter-provider-hetzner/pkg/providers/imagefamily"
 	"github.com/paperclipinc/karpenter-provider-hetzner/pkg/providers/instance"
 	"github.com/paperclipinc/karpenter-provider-hetzner/pkg/providers/instancetype"
 )
 
+// clusterUIDTimeout bounds the one API-server read the operator makes before the
+// manager -- and therefore the health probes -- are running.
+const clusterUIDTimeout = 30 * time.Second
+
 func main() {
 	ctx, op := operator.NewOperator()
 
@@ -38,8 +48,27 @@ func main() {
 		return
 	}
 
+	// Identify this cluster independently of its operator-chosen name. CLUSTER_NAME
+	// is not guaranteed unique, and two clusters sharing one in a single Hetzner
+	// project would otherwise each treat the other's servers as its own -- which
+	// now means deleting them. The kube-system UID is unique per cluster and
+	// stable for its lifetime. Read through the API reader because the manager's
+	// cache is not running yet.
+	//
+	// Bound it: this runs before the manager starts, so the health probes are not
+	// listening yet and an apiserver that accepts the connection but never answers
+	// would hang the process where nothing can observe it. The rest config sets no
+	// per-request timeout of its own.
+	uidCtx, cancelUID := context.WithTimeout(ctx, clusterUIDTimeout)
+	clusterUID, err := hetznerop.ClusterUID(uidCtx, op.GetAPIReader())
+	cancelUID()
+	if err != nil {
+		log.FromContext(ctx).Error(err, "failed to read the cluster UID")
+		return
+	}
+
 	// Create the three providers.
-	instanceProvider := instance.NewProviderWithPlacementGroups(&hcloudClient.Server, &hcloudClient.PlacementGroup, cfg.ClusterName, &hcloudClient.Action)
+	instanceProvider := instance.NewProviderWithPlacementGroups(&hcloudClient.Server, &hcloudClient.PlacementGroup, cfg.ClusterName, clusterUID, &hcloudClient.Action)
 	typeProvider := instancetype.NewProvider(&hcloudClient.ServerType)
 	imageProvider := imagefamily.NewProvider(&hcloudClient.Image)
 
@@ -60,6 +89,38 @@ func main() {
 	// Our NodeClass status controller (network + image validation, Ready).
 	nodeClassController := nodeclass.NewController(op.GetClient(), &hcloudClient.Network, &hcloudClient.Firewall, &hcloudClient.SSHKey, imageProvider)
 
+	providerControllers := []controller.Controller{nodeClassController}
+	// Reap servers whose NodeClaim is gone. Karpenter core only garbage collects
+	// the opposite direction (NodeClaims with no instance), so without this an
+	// orphaned server runs and bills indefinitely.
+	//
+	// Every mode is logged, not just the unusual ones: this controller deletes
+	// machines, so which mode took effect must be answerable from the operator's
+	// own startup logs rather than inferred from a values file.
+	//
+	// Every mode is named explicitly and `default` refuses to start. Routing the
+	// unknown case to the deleting branch would re-open, one layer down, exactly
+	// the hole parseGCMode exists to close: GCMode's zero value is "", not
+	// "enabled", so any Config built without LoadConfig -- or any mode added to
+	// the parser and forgotten here -- would silently select "delete servers".
+	switch cfg.InstanceGarbageCollectionMode {
+	case hetznerop.GCDisabled:
+		log.FromContext(ctx).Info("instance garbage collection is disabled; " +
+			"servers whose NodeClaim is gone will not be reclaimed")
+	case hetznerop.GCObserve, hetznerop.GCEnabled:
+		mode := instancegc.Mode(cfg.InstanceGarbageCollectionMode)
+		log.FromContext(ctx).Info("instance garbage collection is active",
+			"mode", string(mode),
+			"reclaims", mode == instancegc.ModeEnabled)
+		providerControllers = append(providerControllers,
+			instancegc.NewController(op.GetClient(), instanceProvider,
+				cfg.ClusterName, clusterUID, mode, op.Clock))
+	default:
+		log.FromContext(ctx).Error(nil, "unhandled instance garbage collection mode; refusing to start",
+			"mode", string(cfg.InstanceGarbageCollectionMode))
+		return
+	}
+
 	// Wire and start all controllers.
 	op.WithControllers(ctx, append(
 		controllers.NewControllers(
@@ -73,6 +134,6 @@ func main() {
 			clusterState,
 			op.InstanceTypeStore,
 		),
-		nodeClassController,
+		providerControllers...,
 	)...).Start(ctx)
 }