Emergency Node Reclaim by Process Match

[ebourgeois]

Description of Problem

Status: ⏳ Proposed
Author: Erick Bourgeois
Created: 2026-04-19
Scope: New node-side binary (or shell script) plus a small reconciler hook in the 5-Spot controller. Complements time-based ScheduledMachine scheduling with an event-driven, process-triggered emergency path.

---

Context

5-Spot today removes a physical machine from a k0smotron cluster on a time schedule — weekdays 9-17, weekends off, custom cron, etc. That model assumes the human owner of the machine doesn't need it back right now; they'll get it back when the schedule flips.

The real-world pattern breaks that assumption. A developer whose workstation doubles as a cluster node wants the box back the second they open IntelliJ / fire up a JVM / launch a game. Waiting for the next schedule boundary — or even a graceful drain (tens of seconds to minutes) — is unacceptable when the user is staring at a spinning JVM startup and the cluster is eating their cores.

We need a nuclear eject path: watch for a configured set of process names / command patterns on the node, and the moment one is running, rip the node out of the cluster as fast as physically possible.

Goal

Ship a small node-side agent that:

1. Watches for processes whose command / argv matches a user-supplied list (e.g. java, idea, steam, blender).
2. The moment a match is seen, triggers an ASAP-mode removal of the node from its k0smotron cluster — faster than the normal scheduled-drain path.
3. Integrates with the existing 5-Spot controller so the removal is observable via the ScheduledMachine status and audit-visible like any other reconciler-driven action — no out-of-band kubectl delete that the controller then has to reconcile around.

Non-goals

• Not a replacement for time-based scheduling — this is additive. ScheduledMachine still owns the day-to-day add/remove cycle; this agent fires only for the emergency path.
• Not a general-purpose process reaper. Scope is "signal to 5-Spot that this node needs out now", not "kill the matching process" or "pause the workload".
• Not graceful. A graceful drain is already available via the schedule path; the point of this feature is to skip it. Pods on the node will be forcibly terminated.
• Not cross-node. Each agent is scoped to the node it runs on. A user launching Java on node A does not reclaim node B.
• Not a daemon that talks to the Kubernetes API directly from the node with broad credentials. The node-side surface should be the smallest possible trigger; all cluster-mutating work stays in the controller.

Potential Solutions

Phases

Phase 1 — Trigger contract between node and controller

Outcome: A documented way for the node agent to signal "reclaim me" to the 5-Spot controller without broad API access.

Chosen shape (subject to review — see Open questions): Node annotation. The node agent writes

5spot.finos.org/reclaim-requested: "true"
5spot.finos.org/reclaim-reason: "process-match: java"
5spot.finos.org/reclaim-requested-at: "2026-04-19T21:30:00Z"

onto its own Node object (RBAC: patch nodes/<this-node> only, via the kubelet node-scoped token, no cluster-wide write). The 5-Spot controller watches Node updates and, on seeing the annotation, kicks the corresponding ScheduledMachine into an emergency-remove phase.

Rationale vs. alternatives:

• A dedicated ReclaimRequest CRD is cleaner semantically but needs a second RBAC path for the node and another CRD to version-manage. Annotation reuses existing kubelet credentials.
• Calling the k0smotron API directly from the node bypasses the controller's state machine and makes the ScheduledMachine status lie. Rejected.

Phase 2 — Node-side agent (the "nuke button")

Outcome: A single static binary (Rust) that, given a config file listing match patterns, watches for matches and writes the annotation.

• Match surface: exact command basename, substring of argv, or regex. Start with exact-basename + substring; regex only if Phase 1 users ask. Read config from /etc/5spot/reclaim.toml:

  # Trigger immediately when any of these are seen running.
  match_commands = ["java", "idea", "steam"]
  match_argv_substrings = ["intellij", "blender"]
  poll_interval_ms = 250   # cap on detection latency

• Detection — two-rung ladder, allow-and-react only. Stance is observe, signal, react — we never block the user's exec, so BPF-LSM's denial capability is out of scope, and plain eBPF (tracepoint / kprobe) offers no meaningful win over the simpler netlink path below. eBPF is explicitly not on this ladder.

  Rung	Mechanism	Latency floor	Kernel / capability requirements
  1. MVP	Poll /proc/*/comm + /proc/*/cmdline every poll_interval_ms (default 250 ms)	~250 ms, bounded by poll interval; can miss sub-poll-interval processes	Any Linux kernel, no capabilities beyond reading /proc (runs as root for annotation patch path anyway). Zero kernel-config assumptions.
  2. Default once proven	Netlink connector on NETLINK_CONNECTOR / CN_IDX_PROC, subscribed to PROC_EVENT_EXEC	Sub-millisecond; catches every exec including short-lived ones	CONFIG_PROC_EVENTS=y (on by default in every mainstream distro kernel ≥ 2.6.15), and CAP_NET_ADMIN on the agent process. Userspace code is plain netlink socket + struct parsing — no BPF program, no verifier, no libbpf/aya dependency.

  Rung 2 replaces rung 1 once the MVP has soaked. The agent's match / annotation logic is identical between rungs — only the event source changes — so the upgrade is a module swap, not a rewrite. Keep rung 1 code in-tree as a --poll flag fallback for kernels (or hardened distros) that strip proc connector support.

  Explicitly rejected alternatives:

  • eBPF (tracepoint / kprobe / LSM). Adds CAP_BPF / CAP_PERFMON, a kernel floor of 4.18+ (LSM: 5.7+), a BPF program to audit in the regulated-banking review path, and build-time complexity (aya / libbpf). Buys only a marginal CPU saving over rung 2 and zero latency benefit for our use case. The only thing eBPF uniquely enables is blocking the exec via BPF-LSM — which is the opposite of the allow-and-react stance. Revisit only if a future requirement genuinely needs in-kernel denial.
  • "Watchdog" as a detection mechanism. The term is ambiguous: a systemd / hardware watchdog is a liveness signal from a unit to its supervisor, not a process-match mechanism, and does not apply here. In the other, looser sense of "a process that watches for X and acts on it", the node agent itself IS the watchdog — rung 1 and rung 2 are two implementations of that same watchdog loop. Using "watchdog" as a Phase-2 name adds nothing over "node agent" and is avoided in this roadmap.
  • fanotify / auditd / inotify on exec. Either doesn't give us argv (inotify), conflicts with existing cluster audit rules (auditd), or buys nothing over proc connector (fanotify FAN_OPEN_EXEC).

• Action: single PATCH nodes/<hostname> with the three annotations above. No retry loop that waits for "reclaim completed" — the controller owns completion; the agent's job ends when the annotation is written.

• Idempotence: if the annotation is already present, skip the patch and exit (or sleep). Re-running the agent after a match should not double-fire.

• Distribution: statically-linked Rust binary, released from the same pipeline as the main 5-Spot controller. Shipped as:

  • A tarball release asset (signed, SBOM'd, VEX'd — reuses the pipeline from 5spot-vex-generation-and-signing.md).
  • A systemd unit file in deploy/node-agent/ with a sample reclaim.toml.

• Bash fallback: a tools/reclaim-on-match.sh shell script that does the same thing with pgrep and kubectl annotate node. Handy for one-off testing and for nodes where installing the Rust binary is premature. Not the long-term answer — pgrep races and shell signal handling is a known footgun — but useful for Phase 1 validation.

Phase 2.5 — Opt-in installation, gated by spec.killIfCommands

Outcome: The node agent is installed only on nodes whose ScheduledMachine actually asks for it, via a ConfigMap-driven DaemonSet that mirrors the kata-deploy pattern.

The premise: the reclaim agent is not universally wanted. Most ScheduledMachines run on dedicated cluster boxes where no human is ever going to launch Java locally. The agent should be deployed only where a killIfCommands list is explicitly set on the ScheduledMachine. This keeps the attack surface and node-side footprint proportional to declared intent, and mirrors the shape already established in deploy/admission/child-cluster-kata-runtime-mutatingpolicy.yaml for kata: opt-in via CRD-driven node labels, DaemonSet nodeSelector picks it up.

CRD change — ScheduledMachine.spec:

spec:
  # ... existing fields ...

  # Optional. When non-empty, the 5-Spot controller installs the reclaim
  # agent on every Node backing this ScheduledMachine and configures it
  # with these match patterns. When absent or empty, no agent is installed
  # and no emergency-reclaim path exists for this machine — behaviour is
  # time-based scheduling only.
  killIfCommands:
    - java
    - idea
    - steam

No separate match_argv_substrings at the CRD level — keep the spec surface minimal; single list of basename-or-substring patterns that the agent evaluates against both /proc/*/comm and /proc/*/cmdline. If a future user needs the distinction, extend to an object form [{comm: "...", argv: "..."}].

CRD source of truth is src/crd.rs — after editing, the regen-crds + regen-api-docs skills must run (per project memory feedback_crdgen_crddoc.md).

Controller change — label the Node from the spec:

When reconciling a ScheduledMachine whose spec.killIfCommands is non-empty, the controller (on the management cluster) reaches into the child cluster and:

1. Stamps label 5spot.finos.org/reclaim-agent=enabled onto each Node backing this machine.
2. Writes a per-node ConfigMap (in a dedicated 5spot-system namespace on the child cluster) containing the killIfCommands list in the agent's TOML shape. ConfigMap name: reclaim-agent-<node-name>. This keeps the match list authoritative in the ScheduledMachine spec; the ConfigMap is a projection, not user-edited.
3. On spec change (commands added/removed), re-projects the ConfigMap. On killIfCommands being cleared back to empty, removes the label and the ConfigMap, which tears the DaemonSet pod off the node.

Installation — reclaim-agent-deploy DaemonSet (child cluster):

Ships as a manifest under deploy/node-agent/ alongside the existing kata policy pair. Shape mirrors kata-deploy exactly:

• DaemonSet with nodeSelector: { 5spot.finos.org/reclaim-agent: enabled }.
• Pod mounts / of the host (as hostPath) plus the per-node ConfigMap, hostPID: true so /proc is the real host /proc once rung 2 lands.
• An init-container copies the static binary into /usr/local/bin/5spot-reclaim-agent on the host and drops a systemd unit + reclaim.toml (rendered from the ConfigMap) into /etc/systemd/system/ and /etc/5spot/.
• Then systemctl daemon-reload && systemctl enable --now 5spot-reclaim-agent.service via nsenter into PID 1.
• Main container is an idle sleep that holds the DaemonSet pod open so removal (label goes away) triggers a teardown hook that systemctl disable --nows the unit and deletes the binary / config.

This is the same "DaemonSet as installer, systemd as runtime" pattern kata-deploy uses — we're not reinventing the shape, just applying it to our agent.

Lifecycle interaction with the kata-label policy:

The existing child-cluster-kata-runtime-mutatingpolicy.yaml labels Nodes at CREATE. The reclaim-agent label is applied by the controller, not by admission, because it's driven by per-ScheduledMachine spec — not every node gets it. The two labels are independent; a Node can have kata, reclaim-agent, both, or neither.

Why not install by default:

• Opt-in matches the principle that 5-Spot should be invisible on nodes that don't need its emergency path.
• Nuclear-drain semantics are easy to misconfigure. Requiring spec.killIfCommands to be an explicit list forces the operator to name the trigger patterns, which appear in audit logs and PR diffs. "Everyone has the agent installed with empty match list" is the failure mode we want to make impossible.
• For regulated-banking environments, a node-side binary with CAP_NET_ADMIN (rung 2) is not something to ship to nodes that haven't declared they want it.

Phase 3 — Controller-side emergency remove phase

Outcome: The 5-Spot reconciler honours the reclaim annotation and skips the graceful-drain steps.

• Watch Node objects (already event-driven per the event-driven-watches roadmap); on update, check for the reclaim annotation.
• Map node → ScheduledMachine via existing clusterName + infrastructure linkage.
• Enter a new Phase::EmergencyRemove state (new variant in the ScheduledMachine status enum). Semantics:
  • gracefulShutdownTimeout and nodeDrainTimeout are ignored.
  • kubectl drain --grace-period=0 --force --disable-eviction (no PDB respect).
  • k0smotron Machine object deleted immediately after the kubelet stop signal fires.
• Emit a Kubernetes Event on the ScheduledMachine with reason EmergencyReclaim and the reason string from the annotation, so it shows up in kubectl describe scheduledmachine and the audit log.
• Clear the annotation as the last step (so re-adding the node to the cluster later doesn't immediately re-fire the trigger).

Phase 4 — Tests

• Unit tests (in reconcilers/scheduled_machine_tests.rs):
  • annotation → Phase::EmergencyRemove transition, happy / missing-node / unmapped-node paths.
  • killIfCommands set → node label + projected ConfigMap present; cleared → both torn down; changed → ConfigMap re-projected with the new list (no stale list lingers).
• Node-agent unit tests: /proc fixture directory, match/no-match, annotation payload shape.
• CRD tests (in src/crd_tests.rs): killIfCommands parses as Vec<String>, empty and absent both round-trip as "no agent", non-empty survives a roundtrip through serde_yaml.
• Integration test (in tests/): kind cluster + stub k0smotron, set spec.killIfCommands on a ScheduledMachine, assert the DaemonSet lands on the labelled node, then the agent fires and the graceful-drain path is not taken and the machine is removed within N ms of annotation appearing.
• Manual verification: one real node, launch Java, stopwatch from JVM main to node leaving cluster. Target: < 5 s end-to-end on a quiet cluster.

---

Open questions

1. Annotation vs. CRD for the trigger contract. Annotation is lighter; CRD is auditable in more tools (FluxCD, Kyverno reporters, etc.). Decision depends on how much we care about the trigger itself being a first-class object vs. ephemeral. Defaulting to annotation; revisit if a Phase 1 user asks for CRD.
2. Do we kill the matched process? Current answer: no. The user started it on purpose — killing their JVM to "help them get their node back" is exactly wrong. Only 5-Spot's side of the contract is in scope.
3. What if the user kill -9s the matched process before the reconciler sees the annotation? Annotation stays; node still gets reclaimed. That's the intended contract — match is a trigger, not a live predicate. If we ever want "reclaim only while the process is running" semantics, that's a separate RFC.
4. Multiple nodes matching at once. Each node annotates itself; controller processes them independently. No coordination needed.
5. Abuse surface. The annotation key is under our namespace (5spot.finos.org/); kubelet's node-scoped token can already label its own node, so this is not a new privilege. Still: document it, and consider a ValidatingAdmissionPolicy on the child cluster that rejects the reclaim annotations from non-kubelet writers.

Dependencies

• 5spot-event-driven-watches-and-status-enrichment.md — Phase 3 assumes the controller is already watching Node updates rather than polling. If that roadmap lands first, Phase 3 is cheap; if not, Phase 3 also needs to wire up the watch.
• 5spot-vex-generation-and-signing.md — the node-agent binary reuses the existing supply-chain pipeline (Cosign, SBOM, OpenVEX) rather than inventing a second one.
• CRD regeneration — src/crd.rs is the source of truth. Any edit to add killIfCommands must run the regen-crds and then regen-api-docs skills (in that order) before merge. The CRD YAML in deploy/crds/ is auto-generated and will silently drift if regen is skipped.

Out of scope (explicitly)

• A GUI / tray icon. CLI + config file only.
• "Scheduled reclaim windows" on the agent side (e.g. "only fire if it's weekday-evening"). The schedule already belongs to ScheduledMachine; the agent is the unconditional nuclear button.
• Cross-cluster / federation. One node → its own cluster, full stop.

[ebourgeois]

Closed as part of #37