Implement reclaim-agent host-identity check/validation

.claude/CHANGELOG.md

@@ -9,6 +9,66 @@ The format is based on the regulated environment requirements:
 
 ---
 
+## [2026-04-26 00:30] - Phase 4 of security audit: reclaim-agent host-identity verification
+
+**Author:** Erick Bourgeois
+
+### Changed
+- `src/reclaim_agent.rs`: new pure helpers `read_host_machine_id(path)`
+  and `compare_machine_ids(node, node_name, expected_host_id)` plus a
+  `HostIdentityError` enum with four variants (`ReadFailed`, `Empty`,
+  `Mismatch`, `NodeMachineIdMissing`). Both helpers are I/O-light and
+  unit-testable without a kube client.
+- `src/bin/reclaim_agent.rs`: read `/etc/machine-id` once at startup
+  via `read_host_machine_id`; cache as `Option<String>`. Before each
+  `annotate_node` PATCH, fetch the target Node and call
+  `compare_machine_ids` against the cached host id. On mismatch the
+  agent logs `error!` and bails (no PATCH). New `--machine-id-path`
+  (env `MACHINE_ID_PATH`, default `/etc/machine-id`) and
+  `--skip-host-id-check` (env `SKIP_HOST_ID_CHECK`, default false /
+  strict) CLI flags.
+- `deploy/node-agent/daemonset.yaml`:
+  - Single-file read-only mount of `/etc/machine-id` →
+    `/host/etc/machine-id`. `hostPath.type: File` so kubelet fails to
+    schedule the pod if the host file is missing (fail-fast over
+    silent skip-check).
+  - `MACHINE_ID_PATH=/host/etc/machine-id` env var to point the agent
+    at the mounted location.
+  - Commented `SKIP_HOST_ID_CHECK` env var documenting the bypass for
+    operators who deploy in environments without `/etc/machine-id`.
+- `src/reclaim_agent_tests.rs`: 9 new tests covering both helpers —
+  `read_host_machine_id` happy path / missing file / empty file /
+  whitespace-only file; `compare_machine_ids` match / mismatch (asserts
+  both ids appear in the error message for forensics) / no status /
+  empty machine-id / whitespace-only machine-id.
+- `docs/src/concepts/emergency-reclaim.md`: new "Host-identity
+  verification" section documenting the contract, the two modes
+  (strict default vs `--skip-host-id-check` opt-out), the mount shape
+  in the DaemonSet, and what the check does NOT defend against
+  (TPM-grade attestation is out of scope).
+
+### Why
+Phase 4 of the 2026-04-25 security audit roadmap. Without the
+cross-check, an attacker with `update daemonsets` in `5spot-system`
+could override the agent's `NODE_NAME` env var (downward API value →
+hard-coded victim) and cause the agent to trigger emergency-reclaim on
+a Node it has no business touching. The exploit precondition is
+cluster-admin-equivalent in most clusters, so this is filed Low
+severity / defence-in-depth, but the binding cost is small and removes
+the impersonation primitive entirely.
+
+### Impact
+- [ ] Breaking change
+- [x] Requires cluster rollout (DaemonSet manifest now mounts
+      `/etc/machine-id`; existing deployments need `kubectl apply` of
+      the new manifest. Hosts that lack `/etc/machine-id` will fail
+      pod scheduling — operators must either populate the host file
+      or set `SKIP_HOST_ID_CHECK=true` for those nodes.)
+- [ ] Config change only
+- [ ] Documentation only
+
+---
+
 ## [2026-04-25 22:00] - Phase 3 of security audit: route Node→SM via canonical CAPI Machine
 
 **Author:** Erick Bourgeois

deploy/node-agent/daemonset.yaml

@@ -111,6 +111,23 @@ spec:
               valueFrom:
                 fieldRef:
                   fieldPath: spec.nodeName
+            # MACHINE_ID_PATH points at the host's /etc/machine-id, mounted
+            # read-only via the host-machine-id volume below. The agent
+            # cross-checks its value against the target Node's
+            # status.nodeInfo.machineID before each PATCH; a mismatch
+            # refuses the write. Closes the "modified-DaemonSet
+            # → impersonate-victim-Node" attack documented in Phase 4 of
+            # the 2026-04-25 security audit roadmap.
+            - name: MACHINE_ID_PATH
+              value: "/host/etc/machine-id"
+            # SKIP_HOST_ID_CHECK opts out of the cross-check above. Default
+            # is "false" (strict). Enable ONLY for environments where
+            # /etc/machine-id is genuinely unavailable (containers without
+            # the host file, dev sandboxes, kubelet variants that do not
+            # populate status.nodeInfo.machineID). Production must stay
+            # strict — leaving this commented out keeps the binary default.
+            # - name: SKIP_HOST_ID_CHECK
+            #   value: "true"
           # Reading every `/proc/<pid>/comm` + `cmdline` requires root;
           # CAP_SYS_PTRACE is not strictly required for rung 1 but is kept
           # dropped. CAP_NET_ADMIN will be re-added for rung 2 (netlink
@@ -141,13 +158,32 @@ spec:
             - name: host-proc
               mountPath: /host/proc
               readOnly: true
+            # Single-file mount of the host's /etc/machine-id into
+            # /host/etc/machine-id (matches MACHINE_ID_PATH above).
+            # Mounted read-only — the agent only ever reads it. Single
+            # file rather than the whole /etc to minimise host filesystem
+            # exposure: the only path the agent needs is machine-id, so
+            # everything else stays out of the container's view.
+            - name: host-machine-id
+              mountPath: /host/etc/machine-id
+              readOnly: true
       volumes:
         # Host /proc — the detection surface. Mounted read-only; the agent
         # never writes to /proc, only reads comm + cmdline.
         - name: host-proc
           hostPath:
             path: /proc
             type: Directory
+        # Host machine-id (Phase 4 of the 2026-04-25 security audit
+        # roadmap). hostPath type=File ensures kubelet refuses to
+        # schedule the pod if the file is missing on the host — fail
+        # fast rather than silently degrading to skip-check mode. The
+        # file is written at host boot by systemd-machine-id-setup
+        # (or kairos / k0s-installer) and is stable across reboots.
+        - name: host-machine-id
+          hostPath:
+            path: /etc/machine-id
+            type: File
         # No ConfigMap volume. The agent uses the kube API directly to watch
         # its per-node ConfigMap (reclaim-agent-<NODE_NAME>) and reloads
         # live on every change — see src/bin/reclaim_agent.rs. This sidesteps

docs/src/concepts/emergency-reclaim.md

@@ -374,6 +374,80 @@ These are out of scope — recorded here because operators often ask:
 
 ---
 
+## Host-identity verification
+
+Before the agent PATCHes any Node with reclaim annotations, it
+cross-checks the host's `/etc/machine-id` against the target Node's
+`status.nodeInfo.machineID`. Both are populated from the same source
+(`systemd-machine-id-setup` on host boot; kubelet on Node registration),
+so they agree on a healthy node — and a mismatch is a strong signal
+that the agent is about to write to the wrong Node.
+
+### Why
+
+This closes the *modified-DaemonSet → impersonate-victim-Node* path:
+
+1. Without the check, the agent trusts whatever value `NODE_NAME` carries
+   (sourced from the downward API: `spec.nodeName`).
+2. An attacker with `update daemonsets` in `5spot-system` can override
+   `NODE_NAME` to a hard-coded value (e.g. `victim-node`).
+3. The agent runs on host A, sees a process match, and PATCHes
+   `victim-node` instead of host A — triggering `Phase::EmergencyRemove`
+   on a node it has no business reclaiming.
+
+The cross-check makes the impersonation visible: the spoofed Node's
+`status.nodeInfo.machineID` belongs to a different host, so the
+comparison fails and the agent refuses to write.
+
+The exploit precondition (`update daemonsets`) is cluster-admin in most
+clusters, so this is **defence-in-depth** — but the check is cheap and
+removes a category of impersonation. Filed as Phase 4 of the
+2026-04-25 security audit roadmap.
+
+### Modes
+
+| Mode | Flag / env | Behaviour |
+|---|---|---|
+| **Strict (default)** | `--skip-host-id-check=false` / `SKIP_HOST_ID_CHECK=false` | Read `/etc/machine-id` at startup; fail to start if missing or empty. Before each PATCH, fetch the target Node and refuse if `status.nodeInfo.machineID` does not match. |
+| Bypass | `--skip-host-id-check=true` / `SKIP_HOST_ID_CHECK=true` | Trust `NODE_NAME` blindly (pre-Phase-4 behaviour). Use **only** when `/etc/machine-id` is genuinely unavailable: containers without the host file mounted, dev sandboxes, kubelet variants that do not populate `status.nodeInfo.machineID`. |
+
+### How `/etc/machine-id` is exposed
+
+The DaemonSet mounts the host file as a single read-only file (not the
+whole `/etc`):
+
+```yaml
+volumeMounts:
+  - name: host-machine-id
+    mountPath: /host/etc/machine-id
+    readOnly: true
+volumes:
+  - name: host-machine-id
+    hostPath:
+      path: /etc/machine-id
+      type: File
+```
+
+`type: File` makes kubelet refuse to schedule the pod if the host file
+is missing — fail-fast is preferable to silently degrading to skip-check
+mode. The agent reads the path from `MACHINE_ID_PATH` (default
+`/host/etc/machine-id` when deployed via the manifest).
+
+### What it does NOT defend against
+
+- A compromise that lets an attacker modify *both* the DaemonSet **and**
+  the host's `/etc/machine-id` (would require root on the node).
+- Kubelet bugs or out-of-band manipulation of
+  `Node.status.nodeInfo.machineID`. The kubelet writes this field; if
+  the cluster trusts a misbehaving kubelet, identity verification has
+  no anchor.
+
+Both gaps are out of scope for a node-side agent; remediating them
+requires a stronger node-attestation primitive (TPM-backed identity,
+SPIFFE SVIDs, etc.) — not in this controller's scope.
+
+---
+
 ## Related
 
 - [Machine Lifecycle](./machine-lifecycle.md) — full phase state machine including `EmergencyRemove`

src/bin/reclaim_agent.rs

@@ -39,10 +39,11 @@ use std::path::{Path, PathBuf};
 use std::sync::Arc;
 use std::time::Duration;
 
-use anyhow::{anyhow, Context as _, Result};
+use anyhow::{anyhow, bail, Context as _, Result};
 use clap::Parser;
 use five_spot::reclaim_agent::{
-    already_requested, build_patch_body, configmap_to_config, scan_proc, Config, Match,
+    already_requested, build_patch_body, compare_machine_ids, configmap_to_config,
+    read_host_machine_id, scan_proc, Config, Match,
 };
 use futures::StreamExt;
 use k8s_openapi::api::core::v1::{ConfigMap, Node};
@@ -57,6 +58,11 @@ use tracing::{debug, error, info, warn};
 /// Default path the agent reads as `/proc`. Overridable for testing.
 const DEFAULT_PROC_ROOT: &str = "/proc";
 
+/// Default path the agent reads for the host machine-id. Mounted into
+/// the DaemonSet via a single-file hostPath; the file is set at host
+/// boot by `systemd-machine-id-setup` (or kairos / k0s-installer).
+const DEFAULT_MACHINE_ID_PATH: &str = "/etc/machine-id";
+
 /// Field manager name used on PATCH. Distinct from the main controller
 /// so audit logs can tell apart a controller-side write from an
 /// agent-side write.
@@ -89,6 +95,29 @@ struct Cli {
     /// for one-shot invocations and for smoke tests.
     #[clap(long)]
     oneshot: bool,
+
+    /// Path to the host machine-id file. Default `/etc/machine-id`. Mounted
+    /// into the DaemonSet via a single-file hostPath; override only for
+    /// tests / sandboxes where the file lives elsewhere.
+    #[clap(long, env = "MACHINE_ID_PATH", default_value = DEFAULT_MACHINE_ID_PATH)]
+    machine_id_path: PathBuf,
+
+    /// Skip the host-identity cross-check before patching the Node.
+    ///
+    /// Default: false (strict). Before each PATCH the agent fetches the
+    /// target Node, reads its `status.nodeInfo.machineID`, and refuses to
+    /// proceed if it does not match `/etc/machine-id` from the agent's
+    /// host. This closes the "modified DaemonSet hard-codes NODE_NAME"
+    /// impersonation vector documented in Phase 4 of the 2026-04-25
+    /// security audit roadmap.
+    ///
+    /// Set to true ONLY for environments where `/etc/machine-id` is
+    /// genuinely unavailable (containers without the host file mounted,
+    /// dev sandboxes, kubelet variants that do not populate
+    /// `status.nodeInfo.machineID`). The strict default is the safe
+    /// posture for production.
+    #[clap(long, env = "SKIP_HOST_ID_CHECK", default_value_t = false)]
+    skip_host_id_check: bool,
 }
 
 #[tokio::main]
@@ -105,6 +134,30 @@ async fn main() -> Result<()> {
         .context("build in-cluster kube client")?;
     let nodes: Api<Node> = Api::all(client.clone());
 
+    // Read host machine-id once at startup. Failing here (file missing,
+    // empty, etc.) is fatal in strict mode so an operator notices and
+    // either fixes the mount or sets --skip-host-id-check explicitly.
+    let host_machine_id: Option<String> = if cli.skip_host_id_check {
+        warn!(
+            machine_id_path = %cli.machine_id_path.display(),
+            "--skip-host-id-check set: agent will trust NODE_NAME without verifying \
+             /etc/machine-id. Use only when the file is genuinely unavailable."
+        );
+        None
+    } else {
+        let id = read_host_machine_id(&cli.machine_id_path).with_context(|| {
+            format!(
+                "read host machine-id from {} (set --skip-host-id-check=true to bypass)",
+                cli.machine_id_path.display()
+            )
+        })?;
+        info!(
+            machine_id_path = %cli.machine_id_path.display(),
+            "host machine-id loaded; will cross-check against Node.status.nodeInfo.machineID before each patch"
+        );
+        Some(id)
+    };
+
     if is_already_requested(&nodes, &cli.node_name).await? {
         info!(node = %cli.node_name, "reclaim annotation already present — exiting idempotently");
         return Ok(());
@@ -132,7 +185,15 @@ async fn main() -> Result<()> {
     // resubscribe inside `kube::runtime::watcher`.
     let watcher_handle = tokio::spawn(run_config_watcher(client, cm_name, tx));
 
-    let scanner_result = run_scanner(&nodes, &cli.node_name, &cli.proc_root, rx, cli.oneshot).await;
+    let scanner_result = run_scanner(
+        &nodes,
+        &cli.node_name,
+        &cli.proc_root,
+        rx,
+        cli.oneshot,
+        host_machine_id.as_deref(),
+    )
+    .await;
     watcher_handle.abort();
     scanner_result
 }
@@ -239,6 +300,7 @@ async fn run_scanner(
     proc_root: &Path,
     mut rx: watch::Receiver<Option<Config>>,
     oneshot: bool,
+    host_machine_id: Option<&str>,
 ) -> Result<()> {
     loop {
         let cfg = rx.borrow().clone();
@@ -264,7 +326,7 @@ async fn run_scanner(
             Some(cfg) => match scan_proc(proc_root, &cfg) {
                 Ok(Some(m)) => {
                     info!(pid = m.pid, pattern = %m.matched_pattern, "match → annotating node");
-                    annotate_node(nodes, node_name, &m).await?;
+                    annotate_node(nodes, node_name, &m, host_machine_id).await?;
                     return Ok(());
                 }
                 Ok(None) => {
@@ -283,7 +345,36 @@ async fn run_scanner(
     }
 }
 
-async fn annotate_node(nodes: &Api<Node>, node_name: &str, m: &Match) -> Result<()> {
+/// PATCH the Node with reclaim annotations.
+///
+/// Before the PATCH (when `host_machine_id` is `Some`) the agent fetches
+/// the target Node and cross-checks `status.nodeInfo.machineID` against
+/// the host machine-id loaded at startup — refusing to patch a Node
+/// that does not match. This blocks the
+/// "modified-DaemonSet → impersonate-victim-Node" attack documented in
+/// Phase 4 of the 2026-04-25 security audit roadmap.
+///
+/// `host_machine_id == None` means the operator passed
+/// `--skip-host-id-check`; the function falls back to the pre-Phase-4
+/// behaviour of trusting `NODE_NAME` blindly.
+async fn annotate_node(
+    nodes: &Api<Node>,
+    node_name: &str,
+    m: &Match,
+    host_machine_id: Option<&str>,
+) -> Result<()> {
+    if let Some(expected) = host_machine_id {
+        let node = nodes
+            .get(node_name)
+            .await
+            .with_context(|| format!("fetch Node/{node_name} for host-identity check"))?;
+        if let Err(e) = compare_machine_ids(&node, node_name, expected) {
+            error!(error = %e, "host-identity check failed — refusing to patch Node");
+            bail!(e);
+        }
+        debug!(node = %node_name, "host-identity check passed");
+    }
+
     let ts = chrono::Utc::now().to_rfc3339();
     let patch = build_patch_body(m, &ts);
     let params = PatchParams::apply(FIELD_MANAGER).force();