Initial phase 1-3 of a per-node Kata config delivery

Closes: #77

Signed-off-by: Erick Bourgeois <erick@jeb.ca>

Author: ebourgeois

.claude/CHANGELOG.md

@@ -9,6 +9,251 @@ The format is based on the regulated environment requirements:
 
 ---
 
+## [2026-06-09 16:30] - Implement spec.kata workload-read delivery (ADR 0002)
+
+**Author:** Erick Bourgeois
+
+### Changed
+- `src/crd.rs`: rename `spec.kataConfigRef`/`KataConfigRef` → `spec.kata`/`KataConfig`;
+  add `namespace` field (workload-cluster target, default `5spot-system`) with
+  schema + default; docs reflect workload-cluster resolution.
+- `src/reconcilers/helpers.rs`: replace the per-node ConfigMap projection
+  (`reconcile_kata_config_provision`, `build_kata_config_configmap`,
+  `per_node_kata_configmap_name`, `extract_*`, `resolve_kata_config_content`)
+  with `reconcile_kata_config_delivery` (read-only workload `get_opt` of the
+  source object + namespace probe → Node opt-in label/annotation, or fail-fast
+  `SourceNotFound`/`TargetNamespaceMissing`), `build_kata_config_ref_annotation_patch`,
+  and the `KataDeliveryOutcome` enum. No controller ConfigMap/Secret writes.
+- `src/reconcilers/scheduled_machine.rs`: call site reads on the workload client,
+  logs the fail-fast outcome.
+- `src/constants.rs`: add `KATA_CONFIG_REF_ANNOTATION`; drop the
+  projection-only `KATA_CONFIG_CONFIGMAP_PREFIX` / `KATA_CONFIG_DATA_KEY` /
+  `KATA_CONFIG_FIELD_MANAGER`.
+- `src/crd_tests.rs`, `src/reconcilers/helpers_tests.rs`: migrate tests to the
+  new contract (namespace default/override, annotation builders, mock-API
+  delivery tests for present / source-missing / namespace-missing / disabled);
+  remove obsolete projection tests. 555 lib tests pass.
+- `src/bin/crddoc.rs`: regenerate API docs text for `spec.kata`.
+- `deploy/crds/scheduledmachine.yaml`, `docs/src/reference/api.md`: regenerated.
+- `deploy/deployment/rbac/clusterrole.yaml`: kata is read-only (`configmaps: get`
+  + new `namespaces: get`); the `configmaps` write verbs are re-attributed to the
+  reclaim-agent projection (the honest KSV-0049 owner). `.trivyignore`: re-add
+  `AVD-KSV-0049` attributed to reclaim.
+- `docs/architecture/calm/architecture.json`: kata flow updated to read-not-project;
+  `make calm-validate` clean; diagrams regenerated.
+- `docs/adr/0002-kata-config-delivery-via-spec-kata.md`: §5/Consequences corrected
+  — kata adds no write privilege but does NOT eliminate KSV-0049 (reclaim still
+  holds it).
+
+### Why
+Implements the approved ADR 0002 redesign: 5-Spot resolves the Flux-delivered
+kata object on the workload cluster and opts the Node in (or fails fast with a
+status reason), instead of projecting per-node ConfigMaps. Removes duplication,
+the controller-owned sync burden, and kata's contribution to the ConfigMap write
+grant.
+
+### Impact
+- [x] Breaking change (CRD field `spec.kataConfigRef` → `spec.kata`; unreleased)
+- [ ] Requires cluster rollout
+- [ ] Config change only
+- [ ] Documentation only
+
+## [2026-06-09 15:00] - Redesign kata config delivery (ADR 0002): workload-read, no writes
+
+**Author:** Erick Bourgeois
+
+### Changed
+- `docs/adr/0002-kata-config-delivery-via-spec-kata.md`: rewritten (and renamed
+  from `0002-per-node-kata-config-delivery.md`) to the accepted design — replace
+  `spec.kataConfigRef` with `spec.kata`; the controller **resolves** the kata
+  object on the workload cluster (via `kubeconfig-<clusterName>`) and either opts
+  the Node in (label + reference annotation) or **fails fast** with a status
+  reason (`SourceNotFound` / `TargetNamespaceMissing`). 5-Spot performs **no**
+  ConfigMap/Secret writes and **no** namespace creation for kata delivery; the
+  config must pre-exist (Flux-delivered). Target namespace defaults to
+  `5spot-system`, overridable. KSV-0049 eliminated by construction.
+- `docs/adr/0003-in-pod-host-service-restart-via-nsenter.md`: aligned with the
+  above — controller reads (not projects); agent consumes the object via the
+  workload kube API keyed by Node annotation (no ConfigMap volume); `spec.kata`.
+- `docs/adr/README.md`: index reflects the consolidated set (0001–0003).
+- Removed the two intermediate, never-committed-to-main ADRs (the per-node
+  `spec.kataConfigRef` projection and its KSV-0049 ownership/RBAC defense) and
+  reverted this session's per-node documentation (concept page, `.trivyignore`
+  `AVD-KSV-0049`, threat-model rows, `clusterrole.yaml` / `helpers.rs` comments).
+
+### Why
+Review against the platform model (tenants create nothing; Flux delivers config;
+a tenant namespace backs 1+ k0smotron control planes; the agent runs on the
+workload cluster the controller already has kubeconfig access to) rejected the
+per-node ConfigMap projection: wrong granularity, controller-owned duplication, a
+sync burden 5-Spot shouldn't own, and an unnecessary ConfigMap write grant. The
+accepted design makes 5-Spot a read-only resolver + opt-in labeler that fails
+fast when config is absent.
+
+### Impact
+- [ ] Breaking change
+- [ ] Requires cluster rollout
+- [ ] Config change only
+- [x] Documentation only (ADR/design; implementation tracked separately)
+
+## [2026-06-09 01:00] - Kata config delivery Phase 3 — node agent host-sync engine
+
+**Author:** Erick Bourgeois
+
+### Changed
+- `src/kata_config_agent.rs`: new host-filesystem sync engine — `sha256_hex`,
+  `file_sha256`, `SyncAction` + `decide_action` (pure decision), `atomic_write`
+  (temp + `rename`, mode 0644, parent-dir create, crash-safe), `remove_if_present`,
+  and `sync_once` → `SyncOutcome` (Wrote/Deleted/Unchanged). Drift self-heals.
+- `src/kata_config_agent_tests.rs`: 22 unit tests (SHA-256 vectors, decision
+  matrix, atomic write incl. mode/parent-dirs/no-temp-leak/crash-safety,
+  idempotent unlink, end-to-end sync incl. drift correction + GitOps delete).
+- `src/bin/kata_config_agent.rs`: new `5spot-kata-config-agent` binary — clap
+  CLI (`SOURCE_PATH`/`DEST_PATH`/`NODE_NAME`/`POLL_INTERVAL_SECS`/`--oneshot`)
+  driving a drift-watch poll loop over `sync_once`.
+- `src/lib.rs`: export `kata_config_agent` module.
+- `Cargo.toml`: add `sha2` dependency (already in the graph transitively) and
+  the `5spot-kata-config-agent` bin target.
+
+### Why
+Phase 3 of ADR 0003 — land the drop-in on the host and self-heal drift. The
+applied-hash node annotation and the `nsenter` host k0s-service restart are
+Phase 4; the DaemonSet manifest, agent RBAC, `.trivyignore`, and metrics land
+in Phase 4 with the final (privileged) security posture so they are decided as
+one unit. The drift-watch is a deliberate node-local poll — out-of-band host
+edits raise no Kubernetes event to watch on.
+
+### Impact
+- [ ] Breaking change
+- [ ] Requires cluster rollout
+- [ ] Config change only
+- [ ] Documentation only
+
+> New binary only; not yet deployed (no DaemonSet until Phase 4). No controller
+> or CRD behavior change. Verified end-to-end via `--oneshot`: write → no-op →
+> delete, parent dirs created, no temp-file leakage.
+
+---
+
+## [2026-06-09 00:00] - Kata config delivery Phase 2 — controller projection + RBAC
+
+**Author:** Erick Bourgeois
+
+### Changed
+- `src/constants.rs`: new kata-config constants — `KATA_CONFIG_LABEL`
+  (`5spot.finos.org/kata-config`), `KATA_CONFIG_LABEL_ENABLED`,
+  `KATA_CONFIG_NAMESPACE`, `KATA_CONFIG_CONFIGMAP_PREFIX`,
+  `KATA_CONFIG_DATA_KEY`, `KATA_CONFIG_FIELD_MANAGER`
+  (`5spot-controller-kata-config`).
+- `src/reconcilers/helpers.rs`: kata projection — pure builders
+  `per_node_kata_configmap_name`, `build_kata_config_configmap`,
+  `build_kata_config_label_patch`; content extractors
+  `extract_kata_content_from_configmap` / `_from_secret`; async
+  `resolve_kata_config_content` and `reconcile_kata_config_provision`
+  (resolve source → stamp label → SSA per-node ConfigMap; tear down on
+  `None`, 404-on-delete benign). Mirrors the reclaim-agent projection.
+- `src/reconcilers/helpers_tests.rs`: 18 new tests — pure builders/extractors
+  (happy/missing-key/non-UTF-8) plus five mock-client orchestrator tests
+  (ConfigMap source, Secret source w/ custom key, tear-down, 404-benign,
+  source-404 short-circuit).
+- `src/reconcilers/scheduled_machine.rs`: `provision_kata_config_best_effort`
+  wired into `handle_active_phase` beside reclaim/taints — child-cluster-scoped
+  writes, management-cluster source read, non-fatal.
+- `deploy/deployment/rbac/clusterrole.yaml`: new `configmaps`
+  get/create/patch/delete rule (first such rule); secrets rule comment notes
+  kataConfigRef source reads.
+
+### Why
+Phase 2 of ADR 0002 — project the referenced kata drop-in into a per-node
+ConfigMap on the child cluster and stamp the opt-in label, so the Phase 3 node
+agent can consume it. Implements the `rel-controller-kata-config-projection`
+relationship already modeled in CALM.
+
+### Impact
+- [ ] Breaking change
+- [x] Requires cluster rollout
+- [ ] Config change only
+- [ ] Documentation only
+
+> Apply the updated controller ClusterRole (new `configmaps` rule) before
+> rolling out the controller, or kata-config projection will fail with RBAC
+> Forbidden. Behavior is inert for ScheduledMachines without `kataConfigRef`.
+
+---
+
+## [2026-06-08 01:00] - Kata config delivery Phase 1 — CRD `spec.kataConfigRef`
+
+**Author:** Erick Bourgeois
+
+### Changed
+- `src/crd.rs`: new `kata_config_ref: Option<KataConfigRef>` field on
+  `ScheduledMachineSpec`; new `KataConfigRef` struct (kind/name/key/destPath/
+  restartService, `deny_unknown_fields`, in-namespace only) and
+  `KataConfigSourceKind` enum (`ConfigMap`/`Secret`); four field schema
+  functions and three default functions.
+- `src/crd_tests.rs`: 12 new TDD tests (serde defaults, round-trip, PascalCase
+  kind, rejects invalid kind / unknown fields, required kind+name, optional in
+  schema, name/destPath/restartService schema bounds). Existing spec literals
+  updated with `kata_config_ref: None`.
+- `src/reconcilers/{scheduled_machine_tests,helpers_tests,child_client_tests}.rs`,
+  `tests/integration_child_kubeconfig.rs`: spec literals updated with
+  `kata_config_ref: None`.
+- `src/bin/crddoc.rs`: documents the new `kataConfigRef` field + example.
+- `deploy/crds/scheduledmachine.yaml`, `docs/src/reference/api.md`: regenerated
+  (`make crds` → `make crddoc`).
+
+### Why
+Phase 1 of ADR 0002 — establish the `spec.kataConfigRef` CRD contract (types,
+schema bounds, serde defaults) test-first before the controller projection
+(Phase 2) and node agent (Phases 3–4). No behavioral/controller code yet.
+
+### Impact
+- [ ] Breaking change
+- [ ] Requires cluster rollout
+- [x] Config change only
+- [ ] Documentation only
+
+> CRD schema change (additive, optional field). Apply the regenerated
+> `deploy/crds/scheduledmachine.yaml` to pick up `kataConfigRef`; existing
+> ScheduledMachines are unaffected (field is optional, omitted when unset).
+
+---
+
+## [2026-06-08 00:00] - ADR + CALM: per-node Kata config delivery (design only)
+
+**Author:** Erick Bourgeois
+
+### Changed
+- `docs/adr/0002-per-node-kata-config-delivery.md`: new ADR (Proposed) — the
+  `spec.kataConfigRef` contract and controller-side per-node ConfigMap
+  projection, mirroring the reclaim-agent projection; in-namespace source only
+  (like `KubeconfigSecretRef`).
+- `docs/adr/0003-in-pod-host-service-restart-via-nsenter.md`: new ADR (Proposed)
+  — the privileged `5spot-kata-config-agent` DaemonSet that writes the host
+  drop-in and restarts the host k0s service via `nsenter -t 1 … systemctl
+  restart`, with the applied-hash annotation as the restart-loop guard.
+- `docs/adr/README.md`: index updated with ADR-0002 and ADR-0003.
+- `docs/architecture/calm/architecture.json`: modeled the feature — new nodes
+  `service-kata-config-agent` and `data-asset-kata-config-configmap`; new
+  relationships `rel-controller-kata-config-projection`,
+  `rel-kata-agent-workload-kube-api`, `rel-kata-agent-writes-host`,
+  `rel-kata-configmap-stored-in-workload-api`; new flow
+  `flow-kata-config-delivery`; `adrs` linked. `make calm-validate` passes;
+  `make calm-diagrams` renders.
+
+### Why
+ADD requires the decision to be recorded and modeled before code. This entry
+covers the ADR + CALM deliverables only; no Rust, CRD, or manifest changes yet —
+implementation (TDD) is gated on review of these two ADRs.
+
+### Impact
+- [ ] Breaking change
+- [ ] Requires cluster rollout
+- [ ] Config change only
+- [x] Documentation only
+
+---
+
 ## [2026-05-31 00:00] - Adopt Architecture Driven Development (ADD) methodology
 
 **Author:** Erick Bourgeois

.trivyignore

@@ -41,6 +41,23 @@ AVD-KSV-0048
 # which is impractical for a cluster-wide controller.
 AVD-KSV-0041
 
+# KSV-0049 — ClusterRole should not have create/update/patch/delete on
+# 'configmaps'.
+#
+# The `configmaps: create/patch/delete` verbs belong to the **reclaim-agent**
+# per-node ConfigMap projection (`reclaim-agent-<node>`), NOT to kata config
+# delivery (kata is read-only — `configmaps: get` — per ADR 0002). They are
+# exercised only in the degenerate single-cluster case where the resolved
+# client is the controller's own ServiceAccount; in true multi-cluster the
+# projection rides the `kubeconfig-<clusterName>` identity, not this token.
+# Scope is `get, create, patch, delete` — NOT `update`, `deletecollection`,
+# `impersonate`, or `*`. A ClusterRole cannot be scoped by name prefix or
+# namespace, so the grant is cluster-wide; the blast radius is bounded by
+# convention (the controller writes only the `reclaim-agent-*` names it owns,
+# via a dedicated SSA field manager). Fully removing the verbs would require
+# moving the reclaim projection off the controller SA — tracked separately.
+AVD-KSV-0049
+
 # ─────────────────────────────────────────────────────────────────────────────
 # Deployment — deploy/deployment/deployment.yaml
 # ─────────────────────────────────────────────────────────────────────────────

Cargo.lock

@@ -68,6 +68,9 @@ lazy_static = "1.5"
 # enabled explicitly.
 warp = { version = "0.4", default-features = false, features = ["server"] }
 toml = "1.1"
+# SHA-256 for the kata-config agent's content/drift hashing and (Phase 4)
+# the applied-hash node annotation. RustCrypto, pure-Rust, no system deps.
+sha2 = "0.10"
 
 # Linux-only — used by the reclaim-agent's rung 2 netlink proc connector
 # subscriber (`src/netlink_proc.rs`). The portable byte parsers in that
@@ -81,6 +84,10 @@ nix = { version = "0.31", default-features = false, features = ["socket", "net",
 name = "5spot-reclaim-agent"
 path = "src/bin/reclaim_agent.rs"
 
+[[bin]]
+name = "5spot-kata-config-agent"
+path = "src/bin/kata_config_agent.rs"
+
 [dev-dependencies]
 tokio-test = "0.4"
 mockall = "0.14"

Cargo.toml

@@ -134,6 +134,89 @@ spec:
                 - kind
                 - spec
                 type: object
+              kata:
+                description: |-
+                  Optional reference to a `Secret` or `ConfigMap` on the **workload
+                  cluster** holding a Kata containerd drop-in to deliver to the node(s) this
+                  resource owns.
+
+                  When set, the controller resolves the referenced object on the workload
+                  cluster (via the `kubeconfig-<clusterName>` Secret) in `kata.namespace`
+                  (default `5spot-system`). If it is present, the controller stamps the
+                  `5spot.finos.org/kata-config=enabled` opt-in label **and** a reference
+                  annotation on the backing Node; the `5spot-kata-config-agent` DaemonSet —
+                  scheduled onto labelled nodes — reads the object from the workload API,
+                  writes the drop-in to `destPath`, and restarts `restartService` so
+                  containerd reloads it. If the object (or its namespace) is absent, the
+                  controller does **not** label the Node and reports a fail-fast status
+                  condition — 5-Spot never creates the object (it must pre-exist,
+                  Flux-delivered).
+
+                  This is config *delivery*, not a Kata install: the `/opt/kata` binaries
+                  remain `kata-deploy`'s responsibility, and the existing
+                  `katacontainers.io/kata-runtime` opt-in label is unaffected. See ADR 0002
+                  and ADR 0003.
+                nullable: true
+                properties:
+                  destPath:
+                    default: /etc/k0s/container.d/kata-containers.toml
+                    description: |-
+                      Absolute host path the agent writes the drop-in to. Defaults to
+                      `/etc/k0s/container.d/kata-containers.toml` (the k0s containerd drop-in
+                      directory). Configurable because layouts vary (k0s-in-docker, kairos,
+                      vanilla containerd).
+                    maxLength: 4096
+                    minLength: 1
+                    pattern: ^/[^\u0000]*$
+                    type: string
+                  key:
+                    default: kata-containers.toml
+                    description: |-
+                      Key within the source's `data` map whose value is the drop-in content.
+                      Defaults to `kata-containers.toml`.
+                    maxLength: 253
+                    minLength: 1
+                    pattern: ^[A-Za-z0-9._-]+$
+                    type: string
+                  kind:
+                    description: Whether the source is a `ConfigMap` or a `Secret`. Required (no default).
+                    enum:
+                    - ConfigMap
+                    - Secret
+                    type: string
+                  name:
+                    description: |-
+                      Name of the source object on the workload cluster. RFC-1123 DNS subdomain
+                      (max 253 chars).
+                    maxLength: 253
+                    minLength: 1
+                    pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$
+                    type: string
+                  namespace:
+                    default: 5spot-system
+                    description: |-
+                      Workload-cluster namespace the agent reads the object from. Defaults to
+                      `5spot-system` (the agent's own namespace, so no cross-namespace agent
+                      RBAC is needed). Override to place config in a per-tenant namespace.
+                    maxLength: 63
+                    minLength: 1
+                    pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$
+                    type: string
+                  restartService:
+                    default: k0sworker.service
+                    description: |-
+                      systemd unit the node agent restarts (via `nsenter`) so containerd
+                      reloads the drop-in. Defaults to `k0sworker.service`; override with
+                      `k0scontroller.service` on single-node / controller-runs-workloads
+                      layouts.
+                    maxLength: 255
+                    minLength: 1
+                    pattern: ^[A-Za-z0-9@._-]+\.service$
+                    type: string
+                required:
+                - kind
+                - name
+                type: object
               killIfCommands:
                 description: |-
                   Optional list of process patterns that trigger an emergency node