Initial phase 1-3 of a per-node Kata config delivery

Signed-off-by: Erick Bourgeois <erick@jeb.ca>

Author: ebourgeois

.claude/CHANGELOG.md

@@ -9,6 +9,164 @@ The format is based on the regulated environment requirements:
 
 ---
 
+## [2026-06-09 01:00] - Kata config delivery Phase 3 — node agent host-sync engine
+
+**Author:** Erick Bourgeois
+
+### Changed
+- `src/kata_config_agent.rs`: new host-filesystem sync engine — `sha256_hex`,
+  `file_sha256`, `SyncAction` + `decide_action` (pure decision), `atomic_write`
+  (temp + `rename`, mode 0644, parent-dir create, crash-safe), `remove_if_present`,
+  and `sync_once` → `SyncOutcome` (Wrote/Deleted/Unchanged). Drift self-heals.
+- `src/kata_config_agent_tests.rs`: 22 unit tests (SHA-256 vectors, decision
+  matrix, atomic write incl. mode/parent-dirs/no-temp-leak/crash-safety,
+  idempotent unlink, end-to-end sync incl. drift correction + GitOps delete).
+- `src/bin/kata_config_agent.rs`: new `5spot-kata-config-agent` binary — clap
+  CLI (`SOURCE_PATH`/`DEST_PATH`/`NODE_NAME`/`POLL_INTERVAL_SECS`/`--oneshot`)
+  driving a drift-watch poll loop over `sync_once`.
+- `src/lib.rs`: export `kata_config_agent` module.
+- `Cargo.toml`: add `sha2` dependency (already in the graph transitively) and
+  the `5spot-kata-config-agent` bin target.
+
+### Why
+Phase 3 of ADR 0003 — land the drop-in on the host and self-heal drift. The
+applied-hash node annotation and the `nsenter` host k0s-service restart are
+Phase 4; the DaemonSet manifest, agent RBAC, `.trivyignore`, and metrics land
+in Phase 4 with the final (privileged) security posture so they are decided as
+one unit. The drift-watch is a deliberate node-local poll — out-of-band host
+edits raise no Kubernetes event to watch on.
+
+### Impact
+- [ ] Breaking change
+- [ ] Requires cluster rollout
+- [ ] Config change only
+- [ ] Documentation only
+
+> New binary only; not yet deployed (no DaemonSet until Phase 4). No controller
+> or CRD behavior change. Verified end-to-end via `--oneshot`: write → no-op →
+> delete, parent dirs created, no temp-file leakage.
+
+---
+
+## [2026-06-09 00:00] - Kata config delivery Phase 2 — controller projection + RBAC
+
+**Author:** Erick Bourgeois
+
+### Changed
+- `src/constants.rs`: new kata-config constants — `KATA_CONFIG_LABEL`
+  (`5spot.finos.org/kata-config`), `KATA_CONFIG_LABEL_ENABLED`,
+  `KATA_CONFIG_NAMESPACE`, `KATA_CONFIG_CONFIGMAP_PREFIX`,
+  `KATA_CONFIG_DATA_KEY`, `KATA_CONFIG_FIELD_MANAGER`
+  (`5spot-controller-kata-config`).
+- `src/reconcilers/helpers.rs`: kata projection — pure builders
+  `per_node_kata_configmap_name`, `build_kata_config_configmap`,
+  `build_kata_config_label_patch`; content extractors
+  `extract_kata_content_from_configmap` / `_from_secret`; async
+  `resolve_kata_config_content` and `reconcile_kata_config_provision`
+  (resolve source → stamp label → SSA per-node ConfigMap; tear down on
+  `None`, 404-on-delete benign). Mirrors the reclaim-agent projection.
+- `src/reconcilers/helpers_tests.rs`: 18 new tests — pure builders/extractors
+  (happy/missing-key/non-UTF-8) plus five mock-client orchestrator tests
+  (ConfigMap source, Secret source w/ custom key, tear-down, 404-benign,
+  source-404 short-circuit).
+- `src/reconcilers/scheduled_machine.rs`: `provision_kata_config_best_effort`
+  wired into `handle_active_phase` beside reclaim/taints — child-cluster-scoped
+  writes, management-cluster source read, non-fatal.
+- `deploy/deployment/rbac/clusterrole.yaml`: new `configmaps`
+  get/create/patch/delete rule (first such rule); secrets rule comment notes
+  kataConfigRef source reads.
+
+### Why
+Phase 2 of ADR 0002 — project the referenced kata drop-in into a per-node
+ConfigMap on the child cluster and stamp the opt-in label, so the Phase 3 node
+agent can consume it. Implements the `rel-controller-kata-config-projection`
+relationship already modeled in CALM.
+
+### Impact
+- [ ] Breaking change
+- [x] Requires cluster rollout
+- [ ] Config change only
+- [ ] Documentation only
+
+> Apply the updated controller ClusterRole (new `configmaps` rule) before
+> rolling out the controller, or kata-config projection will fail with RBAC
+> Forbidden. Behavior is inert for ScheduledMachines without `kataConfigRef`.
+
+---
+
+## [2026-06-08 01:00] - Kata config delivery Phase 1 — CRD `spec.kataConfigRef`
+
+**Author:** Erick Bourgeois
+
+### Changed
+- `src/crd.rs`: new `kata_config_ref: Option<KataConfigRef>` field on
+  `ScheduledMachineSpec`; new `KataConfigRef` struct (kind/name/key/destPath/
+  restartService, `deny_unknown_fields`, in-namespace only) and
+  `KataConfigSourceKind` enum (`ConfigMap`/`Secret`); four field schema
+  functions and three default functions.
+- `src/crd_tests.rs`: 12 new TDD tests (serde defaults, round-trip, PascalCase
+  kind, rejects invalid kind / unknown fields, required kind+name, optional in
+  schema, name/destPath/restartService schema bounds). Existing spec literals
+  updated with `kata_config_ref: None`.
+- `src/reconcilers/{scheduled_machine_tests,helpers_tests,child_client_tests}.rs`,
+  `tests/integration_child_kubeconfig.rs`: spec literals updated with
+  `kata_config_ref: None`.
+- `src/bin/crddoc.rs`: documents the new `kataConfigRef` field + example.
+- `deploy/crds/scheduledmachine.yaml`, `docs/src/reference/api.md`: regenerated
+  (`make crds` → `make crddoc`).
+
+### Why
+Phase 1 of ADR 0002 — establish the `spec.kataConfigRef` CRD contract (types,
+schema bounds, serde defaults) test-first before the controller projection
+(Phase 2) and node agent (Phases 3–4). No behavioral/controller code yet.
+
+### Impact
+- [ ] Breaking change
+- [ ] Requires cluster rollout
+- [x] Config change only
+- [ ] Documentation only
+
+> CRD schema change (additive, optional field). Apply the regenerated
+> `deploy/crds/scheduledmachine.yaml` to pick up `kataConfigRef`; existing
+> ScheduledMachines are unaffected (field is optional, omitted when unset).
+
+---
+
+## [2026-06-08 00:00] - ADR + CALM: per-node Kata config delivery (design only)
+
+**Author:** Erick Bourgeois
+
+### Changed
+- `docs/adr/0002-per-node-kata-config-delivery.md`: new ADR (Proposed) — the
+  `spec.kataConfigRef` contract and controller-side per-node ConfigMap
+  projection, mirroring the reclaim-agent projection; in-namespace source only
+  (like `KubeconfigSecretRef`).
+- `docs/adr/0003-in-pod-host-service-restart-via-nsenter.md`: new ADR (Proposed)
+  — the privileged `5spot-kata-config-agent` DaemonSet that writes the host
+  drop-in and restarts the host k0s service via `nsenter -t 1 … systemctl
+  restart`, with the applied-hash annotation as the restart-loop guard.
+- `docs/adr/README.md`: index updated with ADR-0002 and ADR-0003.
+- `docs/architecture/calm/architecture.json`: modeled the feature — new nodes
+  `service-kata-config-agent` and `data-asset-kata-config-configmap`; new
+  relationships `rel-controller-kata-config-projection`,
+  `rel-kata-agent-workload-kube-api`, `rel-kata-agent-writes-host`,
+  `rel-kata-configmap-stored-in-workload-api`; new flow
+  `flow-kata-config-delivery`; `adrs` linked. `make calm-validate` passes;
+  `make calm-diagrams` renders.
+
+### Why
+ADD requires the decision to be recorded and modeled before code. This entry
+covers the ADR + CALM deliverables only; no Rust, CRD, or manifest changes yet —
+implementation (TDD) is gated on review of these two ADRs.
+
+### Impact
+- [ ] Breaking change
+- [ ] Requires cluster rollout
+- [ ] Config change only
+- [x] Documentation only
+
+---
+
 ## [2026-05-31 00:00] - Adopt Architecture Driven Development (ADD) methodology
 
 **Author:** Erick Bourgeois

Cargo.lock

@@ -68,6 +68,9 @@ lazy_static = "1.5"
 # enabled explicitly.
 warp = { version = "0.4", default-features = false, features = ["server"] }
 toml = "1.1"
+# SHA-256 for the kata-config agent's content/drift hashing and (Phase 4)
+# the applied-hash node annotation. RustCrypto, pure-Rust, no system deps.
+sha2 = "0.10"
 
 # Linux-only — used by the reclaim-agent's rung 2 netlink proc connector
 # subscriber (`src/netlink_proc.rs`). The portable byte parsers in that
@@ -81,6 +84,10 @@ nix = { version = "0.31", default-features = false, features = ["socket", "net",
 name = "5spot-reclaim-agent"
 path = "src/bin/reclaim_agent.rs"
 
+[[bin]]
+name = "5spot-kata-config-agent"
+path = "src/bin/kata_config_agent.rs"
+
 [dev-dependencies]
 tokio-test = "0.4"
 mockall = "0.14"

Cargo.toml

@@ -134,6 +134,78 @@ spec:
                 - kind
                 - spec
                 type: object
+              kataConfigRef:
+                description: |-
+                  Optional reference to a `Secret` or `ConfigMap` (in this
+                  ScheduledMachine's own namespace) holding a Kata containerd drop-in to
+                  deliver to the node(s) this resource owns.
+
+                  When set, the controller projects the referenced content into a per-node
+                  ConfigMap (`kata-config-<node-name>`) in the `5spot-system` namespace on
+                  the workload cluster and stamps the `5spot.finos.org/kata-config=enabled`
+                  opt-in label on the backing Node. The `5spot-kata-config-agent`
+                  DaemonSet — scheduled onto labelled nodes — writes the drop-in to the
+                  configured `destPath` on the host and restarts `restartService` so
+                  containerd reloads it. Clearing the field (or deleting the SM) removes
+                  the ConfigMap, the label, and the host file (GitOps semantics).
+
+                  This is config *delivery*, not a Kata install: the `/opt/kata` binaries
+                  remain `kata-deploy`'s responsibility, and the existing
+                  `katacontainers.io/kata-runtime` opt-in label is unaffected. Cross-
+                  namespace sources are NOT supported (security boundary). See ADR 0002 and
+                  ADR 0003.
+                nullable: true
+                properties:
+                  destPath:
+                    default: /etc/k0s/container.d/kata-containers.toml
+                    description: |-
+                      Absolute host path the agent writes the drop-in to. Defaults to
+                      `/etc/k0s/container.d/kata-containers.toml` (the k0s containerd drop-in
+                      directory). Configurable because layouts vary (k0s-in-docker, kairos,
+                      vanilla containerd).
+                    maxLength: 4096
+                    minLength: 1
+                    pattern: ^/[^\u0000]*$
+                    type: string
+                  key:
+                    default: kata-containers.toml
+                    description: |-
+                      Key within the source's `data` map whose value is the drop-in content.
+                      Defaults to `kata-containers.toml`.
+                    maxLength: 253
+                    minLength: 1
+                    pattern: ^[A-Za-z0-9._-]+$
+                    type: string
+                  kind:
+                    description: Whether the source is a `ConfigMap` or a `Secret`. Required (no default).
+                    enum:
+                    - ConfigMap
+                    - Secret
+                    type: string
+                  name:
+                    description: |-
+                      Name of the source object. RFC-1123 DNS subdomain (max 253 chars). The
+                      object MUST live in the same namespace as the ScheduledMachine — there
+                      is no `namespace` field by design.
+                    maxLength: 253
+                    minLength: 1
+                    pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$
+                    type: string
+                  restartService:
+                    default: k0sworker.service
+                    description: |-
+                      systemd unit the node agent restarts (via `nsenter`) so containerd
+                      reloads the drop-in. Defaults to `k0sworker.service`; override with
+                      `k0scontroller.service` on single-node / controller-runs-workloads
+                      layouts.
+                    maxLength: 255
+                    minLength: 1
+                    pattern: ^[A-Za-z0-9@._-]+\.service$
+                    type: string
+                required:
+                - kind
+                - name
+                type: object
               killIfCommands:
                 description: |-
                   Optional list of process patterns that trigger an emergency node

deploy/crds/scheduledmachine.yaml

@@ -61,11 +61,31 @@ rules:
     resources: ["events"]
     verbs: ["create", "patch"]
 
-  # Secrets for SSH keys and bootstrap data
+  # Secrets for SSH keys and bootstrap data.
+  #
+  # Read-only: also covers reading a kataConfigRef source Secret in the
+  # ScheduledMachine namespace (ADR 0002). The controller never writes
+  # Secrets, so no create/patch/delete here.
   - apiGroups: [""]
     resources: ["secrets"]
     verbs: ["get", "list", "watch"]
 
+  # ConfigMaps for kata-config delivery (ADR 0002).
+  #
+  # `get` reads a kataConfigRef source ConfigMap in the ScheduledMachine
+  # namespace. `create`/`patch`/`delete` project (server-side apply) and
+  # tear down the per-node `kata-config-<node>` ConfigMap in 5spot-system.
+  # In multi-cluster deployments these writes target the child cluster via
+  # the resolved kubeconfig (whose own RBAC must permit them); this rule
+  # additionally covers the degenerate single-cluster case where the
+  # management client is also the workload client. A ClusterRole cannot be
+  # scoped by name prefix or namespace, so the verbs are cluster-wide — the
+  # blast radius is bounded by the controller writing only to the
+  # `kata-config-*` names it owns via the dedicated SSA field manager.
+  - apiGroups: [""]
+    resources: ["configmaps"]
+    verbs: ["get", "create", "patch", "delete"]
+
   # Nodes for drain operations and reclaim-agent label / taint management.
   #
   # `update` was removed in Phase 5 of the 2026-04-25 security audit