CHANGELOG.md

# Changelog

All notable changes to the 5spot project will be documented in this file.

The format is based on the regulated environment requirements:
- **Author attribution is MANDATORY** for all entries
- Changes are logged in reverse chronological order
- Each entry must include impact assessment

---

## [2026-06-26 10:00] - Switch rustls crypto provider ring -> aws-lc-rs (fixes mTLS to k0smotron-hosted control planes)

**Author:** Erick Bourgeois

### Changed
- `Cargo.toml`: `kube` now uses `default-features = false` + `["…, "rustls-tls", "aws-lc-rs"]`
  instead of the default that pulls the `ring` crypto provider. `ring` leaves the lock; the
  rustls TLS stack now runs on aws-lc-rs (BoringSSL-derived). 682 lib tests pass, fmt/clippy clean.

### Why
5-Spot's child client could not complete the mTLS TLS-1.3 handshake against a k0smotron-hosted
control plane — it stalled after the client-certificate step and timed out
(`client error (Connect) -> deadline has elapsed`), so node taints/drains never happened on the
hosted-CP tier. Proven on a live cluster that `curl --cert/--key` (OpenSSL) and `client-go`
(kubelet/kubectl) complete the *identical* handshake to the *same* endpoint in ~13ms, while
kube-rs/rustls-with-`ring` hangs (captured via `rustls=trace`). aws-lc-rs shares OpenSSL/BoringSSL's
crypto lineage, so it behaves like the clients that work.

### Build note
CI builds **natively per arch** (`ubuntu-24.04` + `ubuntu-24.04-arm`, plain `cargo build --release`),
so aws-lc-rs compiles natively (cmake + cc, present on GitHub runners) — no cross-toolchain and no
OpenSSL runtime lib in the distroless image, i.e. it avoids the cross-compile problem that motivated
`ring` in the first place.

### Impact
- [ ] Breaking change
- [x] Requires cluster rollout
- [ ] Config change only
- [ ] Documentation only

## [2026-06-24 11:30] - Name the HTTP status codes used in API-error matching (no magic 404/409/429)

**Author:** Erick Bourgeois

### Changed
- `src/constants.rs`: new `HTTP_ALREADY_EXISTS` (409), `HTTP_NOT_FOUND` (404) and
  `HTTP_TOO_MANY_REQUESTS` (429), each defined as `http::StatusCode::<VARIANT>.as_u16()`
  so the numeric literal lives only in the `http` crate, never in 5-Spot source. +1 unit
  test pinning each constant to the correct `StatusCode` variant.
- `src/reconcilers/{helpers,child_client,scheduled_machine}.rs`: replaced all 9
  `e.code == 404|409|429` match guards with the named constants.
- `Cargo.toml`: promoted `http` from `[dev-dependencies]` to `[dependencies]` (already
  in the tree transitively; now used by `src/constants.rs`).

### Why
`constants.rs` mandates "no magic numbers in source files". The idempotent-create fix
had introduced `ae.code == 409`, and pre-existing `== 404` / `== 429` guards were latent
violations of the same rule. Naming them centrally makes the branches self-documenting
and keeps the policy globally enforced.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [x] Refactor (no behavior change)

## [2026-06-24 11:00] - Silence RUSTSEC-2026-0173 (proc-macro-error2 unmaintained) with justification

**Author:** Erick Bourgeois

### Changed
- `.cargo/audit.toml` and `deny.toml`: ignore `RUSTSEC-2026-0173` with a documented
  justification (kept in sync across both tools).

### Why
CI `cargo audit` started failing on `proc-macro-error2` being **unmaintained** (a
maintenance-status advisory, not a vulnerability). It is reached only transitively via
`jiff 0.2.29 -> defmt -> defmt-macros`, and `defmt` is jiff's optional embedded/no_std
feature — `cargo tree -e features -i defmt` prints "nothing to print" for the host
target, so the crate is never compiled into the 5-Spot binary; it exists only as a
`Cargo.lock` entry. No runtime or build exposure. Removal awaits upstream (kube bumping
jiff, or defmt-macros dropping proc-macro-error2). Verified `cargo audit` passes.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [x] Config change only
- [ ] Documentation only

## [2026-06-24 10:30] - Idempotency sweep: 409 AlreadyExists / duplicate finalizers no longer Error the ScheduledMachine

**Author:** Erick Bourgeois

### Changed
- `src/reconcilers/helpers.rs` (`create_dynamic_resource`): treat a `409 AlreadyExists`
  from `api.create(...)` as success instead of propagating it. The bootstrap config,
  infrastructure object, and CAPI Machine are all created through this helper; a prior
  reconcile having created them IS the desired state.
- `src/reconcilers/helpers.rs` (`add_finalizer`): only append our finalizer if it isn't
  already present — a double-call can no longer write a duplicate (which would wedge GC).
- `src/reconcilers/helpers_tests.rs`: +3 unit tests (create 409 → Ok; create 500 still
  propagates; add_finalizer writes exactly one copy).
- Audited the rest of the codebase for idempotency and confirmed it already holds: the
  only other `.create()` is a `SelfSubjectAccessReview` (a virtual auth check, never
  409s); all `.delete()` sites swallow 404; the kata ConfigMap uses `Patch::Apply` (SSA);
  Events go through the kube `recorder`; status/spec writes use `Patch::Merge`/`Apply`.

### Why
On every reconcile after the first, `add_machine_to_cluster` re-issues the create and
got `409 AlreadyExists` (e.g. `k0sworkerconfigs "…" already exists`). That bubbled up as
`MachineCreationFailed`, flipping the ScheduledMachine into the **Error** phase — which
skips ALL Active-phase work, including node-taint reconciliation. On the k0smotron
workshop tier this presented as **Flag 1 (worker joins) passing but Flag 2 (the Node
carries the spot taint) never happening**: the SM was wedged in Error, so it never ran
`handle_active_phase`. Idempotent create is standard controller behavior and fixes it.

### Impact
- [ ] Breaking change
- [x] Requires cluster rollout
- [ ] Config change only
- [ ] Documentation only

## [2026-06-24 03:10] - Fix Mermaid flow diagrams broken by special characters

**Author:** Erick Bourgeois

### Changed
- `docs/architecture/calm/architecture.json`: escaped Mermaid-hostile characters in
  three flow transition descriptions using Mermaid entity codes (which render as the
  literal characters):
  - `flow-emergency-reclaim` t1: `reclaim-requested="true"` → `reclaim-requested=#quot;true#quot;`.
    The embedded double quotes terminated the Mermaid label string early, causing a parse
    error that left the **Emergency Reclaim** diagram blank on the docs site.
  - `flow-kata-config-delivery` t2: `/etc/5spot/kata-config/<key>` → `…/#lt;key#gt;`.
  - `flow-kata-config-delivery` t5: `restart <restartService>` → `restart #lt;restartService#gt;`.
    Raw `<…>` placeholders were parsed as HTML tags and silently dropped from the rendered label.
- `docs/src/architecture/flows.md`: regenerated via `make calm-diagrams` (auto-generated;
  not hand-edited).
- `docs/src/concepts/kata-config-delivery.md`: hand-written `sequenceDiagram` — replaced a
  `;` with `,` inside a `Note over Agent,Host: …` label. Mermaid treats `;` as a statement
  separator even inside note text, so it split the note mid-string and failed the whole
  diagram to parse.

### Why
The `flows.md.hbs` template emits transition descriptions verbatim into `flowchart` node
labels (`t1["…"]`). Mermaid cannot contain an unescaped `"` inside a quoted label, and
treats `<…>` as HTML — so these characters either broke parsing (Emergency Reclaim) or
ate label text (Kata Config Delivery). Fixing at the source-of-truth `architecture.json`
keeps the diagrams correct after every regeneration. The `phase -> …` arrows in other
flows contain only `>` and render fine, so they were left unchanged.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [x] Documentation only

---

## [2026-06-24 02:30] - Surface the real cause of child-cluster connection failures (error_chain)

**Author:** Erick Bourgeois

### Changed
- `src/reconcilers/helpers.rs`: new `error_chain(&dyn Error)` helper that flattens an
  error's `source()` chain into one string, and wired it into all six child-cluster
  failure logs (taint GET/PATCH/reconcile, kata clear/stamp). These previously logged
  only the terse top-level `Display` ("client error (Connect)"), hiding the real cause.
- `src/reconcilers/helpers_tests.rs`: +2 unit tests for `error_chain` (multi-level
  source chain; single error with no source).

### Why
A k0smotron-hosted control plane (workshop k0smotron tier) exposes a self-signed API
server cert that **rustls** (5-Spot's TLS backend, kube 3.1 default) rejects where
OpenSSL/curl `-k` accept it — so 5-Spot's child client fails to taint/drain the worker
Node with `client error (Connect)`, with no visible cause. Logging the full chain makes
the exact rustls reason (unknown issuer / SAN mismatch / unsupported signature) visible
so the fix can target it (a proper cert, `tls_server_name`, or `insecure-skip-tls-verify`
on the child kubeconfig — kube already honors the latter via `Config::accept_invalid_certs`).

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [x] Diagnostics only (no behavior change)

## [2026-06-23 11:45] - Ship the spot-schedule provider binaries in the image (ADR 0009 providers were unrunnable)

**Author:** Erick Bourgeois

### Changed
- `Cargo.toml`: added explicit `[[bin]]` entries for `spot-schedule-time-based`
  (`src/bin/spot_schedule_time_based.rs`) and `spot-schedule-capital-markets`
  (`src/bin/spot_schedule_capital_markets.rs`). They were previously only
  auto-discovered under their underscore file-stem names, which did not match the
  hyphenated `command:` in their deploy manifests.
- `Dockerfile`, `Dockerfile.chainguard`: also `COPY` `spot-schedule-time-based` and
  `spot-schedule-capital-markets` into the image (alongside `/5spot`). The main image
  is now multi-binary; `ENTRYPOINT` stays `/5spot` and providers select via `command:`.
- `Makefile` (`prepare-binaries-linux-amd64`, `-arm64`): copy the two provider binaries
  into `binaries/<arch>/` so the Docker build can pick them up.
- `.github/workflows/build.yaml` (Build job): the artifact upload (PR/push + release)
  now includes `spot-schedule-time-based` and `spot-schedule-capital-markets` — CI
  uploads only the named binary, so without this the providers never reach the Docker
  job (the multi-arch build failed with `binaries/arm64/spot-schedule-time-based: not
  found`). `cargo build --release` already builds all `[[bin]]`s.
- `.github/actions/prepare-docker-binaries/action.yaml`: copy the two provider binaries
  from the downloaded artifacts into `binaries/<arch>/` for both arches (the local
  `make prepare-binaries` is not used by CI — this composite action is).
- `deploy/spot-schedule-providers/{time-based,capital-markets}/deployment.yaml`:
  `command` changed to absolute paths (`/spot-schedule-time-based`,
  `/spot-schedule-capital-markets`) — the distroless image has no PATH entry for a bare
  name; the binaries live at `/`.
- `docs/src/guides/time-based-schedule.md`, `docs/src/guides/capital-markets-schedule.md`,
  `docs/src/installation/controller.md`: documented that each provider is a separate,
  **required** Deployment shipped inside the main image (selected by `command:`), and
  that without a provider running no `ScheduledMachine` referencing a schedule activates.

### Why
The spot-schedule providers (ADR 0009) had source and deploy manifests but were never
wired into the build: the binaries were not declared with their deployed names, were
not copied by `prepare-binaries`, and were not in the Dockerfiles. The published image
therefore could not run them — a deployed provider crash-looped with
`exec: "spot-schedule-time-based": executable file not found`, leaving every
`ScheduledMachine` stuck (no `status.active` is ever written). Surfaced while bringing
up the k0smotron workshop tier on a v1beta1 image.

### Impact
- [ ] Breaking change
- [x] Requires cluster rollout
- [ ] Config change only
- [ ] Documentation only (code + docs)

Note: the next image build is required before the provider Deployments can run; their
`:v0.1.0` placeholder tags are pinned at release by `make set-image-version`.

## [2026-06-22 12:00] - CAPI Machine API version: passthrough from bootstrapSpec + mandatory discovery (no hardcoded version)

**Author:** Erick Bourgeois

### Changed
- `src/constants.rs`: removed the hardcoded `CAPI_MACHINE_API_VERSION` /
  `CAPI_MACHINE_API_VERSION_FULL` consts. The version is now a runtime value
  (`OnceLock`) set once at startup via `set_capi_machine_api_version`; the accessor
  `capi_machine_api_version()` panics if read before resolution (an invariant —
  `main` resolves before any reconcile) and returns `v1beta1` only under
  `#[cfg(test)]` for fixtures. Group/kind (`cluster.x-k8s.io`/`Machine`) stay fixed
  by the CAPI contract; only the version is data-driven.
- `src/main.rs`: `resolve_capi_machine_version` now returns `Result` and discovery
  is **mandatory** — precedence is `CAPI_MACHINE_API_VERSION` env override →
  `kube::discovery` of the served `machines.cluster.x-k8s.io` version → error (6×5s
  retry, then the controller exits and is restarted by Kubernetes). No silent
  default. The Machine watch uses the resolved version.
- `src/reconcilers/helpers.rs`: new `machine_api_version_for` helper derives the
  Machine version from the version component of the user's `bootstrapSpec.apiVersion`
  (passthrough), falling back to the discovered served version. Machine **create**
  and **delete** both use it; **watch**/**fetch** use the discovered version (CAPI
  conversion webhooks bridge any mismatch). `extract_machine_refs` relaxed so a
  v1beta2 `nodeRef` (only `name` guaranteed) still resolves (defaults apiVersion→`v1`,
  kind→`Node`).
- `src/reconcilers/helpers_tests.rs`: +5 unit tests — 4 for `machine_api_version_for`
  (v1beta2/v1beta1 passthrough, `None` fallback, malformed fallback) and a mock-client
  test asserting `remove_machine_from_cluster` issues the Machine DELETE at the
  bootstrapSpec version.
- `docs/src/operations/configuration.md`: documented the new `CAPI_MACHINE_API_VERSION`
  env override and added a "CAPI Machine API version" section explaining the
  passthrough + mandatory-discovery model.
- `docs/src/advanced/capi-integration.md`: added a "CAPI API version (passthrough)"
  note to the Machine Creation Flow — the Machine version follows `bootstrapSpec`,
  the `v1beta1` examples are illustrative not fixed.
- `docs/src/operations/troubleshooting.md`: new entry for the controller exiting at
  startup when Cluster API isn't discoverable ("could not resolve the CAPI Machine
  API version").

### Why
5-Spot v0.2.2 hardcoded the CAPI Machine version to `v1beta1`, which crash-loops
modern k0smotron / CAPI ≥ v1.11 (they serve/watch `v1beta2`). Making the version
data-driven (user's bootstrapSpec) with mandatory served-version discovery lets one
controller image work against both v1beta1 (CAPD) and v1beta2 (modern k0smotron)
clusters, with no hardcoded opinion. Enables the workshop's k0smotron tier.

### Impact
- [ ] Breaking change
- [x] Requires cluster rollout
- [ ] Config change only
- [ ] Documentation only (code + docs)

Note: the controller now **requires** CAPI to be discoverable at startup (it retries
~30s then exits→restarts) instead of silently defaulting to `v1beta1`. Security model
unchanged — the bootstrap/infra **group** allowlist (`validate_api_group`) is the
control; the Machine group is fixed and the version is not a security boundary.

## [2026-06-17 09:00] - Pin deploy manifest image tags to the release version; crdgen back to stdout

**Author:** Erick Bourgeois

### Changed
- `Makefile`: NEW `set-image-version VERSION=…` target — rewrites every
  `ghcr.io/finos/5-spot*` image tag under `deploy/` (controller, both
  spot-schedule providers, reclaim agent, kata-config agent) to the given
  version. Tag-suffix-only regex, so it is independent of the tag committed in
  git. Added to `.PHONY`.
- `.github/workflows/build.yaml` (`package-deploy-manifests`, release-only):
  added a "Pin deployment image versions" step running
  `make set-image-version VERSION=${{ needs.extract-version.outputs.tag_name }}`
  before tarring, so the released `deploy-manifests.tar.gz` references the image
  variant just built/pushed. Its "Regenerate CRDs" step now runs `make crds`.
- `src/bin/crdgen.rs`: reverted to **stdout** output (the binary no longer writes
  files or assumes `deploy/crds/` exists — matches `crddoc`). Args are parsed via
  `clap` (a `ValueEnum` selector), consistent with the other binaries and
  avoiding raw `std::env::args` (resolves Semgrep `rust.lang.security.args.args`,
  code-scanning alert #175). No-arg prints all CRDs as a `---`-separated
  multi-doc stream (pipe-to-`kubectl`-friendly); a selector (`scheduledmachine` /
  `timebasedspotschedule` / `capitalmarketsschedule`) prints just one. The
  file-writing approach was only introduced in the Phase-1 multi-version commit to
  host the schedule/spotSchedule CEL injection, which ADR 0009 removed — so crdgen
  is pure serialization again.
- `Makefile` `crds` target: now owns the file layout, redirecting each selector
  to `deploy/crds/<selector>.yaml`.
- `deploy/crds/capitalmarketsschedule.yaml`: regenerated (picks up the
  `TimeBasedSpotScheduleSpec` doc-comment fix in its description).

### Why
The deployment manifests hardcoded `:v0.1.0`; the release pipeline built/pushed
`ghcr.io/finos/5-spot:<tag>` but shipped manifests still pointed at the stale
tag. Pinning at release time keeps the shipped manifests in lockstep with the
image that was actually built. crdgen returning to stdout keeps path/layout
decisions with the caller (Makefile/CI), not baked into the binary.

### Impact
- [ ] Breaking change
- [x] Requires cluster rollout
- [ ] Config change only
- [ ] Documentation only

---

## [2026-06-16 13:00] - ADR 0009: examples, time-based provider deploy manifests, and prose docs

**Author:** Erick Bourgeois

### Changed
- `src/crd.rs`: `ScheduledMachineSpec.schedule` is now a **required**
  `SpotScheduleRef` (was `Option<ScheduleSpec>`); removed `spec.spotSchedule`;
  added top-level `spec.enabled` master switch; removed `ScheduleSpec` and the
  `v1alpha1` `ScheduledMachine` module; added the `TimeBasedSpotSchedule` CRD
  (spec + status) in `spotschedules.5spot.finos.org`. `is_enabled()` now reads
  `spec.enabled`.
- `src/providers/time_based.rs` (+tests) and `src/bin/spot_schedule_time_based.rs`:
  NEW core/default provider controller — `is_active_at`/`next_transition`/
  `reconcile`, mirroring `capital_markets`; publishes `status.active`.
- `src/reconcilers/{scheduled_machine,helpers,spot_schedule_watch}.rs`: drive
  activation solely from the single `spec.schedule` provider verdict; removed
  `evaluate_schedule`; simplified `compose_should_be_active` (verdict + hold-last
  -state); replaced `validate_activation_source` with `validate_schedule_ref`;
  emergency-reclaim loop-breaker now patches `spec.enabled=false` (never the
  shared provider object); watch reverse-index keys off `spec.schedule`.
- `src/metrics.rs` (+tests): added `fivespot_time_based_active` /
  `fivespot_time_based_transitions_total` + record fns.
- `src/bin/{crdgen,crddoc}.rs`: emit the `TimeBasedSpotSchedule` CRD; single
  `v1beta1` `ScheduledMachine` (dropped the multi-version merge + schedule/
  spotSchedule XOR CEL); api.md regenerated.
- `deploy/admission/validatingadmissionpolicy.yaml`: removed the inline-schedule
  rules (cron/day/hour) and the "at least one" rule; the remaining schedule rule
  pins `spec.schedule.apiVersion` to the provider group; `v1beta1`-only.
- `docs/adr/0009-*.md` (+ index; amended 0006 §1/§3 and 0007);
  `docs/architecture/calm/architecture.json` (provider model + flows; validated,
  diagrams regenerated).
- `examples/scheduledmachine-*.yaml`: migrated all to `5spot.finos.org/v1beta1`,
  added `spec.enabled: true`, and replaced every inline `schedule:` / removed
  `spotSchedule:` block with a single `spec.schedule:` provider reference. Basic
  / weekend / tainted / bad-taint / kata / child-cluster now reference a
  `TimeBasedSpotSchedule`; `scheduledmachine-spot-schedule.yaml` references
  `CapitalMarketsSchedule` (`nyse-equities`) alone (no more AND-composition).
- `examples/timebasedspotschedule.yaml`: NEW companion file defining the
  `business-hours` / `weekend-only` (ns `default`) and `business-hours-toronto`
  (ns `hosted-cluster-alpha`) provider objects the SM examples reference.
- `examples/capitalmarketsschedule.yaml`, `examples/workshop/*`: updated
  comments / shape to `spec.schedule` reference + `spec.enabled`; workshop now
  installs the time-based provider and toggles `spec.enabled` / `tbss`.
- `deploy/spot-schedule-providers/time-based/`: NEW provider deploy bundle
  (serviceaccount / clusterrole / clusterrolebinding / deployment /
  kustomization), mirroring the capital-markets bundle — least-privilege RBAC
  (get/list/watch `timebasedspotschedules`, update/patch its `/status`),
  hardened `Deployment` running `spot-schedule-time-based`.
- `docs/src/**` and `docs/mkdocs.yml`: rewrote all spot-schedule prose to the
  ADR 0009 shape (required `spec.schedule` provider ref; `TimeBasedSpotSchedule`
  as the default first-party provider; `spec.enabled` master switch; emergency
  reclaim sets `spec.enabled`; VAP single group-pin rule 4 / `v1beta1`-only
  matchConstraints). Added a new `docs/src/guides/time-based-schedule.md` guide
  and listed it in the nav.

### Why
ADR 0009 reified the former inline `ScheduledMachine.spec.schedule` window into
the first-party `TimeBasedSpotSchedule` provider, made `spec.schedule` a required
provider reference, dropped the SM `v1alpha1` version and the `spotSchedule`
field, and introduced `spec.enabled`. Examples, deploy manifests, and docs had
to match the new contract.

### Impact
- [x] Breaking change (pre-release CRD contract: `spec.schedule` is now a
  required provider reference, `spec.spotSchedule` removed, SM `v1alpha1` dropped)
- [x] Requires cluster rollout (new CRD + provider controller + RBAC; re-apply
  `deploy/crds/` and the time-based provider bundle)
- [ ] Config change only
- [ ] Documentation only

---

**Author:** Erick Bourgeois

### Changed
- `docs/src/guides/create-your-own-provider.md`: NEW standalone guide — a
  complete, copy-pasteable walkthrough building a minimal `ManualSchedule`
  provider end to end: CRD (namespaced, `status` subresource), controller
  (status-patch reconcile, `Ready` semantics, event-driven not polling),
  least-privilege RBAC, deployment, referencing from a `ScheduledMachine`,
  verification, and additive versioning. Language-agnostic.
- `docs/mkdocs.yml`: grouped both provider guides under a **Spot Schedule
  Providers** sub-section in Guides (CapitalMarketsSchedule + Create Your Own
  Provider) — makes the existing capital-markets guide more discoverable.
- `docs/src/reference/spot-schedule-contract.md`: replaced the inline
  "implement your own" walkthrough with a pointer to the new guide (reference =
  spec, guide = how-to); added both guides to See-also.
- `docs/src/guides/capital-markets-schedule.md`: added a See-also section
  cross-linking the concept, contract, and the new provider guide.

### Why
The "create your own provider" how-to was buried inside the contract *reference*
page; operators asked for it (and the CapitalMarketsSchedule guide) as
first-class entries under **Guides**. Splitting how-to from spec also keeps the
reference page focused.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [x] Documentation only

---

## [2026-06-15 10:30] - Restore dropped spot-schedule metric functions (build fix)

**Author:** Erick Bourgeois

### Changed
- `src/metrics.rs`: Added the three spot-schedule metric definitions and
  `record_*` functions that a `main` merge dropped while keeping their call sites
  in `scheduled_machine.rs` and their tests in `metrics_tests.rs`:
  `SPOT_SCHEDULE_RESOLUTIONS_TOTAL` / `record_spot_schedule_resolution`,
  `SPOT_SCHEDULE_RESOLUTION_ERRORS_TOTAL` / `record_spot_schedule_resolution_error`,
  `SPOT_SCHEDULE_TRANSITIONS_TOTAL` / `record_spot_schedule_transition`.

### Why
The crate failed to compile (`E0425: cannot find function …`) after the
`spot-schdule-phase5` merge — the metric implementations were lost but their
callers and unit tests survived. This restores them (the tests were already the
RED spec), unblocking every CI job, including the new `validate-autovex` gate
which compiles the crate via `cargo run`.

### Impact
- [ ] Breaking change
- [x] Requires cluster rollout
- [ ] Config change only
- [ ] Documentation only

---

## [2026-06-15 10:00] - Spot-schedule docs: contract finalization + concept page (Phase 6)

**Author:** Erick Bourgeois

### Changed
- `docs/src/reference/spot-schedule-contract.md`: finalized — resolved the
  `Ready` ambiguity (present non-`True` ⇒ unresolved; **absent ⇒ active
  authoritative**), removed the open TODOs, documented exactly which status
  fields 5-Spot keys off, and added a worked **"implement your own provider"**
  walkthrough (a minimal `ManualSchedule` hello-world) plus links to the
  reference provider guide/source. Status banner → stable/Accepted.
- `docs/src/concepts/spot-schedule.md`: NEW concept page — why duck-typing over
  more `spec.schedule` fields, referencing a provider, AND composition with
  `killSwitch` precedence, the event-driven watch, and the fail-safe
  hold-last-state behavior. Added to `docs/mkdocs.yml` nav.
- `docs/src/concepts/scheduled-machine.md`: documented the optional `schedule`
  and the new `spotSchedule` reference field (+ AND composition note).
- `docs/adr/0006-...md`: §4 reworded to match the §2 table — resolved the
  internal `Ready=False` vs `False/missing` inconsistency (absent ⇒ authoritative).

### Why
Phase 6 (final phase): make the contract implementable from the docs alone — a
provider author can read the contract page + concept page and ship a conformant
provider without reading 5-Spot's source.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [x] Documentation only

---

## [2026-06-14 17:30] - CapitalMarketsSchedule reference provider controller (Phase 5)

**Author:** Erick Bourgeois

### Changed
- `src/providers/capital_markets.rs` (+ `_tests.rs`): NEW reference spot-schedule
  provider. Pure calendar logic — `is_active_at` (holiday → closed; else must be
  in a `sessions[*]` day+hour window, reusing `parse_day_ranges`/`parse_hour_ranges`;
  `earlyCloses` truncates the day) and `next_transition` (hour-by-hour scan to the
  next flip, bounded by `PROVIDER_TRANSITION_HORIZON_HOURS`). `reconcile` patches
  `status.active`/`observedGeneration`/`lastTransitionTime`/`nextTransitionTime`
  + a `Ready` condition (reason `SessionOpen`/`SessionClosed`) and **requeues
  once at the next calendar boundary** (event-driven single timed requeue, no
  polling, no network calls). `run()` builds the `Controller`.
- `src/providers/mod.rs`, `src/lib.rs`: new `providers` module.
- `src/bin/spot_schedule_capital_markets.rs`: NEW thin controller binary
  (tracing + client + `/metrics` server + `capital_markets::run`).
- `src/constants.rs`: `PROVIDER_TRANSITION_HORIZON_HOURS`,
  `PROVIDER_FALLBACK_REQUEUE_SECS`, `PROVIDER_ERROR_REQUEUE_SECS`,
  `REASON_CAPITAL_MARKETS_SESSION_OPEN/CLOSED`.
- `src/metrics.rs`: `fivespot_capital_markets_active{namespace,name}` gauge +
  `fivespot_capital_markets_transitions_total{namespace,name}` counter (+ record fns).
- `deploy/spot-schedule-providers/capital-markets/`: NEW kustomization —
  ServiceAccount, least-privilege ClusterRole (`get/list/watch` on
  `capitalmarketsschedules`, `update/patch` on **only** `/status`) + binding, and
  a hardened Deployment (non-root, RO-rootfs, caps dropped, seccomp). `trivy
  config` clean (0 findings).
- Tests: 13 calendar tests (mid-session/pre-open/post-close/weekend/holiday/
  early-close, invalid tz, no-sessions; next-transition open→close, close→open,
  weekend-skip, none). 627 lib tests green.
- Docs: `docs/src/guides/capital-markets-schedule.md` (+ nav),
  `docs/src/operations/monitoring.md` provider metrics + alert.

### Why
Phase 5 of the spot-schedule roadmap: the reference provider that makes the
Phase 1 `CapitalMarketsSchedule` type *do* something — computing `status.active`
from a declarative exchange calendar so a `ScheduledMachine.spec.spotSchedule`
can follow market sessions. Producer side; independent of the consumer-side
resolver/watch.

### Impact
- [ ] Breaking change
- [x] Requires cluster rollout (apply `deploy/spot-schedule-providers/capital-markets/`)
- [ ] Config change only
- [ ] Documentation only

---

## [2026-06-14 14:30] - Spot-schedule RBAC + admission + threat model (Phase 4)

**Author:** Erick Bourgeois

### Changed
- `deploy/deployment/rbac/clusterrole.yaml`: NEW rule — `get;list;watch` on
  `spotschedules.5spot.finos.org` `*` (read-only, no write). Unblocks the Phase 3
  provider watch; least-privilege per ADR 0006 §6 (providers are untrusted
  inputs the controller never writes).
- `deploy/admission/validatingadmissionpolicy.yaml`: **security fix** —
  `matchConstraints.apiVersions` now covers `v1beta1` as well as `v1alpha1`.
  Since Phase 1 made `v1beta1` the served/storage version, every `v1beta1`
  ScheduledMachine (i.e. every `spotSchedule` machine) was **bypassing the
  entire VAP** — the bootstrap/infra allowlist and privilege-escalation guards
  included. Also: the four schedule rules are now guarded with
  `!has(object.spec.schedule) || …` (schedule is optional since v1beta1, so a
  valid spotSchedule-only machine is no longer wrongly rejected), and NEW rules
  7a (at least one of schedule/spotSchedule) + 7b (spotSchedule group pin)
  mirror the CRD CEL and `validate_activation_source()`.
- `docs/src/security/threat-model.md`: NEW actor (spot-schedule provider,
  untrusted), threat T8a (provider-driven activation control — mitigated:
  read-only RBAC, same-namespace, group-pinned, auditable) and D5 (provider
  flapping — partially mitigated: back-off + lifecycle timers + transition
  metric/alert).
- `docs/src/security/admission-validation.md`: documented the dual-version
  match, the optional-schedule guard, and rules 7a/7b.
- `docs/src/operations/monitoring.md`: suggested `SpotScheduleProviderFlapping`
  and `SpotScheduleProviderUnresolved` Prometheus alerts.
- `docs/architecture/calm/architecture.json`: `rel-controller-spot-schedule-watch`
  least-privilege control evidence now cites the implemented ClusterRole + VAP.
  `make calm-validate` clean.

### Why
Phase 4: realize ADR 0006 §6's least-privilege, providers-are-untrusted posture
in the deployed manifests, and bring the threat model current. The VAP
version-match fix also closes an admission-bypass regression that Phase 1's
v1beta1 bump introduced.

### Impact
- [ ] Breaking change
- [x] Requires cluster rollout (re-apply ClusterRole + ValidatingAdmissionPolicy)
- [ ] Config change only
- [ ] Documentation only

## [2026-06-14 11:00] - Spot-schedule dynamic event-driven provider watch (Phase 3)

**Author:** Erick Bourgeois

### Changed
- `src/reconcilers/spot_schedule_watch.rs`: NEW. `ReverseIndex` (pure:
  `(gvk, ns, name)` → referencing `ScheduledMachine`s, with per-SM record so a
  changed/removed `spec.spotSchedule` updates precisely) + `provider_key_for`.
  `SpotScheduleWatchManager` owns the index and lazily runs **one
  `kube::runtime::watcher` stream per referenced provider GVK** (event-driven,
  not polling); `sync_watchers` reconciles the running watcher set against the
  index's referenced GVKs (spawn new, abort orphaned). Per-GVK task resolves the
  `ApiResource` via `discovery::pinned_kind` with back-off
  (`SPOT_SCHEDULE_DISCOVERY_RETRY_SECS`, handles CRD-installed-late and
  CRD-deleted-while-watched), maps each provider event back through the index,
  and enqueues the referencing SMs.
- `src/main.rs`: a standalone `ScheduledMachine` reflector feeds SM apply/delete
  into the manager (index rebuilt from the reflector's initial list on restart —
  nothing stored outside Kubernetes); a second `Controller::reconcile_on`
  consumes the manager's mpsc so provider `status.active` flips trigger SM
  reconciles at watch latency.
- `src/constants.rs`: `SPOT_SCHEDULE_EVENT_CHANNEL_CAP`,
  `SPOT_SCHEDULE_DISCOVERY_RETRY_SECS`.
- `src/reconcilers/mod.rs`: export the watch types + `compose_should_be_active`.
- Tests: `spot_schedule_watch_tests.rs` (13 — ReverseIndex register/replace/
  deregister/lookup/gvks, `provider_key_for`, manager observe/forget watcher
  lifecycle incl. shared-GVK and watcher-stop-on-last-deref);
  `tests/integration_spot_schedule.rs` (4 — create→active, flip→inactive,
  delete→unresolved+hold/fail-inactive across a mock kube API, and the watch
  index→reconcile mapping). 658 lib tests green.
- Docs: contract page status → Phase 3.

### Why
Phase 3 of the spot-schedule provider roadmap: remove the Phase-2 pull latency.
Provider status transitions are now a reconcile **trigger** (a true watch over
the `spotschedules.5spot.finos.org` group), not something re-checked only on the
controller's own timer — satisfying the project's watch-not-poll rule. Realizes
the `rel-controller-spot-schedule-watch` relationship + `flow-spot-schedule-activation`
already modeled in CALM at Phase 0 (no ADR/CALM change).

### Impact
- [ ] Breaking change
- [x] Requires cluster rollout (new controller image; RBAC for
      `spotschedules.5spot.finos.org` get/list/watch lands in Phase 4)
- [ ] Config change only
- [ ] Documentation only

## [2026-06-13 18:45] - Spot-schedule resolver + composition + status surface (Phase 2)

**Author:** Erick Bourgeois

### Changed
- `src/reconcilers/spot_schedule.rs`: NEW resolver module. `SpotScheduleVerdict`
  enum (`Active`/`Inactive`/`Unresolved`); pure `verdict_from_status` (duck-typed
  `status.active` + `Ready` extraction); async `resolve_spot_schedule`
  (`discovery::pinned_kind` + `get_opt`). Transient API errors propagate (retry);
  CRD-not-installed / object-absent / status-missing / not-Ready are `Unresolved`
  verdicts, never errors or panics. **Ready semantics finalized:** absent `Ready`
  ⇒ `active` authoritative; present non-`True` `Ready` ⇒ unresolved (resolves an
  internal inconsistency in ADR 0006 §4 / contract table — both updated).
- `src/reconcilers/helpers.rs`: NEW `compose_should_be_active` — AND composition
  of the inline schedule and the provider verdict, with hold-last-state on
  Unresolved (fail-inactive when never resolved). `killSwitch` precedence is
  upstream.
- `src/crd.rs`: replaced Phase-1 `effective_schedule()` with `is_enabled()` (a
  spotSchedule-only machine is always enabled; the provider governs activity).
  NEW `SpotScheduleStatus` + `ScheduledMachineStatus.spot_schedule` field (the
  durable hold-last-state surface: `resolved`/`active`/`reason`/`message`/
  `providerGeneration`/`lastTransitionTime`).
- `src/reconcilers/scheduled_machine.rs`: `reconcile_inner` now evaluates the
  optional inline schedule, resolves the provider (pull-on-reconcile), composes
  `should_be_active`, records metrics, and patches `status.spotSchedule`
  (`persist_spot_schedule_status`, a `/status` merge patch that coexists with the
  phase/condition writes). Handler `enabled` guards use `is_enabled()`.
- `src/metrics.rs`: NEW `fivespot_spot_schedule_resolutions_total{namespace,kind,result}`,
  `_resolution_errors_total{namespace,kind,reason}`,
  `_transitions_total{namespace,kind}` (+ record fns; bounded cardinality, never
  keyed by name).
- `deploy/crds/scheduledmachine.yaml` regenerated (status.spotSchedule);
  `docs/src/reference/api.md` regenerated.
- Tests: `spot_schedule_tests.rs` (verdict truth table + 3 mock-client resolve
  paths), `helpers_tests.rs` (12-case composition truth table),
  `scheduled_machine_tests.rs` (persist `/status` patch), `metrics_tests.rs`,
  `crd_tests.rs` (status round-trip + is_enabled). 645 lib tests green.
- Docs: ADR 0006 + contract page `Ready` clarification; contract page status →
  Phase 2; `docs/src/operations/monitoring.md` spot-schedule metrics table.

### Why
Phase 2 of the spot-schedule provider roadmap: make a `spec.spotSchedule`
reference actually drive the machine's active/inactive decision (pull-on-reconcile),
with the ADR-specified hold-last-state fail-safe and an observable status surface.
The event-driven watch (Phase 3) and the reference-provider controller (Phase 5)
build on this.

### Impact
- [ ] Breaking change
- [x] Requires cluster rollout (apply regenerated CRDs + new controller image)
- [ ] Config change only
- [ ] Documentation only

## [2026-06-13 16:30] - Spot-schedule provider API: CRD scaffolding + ScheduledMachine v1beta1 (Phase 1)

**Author:** Erick Bourgeois

### Changed
- `src/crd.rs`: `ScheduledMachine` bumped to **`v1beta1`** (storage) — `schedule`
  is now `Option<ScheduleSpec>` and a new `spot_schedule: Option<SpotScheduleRef>`
  field is added (ADR 0006). New `SpotScheduleRef` type (group-pinned `apiVersion`
  via `x-kubernetes-validations` CEL, `deny_unknown_fields`, same-namespace
  `name`). New `ScheduledMachineSpec::effective_schedule()` accessor +
  `ScheduleSpec::inactive_placeholder()` (a spotSchedule-only machine evaluates
  inactive until the Phase 2 resolver lands). New frozen, **deprecated**
  `pub mod v1alpha1` ScheduledMachine (served for back-compat; superset relation
  keeps `conversion: None` lossless — ADR 0007). New reference provider type
  `CapitalMarketsSchedule` (group `spotschedules.5spot.finos.org/v1alpha1`):
  spec (timezone/sessions/holidays/earlyCloses) + duck-typed
  `status.active`/conditions/observedGeneration.
- `src/constants.rs`: `API_VERSION`/`API_VERSION_FULL` → `v1beta1` (controller's
  storage version, used for owner refs); new `API_VERSION_LEGACY`,
  `SPOT_SCHEDULE_API_GROUP`, `SPOT_SCHEDULE_API_VERSION`,
  `CONDITION_TYPE_SPOT_SCHEDULE_RESOLVED`, and the four
  `REASON_SPOT_SCHEDULE_*` resolution reasons.
- `src/bin/crdgen.rs`: rewritten to `merge_crds(v1alpha1, v1beta1, storage=v1beta1)`
  for `ScheduledMachine` (+ inject the at-least-one-of `schedule`/`spotSchedule`
  CEL into the v1beta1 spec schema) and emit the `CapitalMarketsSchedule` CRD;
  now writes both files under `deploy/crds/` directly. `Makefile` `crds` target
  updated (no stdout redirect).
- `src/bin/crddoc.rs`: documents the v1beta1/v1alpha1 versioning, the
  `spotSchedule` field, and the at-least-one-of rule; `docs/src/reference/api.md`
  regenerated.
- `src/reconcilers/scheduled_machine.rs`: route `spec.schedule` reads through
  `effective_schedule()` (optional schedule); call the new
  `validate_activation_source` in `reconcile_inner`'s defence-in-depth block.
- `src/reconcilers/helpers.rs`: new `validate_activation_source` — runtime mirror
  of the CRD spec-level CEL (at least one of `schedule`/`spotSchedule`, plus the
  `spotschedules.5spot.finos.org` group pin), so the invariant holds even
  without the apiserver CEL (pre-1.25, pre-existing stored objects, direct API
  writes). The CRD CEL is structural-schema validation (apiserver-enforced), not
  a VAP — this is the belt-and-braces runtime guard.
- `deploy/crds/scheduledmachine.yaml` (multi-version, regenerated) +
  `deploy/crds/capitalmarketsschedule.yaml` (NEW).
- `examples/scheduledmachine-spot-schedule.yaml`,
  `examples/capitalmarketsschedule.yaml`: NEW.
- Tests: `src/crd_tests.rs` adds SpotScheduleRef round-trip/group-pin,
  effective_schedule, spotSchedule-only round-trip, multi-version single-storage
  invariant, and CapitalMarketsSchedule coverage; all SM-spec literals across
  `*_tests.rs` + `tests/` updated for `Option` schedule + `spot_schedule`.
- Docs: `docs/src/installation/crds.md`, `docs/src/reference/spot-schedule-contract.md`.

### Why
Phase 1 of the spot-schedule provider roadmap: introduce the `spec.spotSchedule`
contract surface and the reference provider CRD, landed on a new `ScheduledMachine`
`v1beta1` version (first exercise of the ADR 0007 multi-version machinery). The
controller-side resolver/watch follow in later phases.

### Impact
- [ ] Breaking change
- [x] Requires cluster rollout (apply regenerated CRDs: `kubectl apply -f deploy/crds/`)
- [ ] Config change only
- [ ] Documentation only

## [2026-06-13 14:00] - ADR 0006/0007 + CALM: pluggable spot-schedule provider contract (Phase 0)

**Author:** Erick Bourgeois

### Changed
- `docs/adr/0006-pluggable-spot-schedule-provider-contract.md`: NEW — accept a
  duck-typed external provider contract. `ScheduledMachine.spec.spotSchedule`
  (`apiVersion`/`kind`/`name`, same namespace) delegates the active/inactive
  decision to a CRD in the new `spotschedules.5spot.finos.org` group; 5-Spot
  reads only `status.active` (+ `Ready`), never the provider spec, never writes
  it. `spec.schedule` becomes optional (CEL: at least one of schedule/
  spotSchedule); both set ⇒ logical AND; `killSwitch` overrides. Unresolved
  references hold last-known state (fail-inactive when never resolved) and
  surface `SpotScheduleResolved=False` — no flapping. Event-driven dynamic
  per-GVK watch + reverse index. Providers are untrusted inputs (read-only,
  group-pinned RBAC; same-namespace only).
- `docs/adr/0007-crd-multi-version-and-conversion.md`: NEW — CRDs are
  multi-version-capable (one Rust struct per served version, single
  `storage: true`), `conversion.strategy: None` with additive-only evolution;
  resolvers/watchers key off group+kind, never a pinned `apiVersion`. The first
  breaking change is the trigger for a webhook (superseding ADR). No CALM impact
  (generation/versioning policy). Applies to `ScheduledMachine` and the new
  provider group.
- `docs/adr/README.md`: index rows for ADR 0006 and 0007.
- `docs/architecture/calm/architecture.json`: NEW node
  `service-spot-schedule-provider` (external, untrusted) + data-asset
  `data-asset-spot-schedule-cr`; relationships `rel-spot-schedule-cr-stored-in-api`,
  `rel-provider-writes-spot-schedule-status`, `rel-controller-spot-schedule-watch`
  (read-only, group-pinned least-privilege control); flow
  `flow-spot-schedule-activation`. `make calm-validate` clean (0/0/0);
  `make calm-diagrams` re-rendered `docs/src/architecture/{system,flows}.md`.
- `docs/src/reference/spot-schedule-contract.md`: NEW — provider contract
  skeleton (reference, status table, AND composition, unresolved behavior,
  versioning policy); added to `docs/mkdocs.yml` Reference nav.

### Why
Phase 0 (ADD gate) of the spot-schedule provider roadmap: record and visualize
the architecture before any code, so activation semantics (capital-markets
calendars, PromQL emitters, change-freeze gates) can be implemented out-of-tree
against a stable, versioned public contract. Team-approved to proceed
2026-06-13.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [x] Documentation only

## [2026-06-10 11:30] - ADR 0005: remove spec.kata.destPath — fixed host path, H-1 gate closed

**Author:** Erick Bourgeois

### Changed
- `docs/adr/0005-remove-kata-destpath-fixed-host-path.md`: NEW — remove the
  user-configurable `destPath` entirely rather than allowlist it; the drop-in
  destination is the compile-time constant `/etc/k0s/containerd.d/kata.toml`
  (the real k0s containerd import dir per docs.k0sproject.io/stable/runtime —
  the previous `container.d` default was a typo k0s never read). Index updated.
- `src/crd.rs`: `KataConfig.dest_path` field, its schema fn, and its default
  removed; `deny_unknown_fields` makes a lingering `destPath` a hard admission
  error. Spec docs updated.
- `src/constants.rs`: new `KATA_CONFIG_DEST_PATH` + `KATA_CONFIG_DEST_BASE`;
  `KATA_CONFIG_APPLIED_ANNOTATION` doc rewritten (bare hash, no path).
- `src/kata_config_agent.rs`: NEW `confine_dest_path` — fail-closed
  defense-in-depth resolver (`/etc/k0s/` slash-aware prefix + `.toml` suffix,
  lexical `..`/`.`/empty-segment rejection, `canonicalize(parent)` must stay
  under the canonicalized base — defeats symlinked-directory escapes) run
  before EVERY host write and tear-down unlink; `KataRef.dest_path` removed
  (legacy annotations with the field still parse, path ignored);
  `AppliedRecord`/`parse_applied_record` deleted — the applied annotation is a
  bare content hash a forged value cannot weaponize.
- `src/bin/kata_config_agent.rs`: deliver + tear-down both target the fixed
  constant through `confine_dest_path`; tear-down no longer needs a recorded
  path (always reconciles the fixed path to absent); `host_path` helper gone.
- `src/reconcilers/helpers.rs`: ref-annotation builder no longer emits
  `destPath` — no Node annotation carries a host path.
- `deploy/kata-config-agent/daemonset.yaml`: hostPath mount narrowed from `/`
  to `/etc/k0s` (`type: Directory` fail-fast); comments rewritten.
- Tests (all four suites): fixtures drop `dest_path`; new coverage — schema has
  no destPath property, serde rejects a lingering destPath, legacy-annotation
  tolerance, 10 `confine_dest_path` tests (traversal, symlink escape, prefix
  sibling `/etc/k0s.evil`, non-toml, relative, curdir), ref-annotation
  carries-no-path assertions. CRDs + api.md regenerated (crddoc kata section
  rewritten).
- `src/labels.rs` → `src/labels_tests.rs`: inline `mod tests` body moved out —
  the last violation of the separate-`_tests.rs` rule; `.claude/CLAUDE.md`
  hardened to state the rule is exception-free for ALL targets including
  `src/bin/*.rs`, with only the three-line `#[path]` declaration allowed in
  source files.
- Docs: concepts page (annotation-contract table, diagrams, security table,
  ruled-out list), operator guide (fixed-path prereqs/steps/troubleshooting),
  scheduled-machine spec row, example manifest — all rewritten for the fixed
  path; `docs/architecture/calm/architecture.json` `host-path-containment`
  control + agent/flow descriptions updated (`make calm-validate` clean,
  diagrams re-rendered).

### Why
Closes the H-1 latent-High gate from the 2026-06-09 security roadmap, stronger
than originally specified: with the agent running privileged on the host, a
CRD-supplied path was a node-root-via-CRD vector (write any host file, unlink
any host file via a forged applied annotation). Per operator decision the
capability is removed, not validated — no path-shaped attacker input exists in
the contract at all, and the writable host surface shrinks to /etc/k0s.

### Impact
- [x] Breaking change (pre-release: any SM setting kata.destPath now fails admission)
- [ ] Requires cluster rollout
- [ ] Config change only
- [ ] Documentation only

## [2026-06-10 09:30] - Agent pod-security exception boundary VAP (ADR 0004)

**Author:** Erick Bourgeois

### Changed
- `docs/adr/0004-agent-pod-security-exception-boundary-vap.md`: new ADR — the
  privileged node agents need a PSA/Gatekeeper/Kyverno exemption for
  `5spot-system` on the workload cluster; since Kubernetes admission is
  conjunctive (no VAP can override another engine's deny), the exemption lives
  in the baseline engine and we ship a deny-by-default compensating VAP behind
  it. Index updated in `docs/adr/README.md`.
- `deploy/admission/agent-pod-security-policy.yaml`: new `ValidatingAdmissionPolicy`
  (`5spot-agent-pod-security`, k8s ≥ 1.30, `failurePolicy: Fail`) — identity-pinned
  allowlist over pods in `5spot-system`: `hostPID`/`hostPath`/explicit-root
  restricted to the two agent ServiceAccounts; `privileged` to the kata agent only;
  hostPath clamped per agent (kata: `/`; reclaim: `/proc`, `/etc/machine-id`);
  capability adds clamped to `NET_ADMIN` on the reclaim agent;
  `hostNetwork`/`hostIPC` denied for everyone; compensating controls
  (readOnlyRootFilesystem on privileged containers, pod seccomp RuntimeDefault)
  made mandatory; ephemeral containers may never be risky. Header documents the
  PSA/Gatekeeper/Kyverno exemption recipes this policy pairs with.
- `deploy/admission/agent-pod-security-binding.yaml`: binding scoped to
  `5spot-system` via `namespaceSelector`, `validationActions: [Deny]`.
- `docs/architecture/calm/architecture.json`: `agent-pod-security-boundary`
  control added to the workload-cluster API server node (NIST AC-6/CM-5);
  `make calm-validate` clean, diagrams re-rendered.
- `docs/src/security/admission-validation.md`: new "Agent Pod-Security Exception
  Boundary" section (posture table, conjunctive-admission explanation, per-engine
  exemption recipes); deployment list/commands extended to the fourth policy pair.
- `docs/src/security/threat-model.md`: new threat E5 (abuse of the namespace-wide
  exemption) marked mitigated by ADR 0004; mitigations-summary row added.

### Why
The agents' privileged posture (ADR 0003) cannot pass a pod-security baseline,
and the only exemption mechanisms those engines offer are namespace-wide. An
unguarded exemption would let anything in `5spot-system` run privileged. The
paired deny-by-default VAP makes the "only our two agents, only their exact
posture" claim machine-enforced and auditable. Policy logic verified by
simulating all 12 CEL validations against the real DaemonSet pod templates
(admitted) and 9 rogue/escalation scenarios (each denied by the intended rule).

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [x] Config change only (new admission manifests; operator applies with the
  baseline-engine exemption as a paired deployment)
- [ ] Documentation only

## [2026-06-09 21:30] - Kata config delivery Phase 5 — agent metrics, integration test, docs, security pass

**Author:** Erick Bourgeois

### Changed
- `src/metrics.rs`: six new agent metrics — `fivespot_kata_config_writes_total`,
  `…_deletes_total`, `…_drift_corrected_total`, `…_restarts_total`,
  `…_sync_errors_total`, and the `…_last_sync_timestamp_seconds` staleness gauge —
  plus their recorder fns and a shared `serve_metrics(port)` (the controller's
  private metrics server moved here so both binaries serve one implementation).
- `src/metrics_tests.rs`: 6 new tests covering every recorder (counter increments,
  drift/write coupling, unchanged-tick gauge-only semantics), serialized on a
  lock because the kata counters are label-less process-globals.
- `src/kata_config_agent.rs` + `_tests.rs`: new pure `is_drift_correction`
  classifier (a rewrite of already-applied content is drift, not a rollout) + 4 tests.
- `src/bin/kata_config_agent.rs`: `--metrics-port` (env `METRICS_PORT`, default
  8080), metrics server spawned in loop mode, and recording wired into
  `reconcile_once` (write/delete/unchanged/restart) and the retry path (errors).
- `src/main.rs`: `run_metrics_server` replaced by `metrics::serve_metrics`.
- `tests/integration_kata_config.rs`: NEW — pins the ADR 0002 annotation contract
  (controller patch builder ↔ agent parser, field-for-field), runs the full
  host-file lifecycle (provision → restart-once → drift-heal-without-restart →
  rollout-restart → tear-down) against a tempdir with a counting executor, and
  adds an `--ignored` real-Node round-trip mirroring `integration_emergency_reclaim.rs`.
- `src/reconcilers/mod.rs`: re-export the two kata patch builders for the test.
- `deploy/kata-config-agent/daemonset.yaml`: `prometheus.io/*` scrape annotations,
  named `metrics` containerPort 8080, `METRICS_PORT` env.
- `deploy/kata-config-agent/kustomization.yaml`: NEW — `kubectl apply -k` entry
  point mirroring `deploy/node-agent/`.
- `examples/scheduledmachine-kata.yaml`: NEW — full `spec.kata` example with the
  source-ConfigMap shape in a comment.
- `.trivyignore`: justify KSV-0001 (allowPrivilegeEscalation cannot be false with
  privileged: true — API server rejects the combination) and KSV-0003/0004
  (drop: [ALL] is a kernel no-op under privileged); `trivy config deploy/` now 0 findings.
- Docs: NEW `docs/src/concepts/kata-config-delivery.md` (architecture, mermaid
  delivery chain + sequence, restart-guard table, security posture) and NEW
  `docs/src/guides/kata-config.md` (operator walkthrough + troubleshooting);
  `spec.kata` row in `concepts/scheduled-machine.md`; agent metrics table in
  `operations/monitoring.md`; nav entries (new Guides section) in `mkdocs.yml`;
  feature bullet in `README.md`.
- Stale-comment sweep: `src/bin/crddoc.rs` emitted the pre-pivot `kataConfigRef:`
  field name into the generated API example — fixed and `make crddoc` re-run;
  pre-pivot "per-node ConfigMap projection" comments rewritten in
  `src/constants.rs`, `src/reconcilers/helpers.rs`, `src/reconcilers/scheduled_machine.rs`.
- `docs/architecture/calm/architecture.json`: agent node gains its `/metrics`
  HTTP interface; least-privilege control text corrected to get-only
  ConfigMap/Secret RBAC (KSV-0113); data-asset wording aligned to the per-tick
  GET (no watcher). `make calm-validate` clean, diagrams re-rendered.

### Why
Closes Phase 5 (final phase) of the per-node Kata config delivery roadmap:
observability for the node agent, an integration test pinning the single
load-bearing controller↔agent contract, user-facing concept + operator docs,
and a clean security scan with every privileged-posture finding justified.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [x] Documentation only (agent/manifests not yet released)

## [2026-06-09 19:30] - Kata config delivery Phase 4 — nsenter host k0s-service restart

**Author:** Erick Bourgeois

### Changed
- `src/kata_config_agent.rs`: add the Phase 4 restart-orchestration units (ADR
  0003) — `nsenter_restart_argv` (builds `nsenter -t 1 -m -u -i -n -p -- systemctl
  restart <service>`), the `RestartExecutor` trait, `AppliedRecord` ({destPath,
  hash} carried in the `kata-config-applied` annotation) + `parse_applied_record`,
  `intended_hash_for`, `needs_restart`, and `restart_if_needed`. `ABSENT_HASH_MARKER`
  marks the torn-down state.
- `src/kata_config_agent_tests.rs`: 14 new tests — exact nsenter command line,
  `AppliedRecord` serde round-trip + garbage→None, `intended_hash_for` per outcome,
  the `needs_restart` guard (no-prior / stale / match), and `restart_if_needed`
  short-circuit / once / error-propagation via a counting `RestartExecutor` fake.
- `src/bin/kata_config_agent.rs`: wire the restart into `reconcile_once` — after a
  sync, when the on-host content hash differs from the last-applied hash, record the
  new `AppliedRecord` **before** restarting via the concrete `NsenterRestartExecutor`.
  `read_node_state`/`record_applied` now use the compact `{destPath, hash}` record
  (replacing the bare-destPath value); unparseable records degrade to "not applied"
  (re-apply + restart, the safe direction).
- `docs/adr/0003-*.md`: refine the applied annotation to the `{destPath, hash}` record.
- `docs/architecture/calm/architecture.json`: align the agent node, the workload-API
  relationship, and the delivery-flow step prose to the compact record; `make
  calm-validate` clean, `make calm-diagrams` re-rendered.

### Why
Completes the load-bearing piece of per-node kata delivery: after writing the
drop-in the agent must bounce the host k0s service so containerd reloads it.
Done in-pod via `nsenter` into host PID 1 (ADR 0003). The applied-hash record,
written before the restart, is the restart-loop guard — the restart SIGKILLs the
agent, and on the next loop a matching hash makes it a no-op (single-cycle
convergence). Drift correction rewrites the file but does not re-restart.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [x] Documentation only (agent/manifests not yet released)

## [2026-06-09 18:45] - Tighten kata-config-agent secret RBAC to get-only (KSV-0113)

**Author:** Erick Bourgeois

### Changed
- `deploy/kata-config-agent/rbac.yaml`: narrow the namespaced Role's
  `configmaps`/`secrets` verbs from `["get", "list", "watch"]` to `["get"]`.
  The agent resolves a single source object by exact name (the ref carried in
  its Node's `kata-config-ref` annotation) via a direct GET — it never
  enumerates and runs no Kubernetes informer (the drift-watch is a node-local
  file poll), so `list`/`watch` were dead grants. Updated the file's RBAC
  banner accordingly.
- `.trivyignore`: add `AVD-KSV-0113` for the now get-only secrets read. The
  finding still fires on any namespace-scoped secret read; the access is
  architecturally required because a Kata drop-in source may be a Secret
  (`spec.kata` → Secret, ADR 0002). Justification mirrors the controller's
  read-only `KSV-0041` suppression.

### Why
GitHub Advanced Security / Trivy flagged KSV-0113 (MEDIUM) on the agent Role.
The grant is read-only — never "manage" — but was broader than needed.
Tightening to `get`-only caps a stolen token's blast radius to objects the
attacker can already name, rather than the whole `5spot-system` namespace, then
the irreducible read-only finding is suppressed with justification.

### Impact
- [x] Config change only
- [ ] Breaking change
- [ ] Requires cluster rollout

## [2026-06-09 18:30] - Suppress KSV-0106 for the privileged kata-config-agent

**Author:** Erick Bourgeois

### Changed
- `.trivyignore`: add `AVD-KSV-0106` to the kata-config-agent block. The rule
  ("drop all capabilities, only add NET_BIND_SERVICE") is moot under
  `privileged: true` (KSV-0017, ADR 0003) — a privileged container holds the
  full capability set regardless of any `capabilities.drop` list, so the drop
  would be a no-op rather than real hardening. The agent needs no
  NET_BIND_SERVICE. Justification mirrors the existing KSV-0017 entry.

### Why
Trivy KSV-0106 (LOW) fired on `deploy/kata-config-agent/daemonset.yaml`. Adding
`capabilities.drop: [ALL]` would satisfy the scanner but misrepresent the
container's effective privileges in an audit, so the finding is suppressed with
a written justification per the `.trivyignore` policy.

### Impact
- [x] Config change only
- [x] Documentation only

## [2026-06-09 18:00] - Kata tear-down handshake + privileged agent DaemonSet

**Author:** Erick Bourgeois

### Changed
- `src/reconcilers/helpers.rs`: `reconcile_kata_config_delivery` tear-down now
  clears **only** the reference annotation and leaves the opt-in label in place
  (so the agent is not descheduled before it can clean up the host file).
- `src/bin/kata_config_agent.rs`: the agent records the applied `destPath` in a
  `5spot.finos.org/kata-config-applied` Node annotation on write; on tear-down
  (ref annotation gone) it unlinks the recorded host file, then removes its own
  opt-in label to deschedule (`read_node_state` / `record_applied` / `clear_optin`).
- `src/constants.rs`: add `KATA_CONFIG_APPLIED_ANNOTATION`.
- `src/reconcilers/helpers_tests.rs`: tear-down tests assert the label is left
  untouched and only the annotation is cleared.
- `deploy/kata-config-agent/daemonset.yaml`: `privileged: true` + `hostPID: true`
  (ADR 0003 — required for the `nsenter` k0s-service restart).
- `deploy/kata-config-agent/rbac.yaml`: agent ClusterRole `nodes: get` → `get, patch`
  (records applied annotation + removes its own label).
- `.trivyignore`: kata-config-agent banner; add `AVD-KSV-0017` (privileged)
  with architectural justification (other findings share the reclaim agent's
  already-suppressed global rule IDs).

### Why
Closes two gaps: (1) the tear-down race where label-removal descheduled the agent
before it could unlink the host drop-in — now the agent owns its own
descheduling after cleanup; (2) the DaemonSet must be privileged to `nsenter`
into host PID 1 and bounce k0s so containerd reloads the drop-in (ADR 0003).

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [x] Documentation only (agent/manifests not yet released)

## [2026-06-09 17:15] - Kata agent: API-read by Node annotation + manifests

**Author:** Erick Bourgeois

### Changed
- `src/kata_config_agent.rs`: add `KataRef` + `parse_kata_ref` (parse the
  `5spot.finos.org/kata-config-ref` Node annotation) and `sync_content`
  (reconcile the host file from in-memory content); `sync_once` now delegates to
  `sync_content`.
- `src/bin/kata_config_agent.rs`: rewrite from mounted-file (`SOURCE_PATH`) to
  **API-read** — each tick reads the Node annotation, `get`s the referenced
  workload `ConfigMap`/`Secret`, extracts the key, and syncs to `HOST_ROOT`+`destPath`.
  No ConfigMap volume.
- `src/kata_config_agent_tests.rs`: tests for `parse_kata_ref` (full/missing/non-JSON),
  `sync_content` (write/delete/noop), and `host_path`. 561 lib tests + bin tests pass.
- `deploy/kata-config-agent/rbac.yaml`: ServiceAccount + namespaced Role
  (`configmaps`/`secrets` get/list/watch in `5spot-system`) + ClusterRole
  (`nodes: get` for the own-Node annotation read). No API writes.
- `deploy/kata-config-agent/daemonset.yaml`: opt-in DaemonSet (nodeSelector
  `5spot.finos.org/kata-config=enabled`), `NODE_NAME` downward API, hostPath `/`
  at `/host`, no ConfigMap volume. Phase-3 posture (root, caps dropped, RO rootfs,
  no privileged/hostPID); Phase 4 adds privileged+hostPID+`nsenter` per ADR 0003.

### Why
Completes ADR 0002 §3: a cluster-wide DaemonSet can't template a `configMap.name`
volume per replica, so the agent reads its source object from the workload API
keyed by the controller-stamped Node annotation.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [x] Documentation only — new node-agent manifests + agent binary (not yet released)

## [2026-06-09 16:30] - Implement spec.kata workload-read delivery (ADR 0002)

**Author:** Erick Bourgeois

### Changed
- `src/crd.rs`: rename `spec.kataConfigRef`/`KataConfigRef` → `spec.kata`/`KataConfig`;
  add `namespace` field (workload-cluster target, default `5spot-system`) with
  schema + default; docs reflect workload-cluster resolution.
- `src/reconcilers/helpers.rs`: replace the per-node ConfigMap projection
  (`reconcile_kata_config_provision`, `build_kata_config_configmap`,
  `per_node_kata_configmap_name`, `extract_*`, `resolve_kata_config_content`)
  with `reconcile_kata_config_delivery` (read-only workload `get_opt` of the
  source object + namespace probe → Node opt-in label/annotation, or fail-fast
  `SourceNotFound`/`TargetNamespaceMissing`), `build_kata_config_ref_annotation_patch`,
  and the `KataDeliveryOutcome` enum. No controller ConfigMap/Secret writes.
- `src/reconcilers/scheduled_machine.rs`: call site reads on the workload client,
  logs the fail-fast outcome.
- `src/constants.rs`: add `KATA_CONFIG_REF_ANNOTATION`; drop the
  projection-only `KATA_CONFIG_CONFIGMAP_PREFIX` / `KATA_CONFIG_DATA_KEY` /
  `KATA_CONFIG_FIELD_MANAGER`.
- `src/crd_tests.rs`, `src/reconcilers/helpers_tests.rs`: migrate tests to the
  new contract (namespace default/override, annotation builders, mock-API
  delivery tests for present / source-missing / namespace-missing / disabled);
  remove obsolete projection tests. 555 lib tests pass.
- `src/bin/crddoc.rs`: regenerate API docs text for `spec.kata`.
- `deploy/crds/scheduledmachine.yaml`, `docs/src/reference/api.md`: regenerated.
- `deploy/deployment/rbac/clusterrole.yaml`: kata is read-only (`configmaps: get`
  + new `namespaces: get`); the `configmaps` write verbs are re-attributed to the
  reclaim-agent projection (the honest KSV-0049 owner). `.trivyignore`: re-add
  `AVD-KSV-0049` attributed to reclaim.
- `docs/architecture/calm/architecture.json`: kata flow updated to read-not-project;
  `make calm-validate` clean; diagrams regenerated.
- `docs/adr/0002-kata-config-delivery-via-spec-kata.md`: §5/Consequences corrected
  — kata adds no write privilege but does NOT eliminate KSV-0049 (reclaim still
  holds it).

### Why
Implements the approved ADR 0002 redesign: 5-Spot resolves the Flux-delivered
kata object on the workload cluster and opts the Node in (or fails fast with a
status reason), instead of projecting per-node ConfigMaps. Removes duplication,
the controller-owned sync burden, and kata's contribution to the ConfigMap write
grant.

### Impact
- [x] Breaking change (CRD field `spec.kataConfigRef` → `spec.kata`; unreleased)
- [ ] Requires cluster rollout
- [ ] Config change only
- [ ] Documentation only

## [2026-06-09 15:00] - Redesign kata config delivery (ADR 0002): workload-read, no writes

**Author:** Erick Bourgeois

### Changed
- `docs/adr/0002-kata-config-delivery-via-spec-kata.md`: rewritten (and renamed
  from `0002-per-node-kata-config-delivery.md`) to the accepted design — replace
  `spec.kataConfigRef` with `spec.kata`; the controller **resolves** the kata
  object on the workload cluster (via `kubeconfig-<clusterName>`) and either opts
  the Node in (label + reference annotation) or **fails fast** with a status
  reason (`SourceNotFound` / `TargetNamespaceMissing`). 5-Spot performs **no**
  ConfigMap/Secret writes and **no** namespace creation for kata delivery; the
  config must pre-exist (Flux-delivered). Target namespace defaults to
  `5spot-system`, overridable. KSV-0049 eliminated by construction.
- `docs/adr/0003-in-pod-host-service-restart-via-nsenter.md`: aligned with the
  above — controller reads (not projects); agent consumes the object via the
  workload kube API keyed by Node annotation (no ConfigMap volume); `spec.kata`.
- `docs/adr/README.md`: index reflects the consolidated set (0001–0003).
- Removed the two intermediate, never-committed-to-main ADRs (the per-node
  `spec.kataConfigRef` projection and its KSV-0049 ownership/RBAC defense) and
  reverted this session's per-node documentation (concept page, `.trivyignore`
  `AVD-KSV-0049`, threat-model rows, `clusterrole.yaml` / `helpers.rs` comments).

### Why
Review against the platform model (tenants create nothing; Flux delivers config;
a tenant namespace backs 1+ k0smotron control planes; the agent runs on the
workload cluster the controller already has kubeconfig access to) rejected the
per-node ConfigMap projection: wrong granularity, controller-owned duplication, a
sync burden 5-Spot shouldn't own, and an unnecessary ConfigMap write grant. The
accepted design makes 5-Spot a read-only resolver + opt-in labeler that fails
fast when config is absent.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [x] Documentation only (ADR/design; implementation tracked separately)

## [2026-06-09 01:00] - Kata config delivery Phase 3 — node agent host-sync engine

**Author:** Erick Bourgeois

### Changed
- `src/kata_config_agent.rs`: new host-filesystem sync engine — `sha256_hex`,
  `file_sha256`, `SyncAction` + `decide_action` (pure decision), `atomic_write`
  (temp + `rename`, mode 0644, parent-dir create, crash-safe), `remove_if_present`,
  and `sync_once` → `SyncOutcome` (Wrote/Deleted/Unchanged). Drift self-heals.
- `src/kata_config_agent_tests.rs`: 22 unit tests (SHA-256 vectors, decision
  matrix, atomic write incl. mode/parent-dirs/no-temp-leak/crash-safety,
  idempotent unlink, end-to-end sync incl. drift correction + GitOps delete).
- `src/bin/kata_config_agent.rs`: new `5spot-kata-config-agent` binary — clap
  CLI (`SOURCE_PATH`/`DEST_PATH`/`NODE_NAME`/`POLL_INTERVAL_SECS`/`--oneshot`)
  driving a drift-watch poll loop over `sync_once`.
- `src/lib.rs`: export `kata_config_agent` module.
- `Cargo.toml`: add `sha2` dependency (already in the graph transitively) and
  the `5spot-kata-config-agent` bin target.

### Why
Phase 3 of ADR 0003 — land the drop-in on the host and self-heal drift. The
applied-hash node annotation and the `nsenter` host k0s-service restart are
Phase 4; the DaemonSet manifest, agent RBAC, `.trivyignore`, and metrics land
in Phase 4 with the final (privileged) security posture so they are decided as
one unit. The drift-watch is a deliberate node-local poll — out-of-band host
edits raise no Kubernetes event to watch on.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [ ] Documentation only

> New binary only; not yet deployed (no DaemonSet until Phase 4). No controller
> or CRD behavior change. Verified end-to-end via `--oneshot`: write → no-op →
> delete, parent dirs created, no temp-file leakage.

---

## [2026-06-09 00:00] - Kata config delivery Phase 2 — controller projection + RBAC

**Author:** Erick Bourgeois

### Changed
- `src/constants.rs`: new kata-config constants — `KATA_CONFIG_LABEL`
  (`5spot.finos.org/kata-config`), `KATA_CONFIG_LABEL_ENABLED`,
  `KATA_CONFIG_NAMESPACE`, `KATA_CONFIG_CONFIGMAP_PREFIX`,
  `KATA_CONFIG_DATA_KEY`, `KATA_CONFIG_FIELD_MANAGER`
  (`5spot-controller-kata-config`).
- `src/reconcilers/helpers.rs`: kata projection — pure builders
  `per_node_kata_configmap_name`, `build_kata_config_configmap`,
  `build_kata_config_label_patch`; content extractors
  `extract_kata_content_from_configmap` / `_from_secret`; async
  `resolve_kata_config_content` and `reconcile_kata_config_provision`
  (resolve source → stamp label → SSA per-node ConfigMap; tear down on
  `None`, 404-on-delete benign). Mirrors the reclaim-agent projection.
- `src/reconcilers/helpers_tests.rs`: 18 new tests — pure builders/extractors
  (happy/missing-key/non-UTF-8) plus five mock-client orchestrator tests
  (ConfigMap source, Secret source w/ custom key, tear-down, 404-benign,
  source-404 short-circuit).
- `src/reconcilers/scheduled_machine.rs`: `provision_kata_config_best_effort`
  wired into `handle_active_phase` beside reclaim/taints — child-cluster-scoped
  writes, management-cluster source read, non-fatal.
- `deploy/deployment/rbac/clusterrole.yaml`: new `configmaps`
  get/create/patch/delete rule (first such rule); secrets rule comment notes
  kataConfigRef source reads.

### Why
Phase 2 of ADR 0002 — project the referenced kata drop-in into a per-node
ConfigMap on the child cluster and stamp the opt-in label, so the Phase 3 node
agent can consume it. Implements the `rel-controller-kata-config-projection`
relationship already modeled in CALM.

### Impact
- [ ] Breaking change
- [x] Requires cluster rollout
- [ ] Config change only
- [ ] Documentation only

> Apply the updated controller ClusterRole (new `configmaps` rule) before
> rolling out the controller, or kata-config projection will fail with RBAC
> Forbidden. Behavior is inert for ScheduledMachines without `kataConfigRef`.

---

## [2026-06-08 01:00] - Kata config delivery Phase 1 — CRD `spec.kataConfigRef`

**Author:** Erick Bourgeois

### Changed
- `src/crd.rs`: new `kata_config_ref: Option<KataConfigRef>` field on
  `ScheduledMachineSpec`; new `KataConfigRef` struct (kind/name/key/destPath/
  restartService, `deny_unknown_fields`, in-namespace only) and
  `KataConfigSourceKind` enum (`ConfigMap`/`Secret`); four field schema
  functions and three default functions.
- `src/crd_tests.rs`: 12 new TDD tests (serde defaults, round-trip, PascalCase
  kind, rejects invalid kind / unknown fields, required kind+name, optional in
  schema, name/destPath/restartService schema bounds). Existing spec literals
  updated with `kata_config_ref: None`.
- `src/reconcilers/{scheduled_machine_tests,helpers_tests,child_client_tests}.rs`,
  `tests/integration_child_kubeconfig.rs`: spec literals updated with
  `kata_config_ref: None`.
- `src/bin/crddoc.rs`: documents the new `kataConfigRef` field + example.
- `deploy/crds/scheduledmachine.yaml`, `docs/src/reference/api.md`: regenerated
  (`make crds` → `make crddoc`).

### Why
Phase 1 of ADR 0002 — establish the `spec.kataConfigRef` CRD contract (types,
schema bounds, serde defaults) test-first before the controller projection
(Phase 2) and node agent (Phases 3–4). No behavioral/controller code yet.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [x] Config change only
- [ ] Documentation only

> CRD schema change (additive, optional field). Apply the regenerated
> `deploy/crds/scheduledmachine.yaml` to pick up `kataConfigRef`; existing
> ScheduledMachines are unaffected (field is optional, omitted when unset).

---

## [2026-06-08 00:00] - ADR + CALM: per-node Kata config delivery (design only)

**Author:** Erick Bourgeois

### Changed
- `docs/adr/0002-per-node-kata-config-delivery.md`: new ADR (Proposed) — the
  `spec.kataConfigRef` contract and controller-side per-node ConfigMap
  projection, mirroring the reclaim-agent projection; in-namespace source only
  (like `KubeconfigSecretRef`).
- `docs/adr/0003-in-pod-host-service-restart-via-nsenter.md`: new ADR (Proposed)
  — the privileged `5spot-kata-config-agent` DaemonSet that writes the host
  drop-in and restarts the host k0s service via `nsenter -t 1 … systemctl
  restart`, with the applied-hash annotation as the restart-loop guard.
- `docs/adr/README.md`: index updated with ADR-0002 and ADR-0003.
- `docs/architecture/calm/architecture.json`: modeled the feature — new nodes
  `service-kata-config-agent` and `data-asset-kata-config-configmap`; new
  relationships `rel-controller-kata-config-projection`,
  `rel-kata-agent-workload-kube-api`, `rel-kata-agent-writes-host`,
  `rel-kata-configmap-stored-in-workload-api`; new flow
  `flow-kata-config-delivery`; `adrs` linked. `make calm-validate` passes;
  `make calm-diagrams` renders.

### Why
ADD requires the decision to be recorded and modeled before code. This entry
covers the ADR + CALM deliverables only; no Rust, CRD, or manifest changes yet —
implementation (TDD) is gated on review of these two ADRs.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [x] Documentation only

---

## [2026-05-31 00:00] - Adopt Architecture Driven Development (ADD) methodology

**Author:** Erick Bourgeois

### Changed
- `docs/adr/`: new directory for Architecture Decision Records —
  `template.md`, `README.md` (conventions + index), and
  `0001-adopt-architecture-driven-development.md` (Accepted) recording the
  adoption decision and rationale.
- `.claude/rules/architecture-driven-development.md`: new rule file defining the
  ADD methodology (`ADR → CALM → TDD → implement → docs`), adapted to 5-Spot
  (`src/crd.rs` source of truth, `make crds`/`crddoc`, CALM targets,
  `cargo-quality`/`sync-docs`, roadmaps external at `~/dev/roadmaps/`).
- `.claude/CLAUDE.md`: added ADD as the governing methodology (new section + as
  the first "CRITICAL Coding Pattern"); links to the rule file and ADR-0001.
- `.claude/SKILL.md`: added an ADD gate to the top of `pre-commit-checklist`
  (ADR written/indexed; CALM updated or "no CALM impact"; ADR precedes code).

### Why
5-Spot already had strong TDD and a CALM model, but no first-class, ordered way
to record *why* an architecturally significant change was made before building
it. ADD (per the sibling banlieue project) layers ADR + CALM on top of TDD so
decisions are auditable and traceable — fitting the regulated context — instead
of reconstructed from diffs and changelog entries. See ADR-0001.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [x] Process / documentation only

## [2026-05-30 12:00] - Revert docs deploy trigger to release:published (CodeQL: pwn-request + cache poisoning)

**Author:** Erick Bourgeois

### Changed
- `.github/workflows/docs.yaml`: reverted the docs **deploy trigger** from
  `workflow_run` (Build completion) back to the trusted `release: published`
  event, removing the `ref: github.event.workflow_run.head_sha` checkout. The
  file now matches `main`'s known-clean version (trusted trigger, no
  event-derived checkout ref, plain `actions/cache`).

### Why
PR #71 had switched the docs deploy trigger to `workflow_run` to gate deploy on
Build success. That introduced 7 CodeQL alerts — 5 critical "Checkout of
untrusted code in a privileged context" and 2 high "Cache Poisoning":
`workflow_run` is a privileged trigger (default-branch context, secrets, write
perms) that fired for every Build completion (incl. fork PRs), then checked out
`head_sha` and ran build steps. A first attempt to *harden* the workflow_run
path (job-level release-only `if` gate + cache restore/save split + least-
privilege id-token) did **not** clear the alerts — CodeQL's queries key
syntactically on the untrusted `head_sha` checkout under a privileged trigger,
regardless of guards. The only root-cause fix is to stop checking out the
untrusted ref, so the trigger was reverted to `release: published`. For a
release, `github.ref` is the tag, so docs still build from the exact released
commit. Trade-off accepted: docs deploy on release publication even if the
binary/image Build job failed (the prior, vulnerable, behaviour gated on it).

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [x] CI / security (workflow only)

## [2026-05-30 00:00] - Embedded metadata: reject name/namespace, support labels/annotations

**Author:** Erick Bourgeois

### Changed
- `src/crd.rs`: `embedded_resource_schema` now declares an optional `metadata`
  block with typed `labels`/`annotations` (string maps) and
  `x-kubernetes-preserve-unknown-fields: true` (so `metadata.namespace`/`name`
  are retained for rejection rather than pruned). Added `EmbeddedResource`
  accessors `metadata_namespace()`, `metadata_name()`, `metadata_labels()`,
  `metadata_annotations()` + `embedded_string_map()` helper.
- `src/reconcilers/helpers.rs`: New `validate_embedded_metadata()` — rejects
  controller-owned `metadata.name`/`metadata.namespace` and reserved-prefix
  labels/annotations; called for both specs in `add_machine_to_cluster()`. New
  `merge_owned_labels()` merges user labels with controller-owned tracking
  labels (controller wins). Bootstrap/infra resources now carry user-supplied
  `metadata.labels`/`annotations`.
- `src/constants.rs`: Added `SCHEDULED_MACHINE_LABEL` (promotes a 3×-repeated
  literal to a constant).
- `deploy/admission/validatingadmissionpolicy.yaml`: Added rules 13c–13f
  rejecting `metadata.namespace`/`metadata.name` on both embedded specs.
- `src/bin/crddoc.rs` + `docs/src/reference/api.md`: Documented embedded
  `metadata.labels`/`annotations` and the name/namespace rejection.
- `deploy/crds/scheduledmachine.yaml`: Regenerated (crdgen).
- `docs/src/security/admission-validation.md`, `docs/src/security/threat-model.md`:
  Documented rules 13c–13f and hardened threat T1.
- `src/crd_tests.rs`, `src/reconcilers/helpers_tests.rs`: Added 14 tests
  (metadata accessors, schema shape, validate_embedded_metadata paths,
  merge_owned_labels precedence).

### Why
A user could set `bootstrapSpec.metadata.namespace`/`name`; the API server
silently *pruned* it (no error), and the controller ignored it. The user
requested a loud rejection. Pruning happens before admission policies run and
`additionalProperties: false` is a no-op for CRDs, so the embedded `metadata`
subtree is now `x-kubernetes-preserve-unknown-fields` to let both the VAP and a
runtime check reject the field explicitly. Per follow-up, `metadata.labels`/
`annotations` are now supported (safe: run through the reserved-prefix allowlist
so `cluster.x-k8s.io/*`/`5spot.finos.org/*` cannot be forged — threat T2).

### Impact
- [ ] Breaking change
- [x] Requires cluster rollout
- [ ] Config change only
- [ ] Documentation only

> **Note:** Re-apply `deploy/crds/scheduledmachine.yaml` and
> `deploy/admission/validatingadmissionpolicy.yaml`. Existing specs without an
> embedded `metadata` block are unaffected.

## [2026-05-29 00:00] - RBAC validation for bootstrapSpec & infrastructureSpec

**Author:** Erick Bourgeois

### Changed
- `src/reconcilers/helpers.rs`: Added `ensure_can_create()` — a pre-flight
  `SelfSubjectAccessReview` confirming the controller's service account may
  `create` the embedded bootstrap, infrastructure, and CAPI `Machine` resource
  types before any are created in `add_machine_to_cluster()`. Extracted
  `resource_plural()` helper (dedupes the two plural-derivation sites) so the
  access review and the create/delete calls always target the same RBAC
  resource.
- `src/reconcilers/scheduled_machine.rs`: New `ReconcilerError::PermissionDenied`
  variant + `record_error("permission_denied")` metric mapping.
- `src/constants.rs`: Added `RBAC_VERB_CREATE`.
- `src/reconcilers/helpers_tests.rs`: Added 8 tests — `resource_plural` cases
  and `ensure_can_create` allowed / denied / denied-no-reason / API-error paths
  (tower mock).
- `deploy/admission/validatingadmissionpolicy.yaml`: Added `spec.variables`
  (group + plural derivation) and two `authorizer.check('create')` validations
  (rules 13a/13b) requiring the *creating user* to hold `create` on the embedded
  bootstrap/infrastructure GVKs — privilege-escalation guard at admission.
- `docs/src/security/admission-validation.md`: Documented rules 13a/13b and the
  two-layer (user @ admission, controller SA @ reconcile) design.
- `docs/src/security/threat-model.md`: Added mitigated threat E1a (escalation
  through the controller).

### Why
A user able to create a `ScheduledMachine` but not the embedded bootstrap/
infrastructure resource could otherwise have the broadly-permissioned controller
create it on their behalf — a privilege escalation through the controller. The
admission `authorizer` checks gate the requesting user; the controller's own SA
is independently gated at reconcile so an RBAC gap surfaces as a clear
`PermissionDenied` instead of an opaque 403 mid-creation.

### Impact
- [ ] Breaking change
- [x] Requires cluster rollout
- [ ] Config change only
- [ ] Documentation only

> **Note:** Re-apply `deploy/admission/validatingadmissionpolicy.yaml`. The
> controller's RBAC is unchanged, but users creating `ScheduledMachine`
> resources now additionally need `create` RBAC on the embedded bootstrap/
> infrastructure resource types in their namespace.

## [2026-05-11 10:00] - Child-cluster Node-watch multiplexer + Phase 2 metrics

**Author:** Erick Bourgeois

### Added
- `src/reconcilers/child_client.rs`: new `ChildWatchHook` trait
  (`on_child_resolved` / `on_child_evicted`) that the cache fires when a
  child `kube::Client` is first built, when a token rotation rebuilds
  one, or when an entry is evicted (manual or LRU). Defaults to a no-op,
  so unit tests + degenerate deployments are unaffected. The cache fires
  the eviction-before-resolved sequence on every RV-driven rebuild so
  the manager never has two watchers running on the same `CacheKey` with
  different credentials.
- `src/reconcilers/child_watch.rs` (~220 LOC) + `_tests.rs` (~270 LOC):
  new `ChildNodeWatchManager` implementing `ChildWatchHook`. Owns a
  `HashMap<CacheKey, JoinHandle<()>>` and spawns one `kube::runtime::
  watcher::watcher` task per active child cluster. Each task maps
  `Node` events through the existing canonical
  `node_to_scheduled_machines_via_machine` (using a snapshot of the
  management cluster's CAPI Machine reflector) and pushes the resulting
  `ObjectRef<ScheduledMachine>`s on a shared `mpsc::Sender`. 10
  lifecycle tests cover spawn-on-resolve, abort-on-evict, idempotent
  back-to-back resolve, double-key independence, and the
  receiver-dropped graceful path.
- `src/main.rs`: builds the mpsc channel, constructs the manager from
  `(tx, machine_store)`, installs it on the cache via
  `context.child_clients.set_hook(...)`, and feeds the receiver into
  `Controller::reconcile_on(ReceiverStream::new(rx))`. Closes the
  event-driven gap left by the Phase 1 resolver work: a child-cluster
  Node going `NotReady` / being cordoned externally / changing labels
  now triggers a reconcile of the owning `ScheduledMachine` within
  watcher latency rather than waiting for the periodic requeue.
- `src/metrics.rs`: two new Prometheus counters —
  `fivespot_child_kubeconfig_resolutions_total{result}` (labels:
  `management`, `child_explicit`, `child_auto`, `cache_hit`, `rebuild`,
  `error`) and `fivespot_child_kubeconfig_errors_total{reason}` (labels:
  `secret_missing_key`, `invalid_yaml`, `unreachable`,
  `non_404_kube_error`). Helper fns `record_child_kubeconfig_resolution`
  + `record_child_kubeconfig_error` wired into
  `ChildClientCache::resolve`.
- `src/constants.rs`: `CHILD_NODE_EVENT_CHANNEL_CAP = 1024` for the
  shared mpsc buffer; `CONDITION_TYPE_CHILD_CLUSTER_REACHABLE =
  "ChildClusterReachable"` (constant added now; the writer wiring is
  deferred — see "Deferred" below).
- `tests/integration_child_kubeconfig.rs` (~330 LOC): hermetic
  end-to-end test exercising the full Context → cache → manager wiring
  via `tower_test::mock`. 5 cases cover:
  explicit-ref resolve spawns one watcher; auto-discovery 404 falls
  back to management with no watcher; RV rotation cancels + restarts
  exactly one watcher; explicit `evict()` cancels; explicit-ref 404
  fails closed (errors AND spawns no watcher).
- 5 new resolver tests in `src/reconcilers/child_client_tests.rs` for
  the hook plumbing: fires-once-on-initial-build, does-not-fire-on-
  cache-hit-same-RV, fires-evicted-then-resolved-on-RV-change,
  explicit-evict-fires-once-per-removal, does-not-fire-for-management-
  fallback.

### Changed
- `Cargo.toml`:
  - Added `tokio-stream = "0.1"` (direct dep) for `ReceiverStream` —
    adapts `mpsc::Receiver` into the `Stream` that
    `Controller::reconcile_on` expects.
  - Enabled the `unstable-runtime` feature on the `kube` dep so
    `Controller::reconcile_on` is callable. The feature is gated as
    "unstable" by upstream (the API may shift between kube-runtime
    minor versions), but it is the canonical way to push external
    triggers into the controller loop and has no replacement in the
    stable surface area.

### Why
The Phase 1 work (2026-05-10) routed Node/Pod operations to the child
cluster correctly but left Node *events* uncovered: a child-cluster
Node going `NotReady`, getting cordoned by an out-of-band operator, or
being deleted upstream would not trigger a reconcile until the
periodic `TIMER_REQUEUE_SECS` requeue (worst case ~60s of latency for
drain progress / status enrichment). This change closes that gap with
one watcher per unique kubeconfig Secret, sharing watcher work across
all SMs that target the same child cluster. The Phase 2 metrics give
operators visibility into resolution outcomes (hit / rebuild / error)
without log-grepping.

### Impact
- [ ] Breaking change
- [x] Requires cluster rollout (controller binary changed; behaviour
      additive — co-located deployments without `kubeconfigSecretRef`
      and without an auto-discoverable `<clusterName>-kubeconfig`
      Secret see no change)
- [ ] Config change only
- [ ] Documentation only

Operational notes:
- Each unique `(namespace, secret_name)` pair backs exactly one
  watcher task. A deployment with 100 SMs spread across 5 child
  clusters runs 5 child-cluster Node watchers, not 100.
- Token / cert rotations driven by CAPI's control-plane provider bump
  the kubeconfig Secret's `resourceVersion`. The cache detects this on
  the next `resolve()`, aborts the stale watcher, and starts a fresh
  one with the rebuilt `Client` — no controller restart required.
- The mpsc buffer (`CHILD_NODE_EVENT_CHANNEL_CAP = 1024`) is sized for
  initial-list bursts across all child clusters; a back-pressure
  scenario (channel full) is logged at debug level and means the
  controller is shutting down.

### Deferred
- **`ChildClusterReachable` condition writer.** The condition type
  constant is in place (`CONDITION_TYPE_CHILD_CLUSTER_REACHABLE`) but
  the writer wiring (translate each `resolve()` outcome into a status
  patch on the owning SM) is left for a follow-up — it requires
  threading the SM ref + the recorder through each call site, which is
  more invasive than the metrics work that closes the same
  observability need today.
- **Real-cluster dual-cluster integration test.** The hermetic
  integration test under `tests/` exercises the wiring deterministically;
  a `kind`-based two-cluster test would add HTTP-level coverage and
  fits the pattern of the existing `integration_node_taints.rs` /
  `integration_emergency_reclaim.rs` real-cluster tests. Left for the
  next development-environment standardisation pass.

---

## [2026-05-10 14:00] - Child-cluster kubeconfig support (CRD + resolver + routing)

**Author:** Erick Bourgeois

### Added
- `src/crd.rs`: New optional `spec.kubeconfigSecretRef` field on
  `ScheduledMachineSpec` plus a new `KubeconfigSecretRef { name, key }`
  struct (with `deny_unknown_fields` to forbid cross-namespace `namespace`
  fields by design). Two new schema helpers
  (`kubeconfig_secret_name_schema`, `kubeconfig_secret_key_schema`) bound
  the name (RFC-1123 DNS subdomain, 253 chars) and the data key. Default
  for `key` is `"value"` — CAPI's convention.
- `src/reconcilers/child_client.rs` (~400 LOC) + `_tests.rs` (~470 LOC):
  new `ChildClientCache` resolver. Public surface: `CacheKey`,
  `ResolvedClient::{Management, Child}`, `ChildClientCache::resolve`
  (and `peek` / `evict` helpers for future per-child watch integration).
  Resolution order: explicit `spec.kubeconfigSecretRef` → auto-discover
  `<spec.clusterName>-kubeconfig` Secret in the SM namespace →
  management-client fallback. Cache keyed by `(namespace, secret_name)`
  and invalidated by Secret `resourceVersion`; bounded-LRU at
  `CHILD_CLIENT_CACHE_CAP = 256`. Built child `kube::Client` inherits
  the same `K8S_API_TIMEOUT_SECS` wire timeouts as the management
  client. 12 new tests cover happy paths, cache hit/miss, RV-rebuild,
  fail-closed for explicit-ref 404, 403 propagation, missing data key,
  and invalid YAML.
- `src/reconcilers/scheduled_machine.rs`: three new `ReconcilerError`
  variants — `KubeconfigSecretMissingKey`, `KubeconfigInvalid`,
  `ChildClusterUnreachable` — each mapped to a distinct Prometheus
  error label via the existing `record_error` switch.
- `src/constants.rs`: `CHILD_CLIENT_CACHE_CAP = 256`.
- `Context.child_clients: Arc<ChildClientCache>` field, constructed in
  `Context::new`. Cheap (`Arc<RwLock<HashMap>>`) so existing
  single-instance deployments pay nothing.
- `docs/src/concepts/child-cluster-kubeconfig.md`: full RBAC
  requirements, resolution order, cache invalidation semantics, and
  threat model.
- `examples/scheduledmachine-child-cluster.yaml`: end-to-end example
  with an explicit `kubeconfigSecretRef`.
- 9 new CRD round-trip tests in `src/crd_tests.rs` covering: optional
  field round-trip, default-key, `deny_unknown_fields`, required
  `name`, schema-bounded name + key, and the new field being absent
  from the `required` list (backward compat).

### Changed
- `src/reconcilers/scheduled_machine.rs`:
  - `provision_reclaim_agent_best_effort` resolves the child client at
    its top and threads it into `reconcile_reclaim_agent_provision`
    (the ConfigMap consumed by the reclaim-agent DaemonSet lives on
    workload nodes; previously this projected to the wrong cluster).
  - `reconcile_node_taints_best_effort` resolves and threads to
    `reconcile_node_taints` so user-declared taints land on
    workload-cluster Nodes.
  - `handle_shutting_down_phase` resolves and threads to
    `drain_node_with_timeout`. The CAPI Machine lookup
    (`get_node_from_machine`) stays on the management client where
    Machine objects live.
- `src/reconcilers/helpers.rs::handle_emergency_remove`: resolves the
  child client once at the top of the flow and threads it to
  `drain_node_with_timeout` + `clear_reclaim_annotations_best_effort`.
  Schedule-disable patch + Machine deletion stay on the management
  client.
- `deploy/crds/scheduledmachine.yaml`: regenerated via
  `cargo run --bin crdgen` to surface the new field in the OpenAPI
  schema (additive — no breaking change to existing CRs).
- `Cargo.toml`: added `base64 = "0.22"` as a dev-dependency (used by
  `child_client_tests.rs` to build base64-encoded Secret payloads for
  the `tower-test` mock server).

### Why
In a CAPI + k0smotron / k0rdent topology, the `Node` and `Pod` objects
5-Spot manipulates (cordon, drain, taint, reclaim-agent labels /
ConfigMaps) live inside the workload (child) cluster, not the
management cluster. The previous single-client model worked correctly
only in the degenerate co-located case (management ≡ workload),
which is the dev/test posture rather than production. This change
makes 5-Spot multi-cluster-correct: a `ScheduledMachine` carrying a
`kubeconfigSecretRef` (or sitting in a namespace with a CAPI-convention
`<clusterName>-kubeconfig` Secret) has its Node/Pod traffic routed
through the child kubeconfig; everything else (the SM CR, the CAPI
Machine, bootstrap, infrastructure) stays on the management client.

Fail-closed by design: a misconfigured or unparseable kubeconfig
surfaces as `ReconcilerError::ChildClusterUnreachable` and back-offs,
rather than silently falling through to the management client and
routing operations to the wrong cluster.

### Impact
- [ ] Breaking change (additive optional field; existing SMs unchanged)
- [x] Requires cluster rollout (controller binary changed; CRD schema
      additive but should be re-applied)
- [ ] Config change only
- [ ] Documentation only

Operational notes:
- Existing single-cluster deployments without a
  `<clusterName>-kubeconfig` Secret in the SM namespace continue to use
  the management client for Node/Pod operations — zero behavioural
  change.
- Operators converting an existing single-cluster SM to a child-cluster
  SM should create the CAPI-format Secret in the SM's namespace
  *before* adding the ref, so the first reconcile sees a valid Secret.
- Token / certificate rotations driven by CAPI's control-plane provider
  bump the Secret's `resourceVersion`; the next reconcile rebuilds the
  child client automatically.

### Deferred to follow-up
- **Per-child-cluster Node watch (Phase 1.9 in
  `~/.claude/plans/i-have-a-very-swift-treehouse.md`).** The management
  Node watch in `main.rs` is unchanged, so co-located deployments keep
  their Node-event-driven responsiveness. Child-cluster Node state
  changes are picked up via the periodic `TIMER_REQUEUE_SECS` requeue
  and via CAPI `Machine` status changes. A subsequent change will add
  lazy per-`(namespace, secret_name)` Node watchers multiplexed into
  the `Controller::reconcile_on` channel via a `mpsc` receiver, closing
  the event-driven gap. Tracked separately.
- **`ChildClusterReachable` status condition + Prometheus counters
  (Phase 2 in the plan).**
- **Integration test against two fake clusters (Phase 1.10 in the
  plan).** Resolver is unit-tested end-to-end against
  `tower_test::mock`; the full reconcile-with-dual-clusters scenario
  is the next test to write.

---

## [2026-05-10 12:00] - Gate docs Pages deploy on successful Build workflow completion

**Author:** Erick Bourgeois

### Changed
- `.github/workflows/docs.yaml`: Replaced the `release: published` deploy
  trigger with `workflow_run` chained to the `Build` workflow's
  completion. Setup Pages, Upload Pages artifact, and the Deploy job
  are now gated on
  `github.event.workflow_run.event == 'release' && ...conclusion == 'success'`. The build job pins
  `actions/checkout` to `github.event.workflow_run.head_sha` on
  workflow_run runs (workflow_run executes in the default-branch
  context, so released docs would otherwise build from `main` rather
  than the tag commit). Concurrency group now keys on
  `workflow_run.head_branch`; `cancel-in-progress` is disabled only
  for release-triggered workflow_run events.

### Why
A v0.2.1 docs deploy was rejected by the `github-pages` environment
protection rules and sat unpublished for ~10 hours until re-run.
Beyond the protection-rule fix, the prior trigger model published
docs whenever a release was created — independent of whether the
release's binaries, container images, and Cosign signatures actually
built cleanly. Chaining the Pages deploy to a successful `Build` run
means: a release whose artifacts fail to build no longer ships docs
claiming those artifacts exist, and end users on the published site
can trust that every visible release tag corresponds to a complete,
signed artifact set.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [x] Config change only
- [ ] Documentation only

Operational notes:
- Docs deploy now waits for `Build` to finish on each release
  (~10–20 min added latency between release publication and Pages
  going live).
- `Build` PR / push-to-main completions trigger a docs validation
  run via `workflow_run` in addition to the existing
  `pull_request` / `push` triggers — redundant by design (a
  Build-completion sanity check); short-circuits before deploy.
- `workflow_run` triggers ignore the `paths:` filter, so every
  `Build` completion shows up in the Actions log even when no docs
  sources changed.

---

## [2026-05-02 23:30] - Emergency-reclaim roadmap closeout: drain stopwatch + loop protection + integration tests

**Author:** Erick Bourgeois

### Added
- `src/loop_protection.rs` (~75 LOC) + `src/loop_protection_tests.rs`
  (14 pure-helper tests). Module exposes `prune_old_reclaim_events`,
  `record_emergency_reclaim_event`, and `should_warn_rapid_re_reclaim`
  — all operate on `&mut VecDeque<DateTime<Utc>>` so the tests drive
  every branch with no clock or controller. Threshold + window come
  from new constants in `src/constants.rs`
  (`RAPID_RE_RECLAIM_THRESHOLD = 3`,
  `RAPID_RE_RECLAIM_WINDOW_SECS = 600`,
  `RAPID_RE_RECLAIM_MAX_TRACKED = 10`,
  `REASON_RAPID_RE_RECLAIM = "RapidReReclaim"`).
- `Context.recent_reclaims:
  Arc<Mutex<HashMap<String, VecDeque<DateTime<Utc>>>>>` — in-memory
  per-SM tracker, keyed by `"namespace/name"`. Documented choice:
  no CRD schema bump, no status-patch round trip; controller restart
  resets the count, which the metric still captures on a trend basis.
- `src/metrics.rs`: three new metrics with helpers —
  `fivespot_emergency_drain_duration_seconds` (HistogramVec,
  outcome={success|timeout|error}, buckets sized for the 60 s drain
  ceiling), `fivespot_emergency_reclaims_total` (CounterVec by
  namespace+name), `fivespot_rapid_re_reclaims_total` (CounterVec by
  namespace+name). Helpers `record_emergency_drain`,
  `record_emergency_reclaim`, `record_rapid_re_reclaim` plus
  `EmergencyDrainOutcome` enum. 4 new tests in `metrics_tests.rs`.
- `src/reconcilers/helpers.rs`: drain call in `handle_emergency_remove`
  is now wrapped in an `Instant::now()` stopwatch; outcome classified
  by error message (timeout substring → Timeout, other Err → Error,
  Ok → Success). Per-SM emergency-reclaim counter bumped once per
  flow (the idempotent recovery handler short-circuits before this
  point on restart). Loop-protection check fires after the bump:
  appends timestamp, prunes old entries, emits `RapidReReclaim`
  Warning Event + bumps the new metric when threshold crosses. New
  pure helper `build_rapid_re_reclaim_event(name)`.
- `tests/integration_netlink_proc.rs` — Linux-only (`#![cfg(target_os
  = "linux")]`) runtime test for the netlink subscriber. Two
  `#[ignore]` cases: one targets a spawned `/bin/true` child and
  asserts <100 ms detection latency per issue #40 acceptance
  criterion; one is a quieter "subscriber sees *some* event"
  smoke test.
- `tests/integration_emergency_reclaim.rs` — kind-cluster annotation
  contract test (graceful-skip if no cluster, `#[ignore]`). Two cases:
  agent's `build_patch_body` round-trips through the controller's
  `node_reclaim_request` parser; partial annotations (reason set,
  requested=true absent) parse to `None`. Plus one always-on
  constants-pinning test.

### Changed
- `src/reconcilers/mod.rs`: re-export `build_clear_reclaim_patch`,
  `node_reclaim_request`, and `ReclaimRequest` from `helpers` so the
  kind-cluster test can use the published API surface.
- Roadmap `~/dev/roadmaps/5spot-emergency-reclaim-by-process-match.md`
  → renamed `completed-5spot-emergency-reclaim-by-process-match.md`
  (per the user-memory rule "rename completed roadmaps with
  `completed-` prefix when every phase is done"). Status header
  rewritten to ✅ Complete; Phase 2.c rung 2 section flipped from
  "deferred to issue #40" to fully shipped; deferred-items section
  rewritten as "Closed follow-ups" with the four checkboxes ticked.

### Why
Closes the four open items the rung 2 implementation called out as
explicit follow-ups:

1. **Drain stopwatch** — operators now have a Prometheus histogram of
   actual emergency-drain duration sliced by outcome, so SLO dashboards
   can compute success-only P95 vs timeout-rate side by side.
2. **Re-enable loop protection** — when a user re-enables a SM whose
   conflicting process is still running, the third reclaim within
   10 minutes now triggers a `RapidReReclaim` Warning Event telling the
   operator to stop the conflicting process before re-enabling. The
   loop is no longer silent.
3. **Linux netlink runtime test** — issue #40 acceptance criterion
   "JVM launch produces a match within <100 ms" is now verifiable via
   `cargo test --test integration_netlink_proc -- --ignored` on a real
   Linux node with `CAP_NET_ADMIN`.
4. **Kind-cluster integration test** — proves the agent → controller
   annotation contract round-trips through a real API server. Catches
   the entire class of "agent writes annotations the controller can't
   parse" regressions that no unit test would surface.

With these closed, every phase of the
`5spot-emergency-reclaim-by-process-match.md` roadmap is shipped and
the file moves to `completed-` prefix.

### Impact
- [ ] Breaking change (no API surface change; new metrics + new event
      reason are additive)
- [x] Requires cluster rollout (binary changes; CRD unchanged so no
      `kubectl apply` of the CRD YAML needed)
- [ ] Config change only
- [ ] Documentation only

### Verification
- `cargo build`: clean.
- `cargo test`: **450 unit tests + 3 always-on integration tests pass**;
  4 `#[ignore]` integration tests (2 emergency-reclaim, 2 netlink) gated
  on a real cluster / Linux + CAP_NET_ADMIN respectively.
- `cargo fmt --check`: clean.
- `cargo clippy --all-targets -- -D warnings`: clean.

---

## [2026-05-02 23:00] - Reclaim-agent rung 2: netlink proc connector subscriber

**Author:** Erick Bourgeois

### Added
- `src/netlink_proc.rs` (~280 LOC) — new module exposing `ProcEvent`,
  `NetlinkError`, portable parsers (`parse_cn_msg`, `parse_proc_event`),
  and `Subscriber`. Linux impl opens an `AF_NETLINK` /
  `NETLINK_CONNECTOR` socket, binds to the `CN_IDX_PROC` multicast
  group, sends the `PROC_CN_MCAST_LISTEN` control message, and yields
  one `ProcEvent` per kernel `exec(2)`. Non-Linux: `Subscriber::new()`
  immediately returns `NetlinkError::Unsupported` so the binary still
  links cleanly on macOS / Windows builds.
- `src/netlink_proc_tests.rs` — 14 byte-level parser tests + 1
  non-Linux stub contract test = **15 new tests, all running on macOS
  CI** without needing a kernel: exec / fork / exit classification,
  truncated buffers, wrong connector id, zero-length payload,
  cn_msg→proc_event round-trip, non-Linux `Subscriber::new` returns
  `Unsupported`.
- `src/bin/reclaim_agent.rs`: new `--detector={auto|netlink|poll}`
  flag (env `RECLAIM_DETECTOR`, default `auto`). `auto` selects
  `netlink` on Linux and `poll` elsewhere. New `run_netlink_scanner`
  loop runs alongside the existing `run_scanner`; selection happens
  once after host-id verification, before either loop starts. The
  netlink loop opens the subscriber only when armed by a non-empty
  ConfigMap and tears it down on config-clear, so an idle agent holds
  no kernel resources. The `recv(2)` syscall runs on a `spawn_blocking`
  thread; `tokio::select!` cancels via the existing config watch
  channel.
- `Cargo.toml`: `target.'cfg(target_os = "linux")'.dependencies.nix =
  { version = "0.30", default-features = false, features = ["socket",
  "net", "uio"] }`. Linux-only — non-Linux builds pull in nothing new.
- `deploy/node-agent/daemonset.yaml`: container `securityContext`
  gains `capabilities.add: [NET_ADMIN]`. The pod-level comment block
  rewritten to reflect that this is no longer "future work."
- `.trivyignore`: new `AVD-KSV-0022` entry justifying the
  `CAP_NET_ADMIN` add against the existing architectural-necessity
  rationale pattern (matches the KSV-0010 / KSV-0012 / KSV-0023 etc.
  blocks already in the file).
- `docs/src/concepts/emergency-reclaim.md`: new "Detector — rung 1
  vs rung 2" subsection with the comparison table, selection flags,
  the heavy-exec-storm tradeoff, and a note on why the cap is granted
  pod-level rather than per-detector.

### Changed
- `src/reclaim_agent.rs`: `match_pid` visibility raised from `fn` to
  `pub fn` so the rung 2 netlink subscriber can resolve a kernel-pushed
  pid through the same match logic rung 1 uses. Body and behaviour
  unchanged. Existing 46 tests stay green.
- `src/lib.rs`: `pub mod netlink_proc` added; module list comment
  block extended.
- `docs/src/concepts/emergency-reclaim.md` status header updated:
  rung 2 is shipped, no longer "follow-up work."

### Why
Closes the deferred Phase 2 rung 2 of
`~/dev/roadmaps/5spot-emergency-reclaim-by-process-match.md` and
GitHub issue #40. Rung 1 already meets the <5 s SLA, so this is
optimisation, not correctness — payoff is detection latency dropping
from up to 250 ms (worst-case poll phase) to <10 ms (kernel push)
and idle CPU dropping to zero on quiet nodes. The detector is opt-in
via `--detector=netlink`; `auto` (the default) picks netlink on Linux
and poll elsewhere. Operators on heavy-exec workloads can pin
`--detector=poll` to avoid the per-exec cost.

The netlink ABI is identical across Linux x86_64 and aarch64; both
build targets in `.github/workflows/build.yaml` get rung 2 with no
per-arch code. Windows is out of scope (no netlink ABI; would need an
ETW-based sibling impl behind the same trait, separate effort).

### Impact
- [ ] Breaking change (new flag is opt-in via `auto` default; existing
      operators with `RECLAIM_DETECTOR` unset continue on the
      platform-appropriate default; rung 1 stays available as `poll`)
- [x] Requires cluster rollout (DaemonSet manifest changes —
      `kubectl apply -k deploy/node-agent/` to pick up `CAP_NET_ADMIN`
      and the new env var support)
- [ ] Config change only
- [ ] Documentation only

### Verification
- `cargo build --bin 5spot-reclaim-agent`: clean.
- `cargo test --lib netlink_proc`: 15/15 pass on macOS aarch64.
- `cargo test --lib reclaim_agent`: 46/46 pass (unchanged from
  pre-change baseline).
- `cargo fmt --check` + `cargo clippy --all-targets -- -D warnings`:
  clean.
- Linux runtime verification deferred to CI (kind cluster) and to
  hardware testing — same gating as rung 1's integration test scope.
  The macOS dev loop uses `--detector=poll` and exercises the bin
  end-to-end without netlink.

### Trust model (unchanged)
The netlink socket is opened with `CAP_NET_ADMIN` granted at the
container level. The node-scoped credentials posture is unchanged —
the agent still binds only to the proc-event multicast group and
cannot send control messages to other netlink subsystems. The
host-identity cross-check (Phase 4 of the 2026-04-25 security audit)
still gates every PATCH regardless of detector choice.

---

## [2026-05-02 10:00] - Symbol-import auto-VEX (roadmap Phase 3, scope-revised)

**Author:** Erick Bourgeois

### Added
- `src/auto_vex_reachability.rs` (library module, ~190 LOC) +
  `src/auto_vex_reachability_tests.rs` (20 unit tests, 100%
  positive/negative/exception coverage per project rule):
  - `parse_nm_output` — parses `nm -D --undefined-only` text into a
    set of imported symbol names; strips `@GLIBC_x.y` version
    suffixes; defensively skips lines whose type column isn't `U`.
  - `load_affected_functions_from_path` — loads a JSON map of CVE id
    → [public-API function names]. Underscore-prefixed keys
    (`_comment`, `_meta`) are sidecar metadata and skipped silently.
  - `compute_reachability_vex` — pure function that, given a Grype
    report, the imported-symbol set, the affected-functions mapping,
    and the already-triaged CVE set, emits one `not_affected +
    vulnerable_code_not_in_execute_path` statement per CVE whose
    listed functions are *all absent* from the imports.
  - Re-exports `Document`, `Statement`, `Vuln`, `Product`,
    `GrypeReport`, `GrypeMatch`, `GrypeVuln`, `GrypeArtifact`,
    `build_document`, `load_triaged_from_vex_dir` from
    `auto_vex_presence` so callers can import the OpenVEX shape from
    one place.
- `src/bin/auto_vex_reachability.rs` — thin clap-driven CLI wrapping
  the library. Reads `--grype-json`, `--binary-symbols` (the path to
  an `nm` output text file), `--affected-functions`, `--vex-dir`,
  `--product-purl`, `--id`, optional `--author` and `--timestamp`.
- `Cargo.toml`: `[[bin]] auto-vex-reachability` entry.
- `.vex/.affected-functions.json` (new, dot-prefixed): curated CVE →
  [function names] mapping covering 9 of the 15 hand-authored
  statements in `.vex/`. Filename is dot-prefixed so default shell
  globs (`vexctl merge .vex/*.json`) skip it. The 6 unmapped CVEs
  (`CVE-2019-1010022/3/4/5`, `CVE-2026-4437`, `GHSA-cq8v-f236-94qc`)
  are not function-bounded — preconditions / adversary-model claims
  / Rust-crate-only — and stay hand-authored. One mapped CVE
  (`CVE-2026-4438`, gethostbyaddr) is *also* retained hand-authored
  because its statement carries the stronger
  `vulnerable_code_cannot_be_controlled_by_adversary` justification;
  the mapping entry stays as a safety net so auto-reachability would
  jump in with the weaker `vulnerable_code_not_in_execute_path` if
  the hand-authored file is ever removed.
- `Makefile`: new `vex-auto-reachability` target that defaults
  `RELEASE_BINARY=target/release/5spot` and
  `AFFECTED_FUNCTIONS=.vex/.affected-functions.json`. macOS Mach-O
  fallback to `nm -gU` since `-D --undefined-only` is ELF-only.
  `.PHONY` extended.
- `.github/workflows/build.yaml`: new `auto-vex-reachability` job
  gated on `github.event_name != 'pull_request'`. `needs: [build,
  extract-version, grype-triage]`. Builds the bin (cached cargo),
  downloads the `5spot-linux-amd64` artifact + the `grype-triage-*`
  artifacts, runs `nm -D --undefined-only` on the binary, runs the
  bin once per variant, merges per-variant outputs via `vexctl
  merge`, uploads `vex-auto-reachability` (the merged VEX doc) and
  `vex-auto-reachability-evidence` (the raw `nm` output, for the
  Security team to re-derive against).

### Changed
- `.github/workflows/build.yaml`:
  - `build-vex` job: `needs` extended with `auto-vex-reachability`.
    Added a download step for the `vex-auto-reachability` artifact
    and an explicit `test -f` guard. `vexctl merge` invocation now
    includes `auto/vex.auto-reachability.json` alongside
    `.vex/*.json` and `auto/vex.auto-presence.json`.
  - Header comments on `build-vex` updated to describe both
    automated phases.
- `src/auto_vex_presence.rs`: `load_triaged_from_vex_dir` now skips
  dot-prefixed `*.json` files, so `.vex/.affected-functions.json`
  isn't mis-parsed as an OpenVEX statement when `auto-vex-reachability`
  loads `--vex-dir`. Added `load_triaged_skips_dotfiles` test in
  `auto_vex_presence_tests.rs`.
- `src/lib.rs`: re-export `pub mod auto_vex_reachability`.
- `.vex/README.md`: rewritten "Automated statements" section to
  describe both Phase 2 (presence) and Phase 3 (reachability)
  artifacts. New sub-section documenting the
  `.vex/.affected-functions.json` mapping format.
- `docs/src/security/vex.md`:
  - CI flow steps renumbered to 7 — added step 3 for
    auto-reachability between presence (step 2) and assemble (step 4).
  - "What we automate" section: added the symbol-import-reachability
    bullet and a new "Why symbol-imports, not LLVM-IR call graphs"
    sub-section explaining the scope revision.
- Roadmap (`~/dev/roadmaps/5spot-automated-vex-generation.md`):
  status line updated to "Phase 3 ✅ shipped (symbol-import variant)".
  Phase 3 section rewritten to document the scope revision with
  rationale. Rollout section updated.

### Removed
- `.vex/CVE-2010-4756.json` (glob/fnmatch — superseded by auto-reachability)
- `.vex/CVE-2018-20796.json` (regcomp/regexec)
- `.vex/CVE-2019-9192.json` (regcomp/regexec)
- `.vex/CVE-2026-27171.json` (crc32_combine[64])
- `.vex/CVE-2026-4046.json` (iconv/iconv_open/iconv_close)
- `.vex/CVE-2026-5358.json` (NIS / yp_*)
- `.vex/CVE-2026-5450.json` (scanf family)
- `.vex/CVE-2026-5928.json` (ungetwc)

These 8 statements all asserted `vulnerable_code_not_in_execute_path`
and are now produced by `auto-vex-reachability` from the curated
mapping + the `nm -D --undefined-only` symbol-import evidence
uploaded by the new CI job. The Security team verifies them by
re-running the same `nm` invocation against the release-attested
binary. `CVE-2026-4438.json` is intentionally retained because its
hand-authored statement carries the stronger
`vulnerable_code_cannot_be_controlled_by_adversary` justification —
see Why below.

### Why
Phase 3 of the automated VEX generation roadmap, with a scope
revision. The original plan targeted Rust LLVM-IR call-graph
reachability against RustSec advisories carrying `affected.functions`.
At implementation time `cargo audit` reports zero open Rust-level
vulnerabilities for this codebase — every CVE in `.vex/` is a
base-image glibc/zlib finding from Grype scanning the Docker image,
and RustSec doesn't track those. The Rust call-graph approach
addressed zero current findings.

The mechanical equivalent for our actual data is symbol-import
absence: if the Rust binary's dynamic-symbol-import table doesn't
contain any of the documented public-API entry points for an
affected glibc function, the affected code path cannot be reached
through documented APIs. This is exactly what the existing 14
hand-authored impact statements claim in prose ("5-Spot performs no
character-set conversions; the glibc iconv() path exercised by this
CVE is unreachable from the controller binary"). Auto-reachability
turns that prose into a verifiable check that the Security team
re-derives during counter-signing.

The shipped variant addresses 9 of the 15 hand-authored statements
through the curated mapping. Of those 9, **8 hand-authored
statements were deleted** in this same change (their
`vulnerable_code_not_in_execute_path` claim is now produced by
auto-reachability and re-derivable from the uploaded `nm` evidence).
The 9th mapped CVE (`CVE-2026-4438`) was retained because its
hand-authored claim is the stronger
`vulnerable_code_cannot_be_controlled_by_adversary` — the auto-VEX
would replace it with the weaker function-absence claim. The
remaining 6 hand-authored statements are not function-bounded
(adversary-model preconditions, ASLR bypasses,
missing-shell-at-runtime, Rust-crate combo) and stay hand-authored.
The original LLVM-IR call-graph approach is deferred to when (a)
`cargo audit` starts returning Rust-side findings with
function-level data, or (b) we adopt a Rust analyzer that exposes
call-graph data without LTO loss.

### Verification
- `cargo fmt --all -- --check`: clean.
- `cargo clippy --all-targets --all-features -- -D warnings`: clean.
- `cargo test --lib`: 418 tests pass (398 pre-existing + 20 new for
  `auto_vex_reachability`).
- Smoke tests:
  - 10 Grype matches (9 mapped + 1 unmapped) + an empty `.vex/` dir +
    nm output without any of the affected glibc symbols → 9
    statements emitted in deterministic CVE-id order. Unmapped CVE
    correctly skipped.
  - Same input but with `glob` added to nm output → 8 statements
    emitted (CVE-2010-4756 correctly excluded because `glob` is
    importable).
  - Same input against the real `.vex/` directory (all 9 mapped CVEs
    are hand-authored there) → 0 statements emitted because
    `load_triaged_from_vex_dir` correctly skips re-deriving
    already-triaged CVEs (and correctly skips
    `.affected-functions.json` thanks to the new dotfile filter).
- `vexctl merge` of the auto-reachability output + `.vex/*.json` +
  vex.auto-presence.json produces a well-formed merged document
  parseable by `vexctl` with no errors.

### Trust model (unchanged from Phase 2)
The merged VEX (hand-authored + auto-presence + auto-reachability)
is Cosign-attested by CI keyless against both image digests. The
Security team independently re-runs `nm -D --undefined-only` against
the release-attested binary and checks each auto-reachability
statement's claim, then counter-signs with their own OIDC identity
if they agree. Downstream consumers gate on both attestations via
`cosign verify-attestation` with twin
`--certificate-identity-regexp` invocations.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [x] Documentation only

*(Not really "documentation only" — net +700 LOC of Rust + tests + a
new CI job — but no runtime, CRD, or API-surface change. Closest
pre-existing box.)*

---

## [2026-04-30 14:00] - Phases 5 + 6 of security audit: RBAC tightening + defence-in-depth docs

**Author:** Erick Bourgeois

### Changed
- `Cargo.toml`: pin `warp = { version = "0.4", default-features = false, features = ["server"] }`.
  warp 0.4 made every previously-default feature opt-in; we only need
  `server` (used by `src/main.rs::run_metrics_server` and
  `src/health.rs::start_health_server`). `default-features = false`
  also drops `multipart`, `websocket`, and `test`, which we don't
  use, shrinking the supply-chain surface visible to cargo-deny /
  Grype / auto-VEX.
- `src/health.rs`: every closure body that returned `&'static str`
  now returns `String` (warp 0.4's `Reply` impl on `&str` is gated
  to `&'static str` only and the if/else inference would otherwise
  pick a non-'static lifetime). Final filter chain calls `.boxed()`
  to type-erase into `BoxedFilter` — without it the filter type
  carries a higher-ranked `AsRef<str>` bound from the new generic
  `path::path<P: AsRef<str>>` signature, which propagates through
  `tokio::spawn` and fails `Send + 'static`.
- `src/main.rs`: `warp::serve(metrics.boxed())` for the same
  HRTB / Send reason as health.rs.
- `Cargo.lock`: regenerated (warp 0.3.7 → 0.4.2; drops the legacy
  hyper 0.14 / http 0.2 / h2 0.3 / base64 0.21 / encoding_rs /
  displaydoc duplicate-stack now that warp pulls hyper 1).

### Why
PR #52 (Dependabot) bumped warp 0.3 → 0.4; the upstream release is
breaking (server/test feature-gated, hyper 1 underneath, narrower
Reply bounds, generic `path()`). The bump is the unfinished tail of
the auto-VEX Phase 3 work in `5spot-automated-vex-generation.md` —
warp 0.3 sits on hyper 0.14 which is end-of-life upstream and visible
to the supply-chain scanners tracked in
`5spot-code-scanning-remediation.md`.

### Impact
- [ ] Breaking change (no API surface change; internal-only)
- [x] Requires cluster rollout (new binary)
- [ ] Config change only
- [ ] Documentation only

### Verification
- `cargo build` clean.
- `cargo test`: 397 passed, 0 failed.
- `cargo fmt --check`: clean.
- `cargo clippy --all-targets -- -D warnings`: clean.

---

## [2026-04-26 00:30] - Phase 4 of security audit: reclaim-agent host-identity verification

**Author:** Erick Bourgeois

### Changed
- `src/reclaim_agent.rs`: new pure helpers `read_host_machine_id(path)`
  and `compare_machine_ids(node, node_name, expected_host_id)` plus a
  `HostIdentityError` enum with four variants (`ReadFailed`, `Empty`,
  `Mismatch`, `NodeMachineIdMissing`). Both helpers are I/O-light and
  unit-testable without a kube client.
- `src/bin/reclaim_agent.rs`: read `/etc/machine-id` once at startup
  via `read_host_machine_id`; cache as `Option<String>`. Before each
  `annotate_node` PATCH, fetch the target Node and call
  `compare_machine_ids` against the cached host id. On mismatch the
  agent logs `error!` and bails (no PATCH). New `--machine-id-path`
  (env `MACHINE_ID_PATH`, default `/etc/machine-id`) and
  `--skip-host-id-check` (env `SKIP_HOST_ID_CHECK`, default false /
  strict) CLI flags.
- `deploy/node-agent/daemonset.yaml`:
  - Single-file read-only mount of `/etc/machine-id` →
    `/host/etc/machine-id`. `hostPath.type: File` so kubelet fails to
    schedule the pod if the host file is missing (fail-fast over
    silent skip-check).
  - `MACHINE_ID_PATH=/host/etc/machine-id` env var to point the agent
    at the mounted location.
  - Commented `SKIP_HOST_ID_CHECK` env var documenting the bypass for
    operators who deploy in environments without `/etc/machine-id`.
- `src/reclaim_agent_tests.rs`: 9 new tests covering both helpers —
  `read_host_machine_id` happy path / missing file / empty file /
  whitespace-only file; `compare_machine_ids` match / mismatch (asserts
  both ids appear in the error message for forensics) / no status /
  empty machine-id / whitespace-only machine-id.
- `docs/src/concepts/emergency-reclaim.md`: new "Host-identity
  verification" section documenting the contract, the two modes
  (strict default vs `--skip-host-id-check` opt-out), the mount shape
  in the DaemonSet, and what the check does NOT defend against
  (TPM-grade attestation is out of scope).

### Why
Phase 4 of the 2026-04-25 security audit roadmap. Without the
cross-check, an attacker with `update daemonsets` in `5spot-system`
could override the agent's `NODE_NAME` env var (downward API value →
hard-coded victim) and cause the agent to trigger emergency-reclaim on
a Node it has no business touching. The exploit precondition is
cluster-admin-equivalent in most clusters, so this is filed Low
severity / defence-in-depth, but the binding cost is small and removes
the impersonation primitive entirely.

### Impact
- [ ] Breaking change
- [x] Requires cluster rollout (DaemonSet manifest now mounts
      `/etc/machine-id`; existing deployments need `kubectl apply` of
      the new manifest. Hosts that lack `/etc/machine-id` will fail
      pod scheduling — operators must either populate the host file
      or set `SKIP_HOST_ID_CHECK=true` for those nodes.)
- [ ] Config change only
- [ ] Documentation only

---

## [2026-04-25 22:00] - Phase 3 of security audit: route Node→SM via canonical CAPI Machine

**Author:** Erick Bourgeois

### Changed
- `src/reconcilers/helpers.rs`: new
  `node_to_scheduled_machines_via_machine(node, machines)` mapper. Routes
  Node events to owning ScheduledMachines via the canonical CAPI Machine
  ownership chain — `Machine.status.nodeRef.name` (written only by the
  CAPI controller) plus the controller-stamped
  `LABEL_SCHEDULED_MACHINE` label on the Machine. Both legs are
  write-controlled by trusted controllers, not by the tenant.
- `src/reconcilers/helpers.rs`: `node_to_scheduled_machines` (the
  pre-existing mapper that trusted `SM.status.nodeRef.name`) marked
  `#[deprecated(since = "0.1.2")]` with a note pointing at the
  replacement. Kept exported and tested for one release so any external
  caller (none expected) gets a compile-time pointer.
- `src/reconcilers/mod.rs`: re-export the new symbol; gate the legacy
  re-export with `#[allow(deprecated)]`.
- `src/main.rs`: wire a standalone `kube::runtime::reflector` for CAPI
  Machines (filtered by `LABEL_SCHEDULED_MACHINE`). Spawn its watcher in
  the background; pass the reader handle into the Node watch closure.
  Switch the Node mapper from `node_to_scheduled_machines` to
  `node_to_scheduled_machines_via_machine`. The
  `Controller::watches_with` Machine watch keeps its own internal
  reflector for the existing Machine→SM route — kube-rs does not expose
  that internal store, so the secondary watch is intentional. Doubled
  watch traffic is minor (same kind, same label selector).
- `src/reconcilers/helpers_tests.rs`: 9 new tests covering the
  canonical-Machine mapper:
  - happy-path single-Machine match returns owning SM
  - Node with no owning Machine → NOT enqueued (closes spoof window)
  - Machine for a different node → NOT enqueued (canonical wins over
    spoofed SM status)
  - unlabelled / namespace-less / no-nodeRef Machine → ignored
  - Node without metadata.name → ignored
  - whitespace-only label value → ignored (mirrors
    `machine_to_scheduled_machine`'s defensive trim)
  - multiple Machines for same Node → both owning SMs enqueued
- Existing `test_node_to_sms_*` tests for the deprecated mapper kept
  with per-test `#[allow(deprecated)]` annotations so the legacy
  surface stays covered until removal.

### Why
Phase 3 of the 2026-04-25 security audit roadmap. The pre-existing
`node_to_scheduled_machines` mapper filtered SMs by
`status.nodeRef.name`, which is writable by anyone with `patch
scheduledmachines/status` on the resource. A tenant could write a
fast-changing node name (e.g. a busy worker that's frequently
updating) into status to amplify how many SM reconciliations they
caused per Node update — a CPU DoS amplifier. The drain target itself
was never affected (the reconciler always reads canonical
`Machine.status.nodeRef` via `get_node_from_machine`), so this was
verified as Low severity, but the routing change closes the
amplification regardless of how the reconciler evolves.

### Impact
- [ ] Breaking change (deprecated symbol still exported)
- [ ] Requires cluster rollout (no schema or RBAC changes)
- [x] Config change only (in-binary; one extra Machine watch on the
      same selector that already exists)
- [ ] Documentation only

---

## [2026-04-25 18:00] - Phase 2 of security audit: finalizer cleanup force-remove on PDB stall

**Author:** Erick Bourgeois

### Changed
- `src/reconcilers/helpers.rs` (`handle_deletion`): refactor the timeout
  branch from `tokio::time::timeout(...).await.map_err(...)??` (which
  silently propagated the `TimeoutError` and **prevented** the finalizer
  from being removed) into an explicit 3-arm match over a new
  `CleanupOutcome` enum. The default mode now force-removes the
  finalizer on timeout so namespace deletion is unblocked, surfaces a
  `FinalizerCleanupTimedOut` Warning event on the SM, and increments
  `fivespot_finalizer_cleanup_timeouts_total`. A real cleanup error (as
  opposed to a timeout) still propagates and is retried — the finalizer
  is kept in place so a transient API failure does not orphan
  resources.
- `src/reconcilers/helpers.rs`: new pure helpers `run_cleanup_with_timeout`
  (returns `CleanupOutcome::{Completed, Failed(err), TimedOut}`) and
  `build_finalizer_timeout_event`. Both are unit-testable without the
  kube API; the timeout test runs in microseconds via
  `#[tokio::test(start_paused = true)]`.
- `src/reconcilers/scheduled_machine.rs` (`Context`): new
  `force_finalizer_on_timeout: bool` field defaulting to `true`, plus a
  `with_force_finalizer_on_timeout(bool)` builder for `main.rs` to
  override from CLI/env. Strict-cleanup mode (`false`) keeps the
  finalizer and propagates `TimeoutError` so reconciliation retries —
  only safe when an external sweep garbage-collects stuck SMs.
- `src/main.rs`: new `--force-finalizer-on-timeout` flag (env
  `FORCE_FINALIZER_ON_TIMEOUT`, default `true`) wired through `Context`.
- `src/metrics.rs`: new `FINALIZER_CLEANUP_TIMEOUTS_TOTAL` counter +
  `record_finalizer_cleanup_timeout()` helper + label-less
  `fallback_counter` constructor. Documented as an operator-alert
  signal for orphan-resource detection.
- `deploy/deployment/deployment.yaml`: added `FORCE_FINALIZER_ON_TIMEOUT`
  env var with `value: "false"` (strict-cleanup mode) per operational
  preference; the binary default remains `true`. Block-comment in the
  manifest documents the trade-off and references the troubleshooting
  runbook.
- `docs/src/operations/troubleshooting.md`: new "Orphan resources after
  finalizer timeout" section with the runbook (find orphan Machines via
  ownerRef walk, inspect bootstrap/infra refs, cascading delete via
  Machine, prevention guidance).
- `src/reconcilers/helpers_tests.rs` / `src/metrics_tests.rs`: 10 new
  tests covering `run_cleanup_with_timeout` (Completed/Failed/TimedOut),
  `build_finalizer_timeout_event` (severity, reason, action,
  note-content invariants), `Context.force_finalizer_on_timeout`
  defaulting, and the metric-increments-on-call contract.

### Why
Finding F-007 from the 2026-04-25 adversarial security audit (filed in
`~/dev/roadmaps/5spot-security-audit-2026-04-25.md`, Phase 2). A
namespace-tenant with `create pods` + `create poddisruptionbudgets` —
common tenant grants — could plant a `minAvailable: 999` PDB on a
workload they own, schedule a machine on the same node, then delete
the SM. Drain blocks indefinitely on the impossible eviction; with the
old code the finalizer was never removed, the SM was stuck with
`deletionTimestamp`, and namespace deletion stalled. The fix unblocks
namespace deletion in the default mode while preserving
strict-cleanup-or-stall semantics for operators with an external sweep.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [x] Config change only (new env var; behaviour change for clusters
      that hit the timeout path — they'll now succeed at deletion
      where they previously stalled)
- [ ] Documentation only

---

## [2026-04-25 06:30] - `gitleaks-install` now wires the local pre-commit hook

**Author:** Erick Bourgeois

### Changed
- `Makefile` (`gitleaks-install`): after installing the binary, invokes
  `install-git-hooks` so a single `make gitleaks-install` leaves the
  developer with both the tool AND the local secret-scanning hook in
  place. Previously the two targets were independent and the hook had
  to be set up via a second command.
- `Makefile` (`install-git-hooks`): no longer depends on
  `gitleaks-install` (would create a circular dependency now that
  `gitleaks-install` calls it). The hook only invokes gitleaks at
  *commit* time, so install-time independence is correct.
- `Makefile` (`install-git-hooks`): now idempotent and non-destructive.
  A `5spot-managed-gitleaks-hook` sentinel embedded in the hook lets
  re-runs detect "already installed" and leave the file alone. If a
  custom pre-commit hook is found (no sentinel), it is preserved at
  `.git/hooks/pre-commit.bak` rather than overwritten silently.
- `Makefile` (hook content): the generated hook now checks for
  gitleaks on PATH and emits a clear "run make gitleaks-install"
  message if it is missing, instead of failing with a cryptic
  "command not found".

### Why
Onboarding friction: developers ran `make gitleaks-install` expecting
the secret scan to fire on commit, then committed secrets because the
hook had never been wired. Banking-environment requirement (no
secrets in commits) was therefore enforced only by CI — too late.
Wiring hook setup into the install target makes the local guard the
default for every dev, while idempotency + backup keeps re-runs safe.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [x] Config change only (developer tooling)
- [ ] Documentation only

---

## [2026-04-24 15:00] - Security audit remediation: grace period, retry-count poisoning, input bounds

**Author:** Erick Bourgeois

### Changed
- `src/reconcilers/helpers.rs` (`check_grace_period_elapsed`): guard
  against negative `elapsed` durations produced by clock-skew (NTP step,
  VM freeze-thaw, backward wall-clock adjustment). The prior comparison
  `elapsed.num_seconds() >= timeout.as_secs() as i64` silently returned
  `false` when `elapsed` was negative, potentially bypassing the
  graceful-drain window entirely. Negative elapsed is now treated as
  "timeout reached" so the reconciler forces progress rather than
  stalling on a misbehaving clock.
- `src/reconcilers/helpers.rs` (`error_policy`): replace
  `ctx.retry_counts.lock().unwrap_or_else(PoisonError::into_inner)`
  with an explicit `match` that aborts the error-policy pass and
  requeues after `ERROR_REQUEUE_SECS` on poison. The prior pattern
  silently recovered a potentially-inconsistent map and continued
  mutating it, corrupting the exponential-backoff schedule for every
  subsequent error.
- `src/reconcilers/helpers.rs`: new `validate_cluster_name()` and
  `validate_kill_if_commands()` with bounds from `src/constants.rs`.
  Called early in `reconcile_inner` (defence-in-depth against clusters
  that have not enabled the ValidatingAdmissionPolicy) and re-called in
  `add_machine_to_cluster` before touching CAPI.
- `src/constants.rs`: new `MAX_CLUSTER_NAME_LEN = 63` (RFC-1123 DNS
  label — the effective CAPI cluster-name cap), plus
  `MAX_KILL_IF_COMMANDS_COUNT = 100` and `MAX_KILL_IF_COMMAND_LEN = 256`.
- `src/crd.rs`: attach bounded JSON-schema generators
  (`cluster_name_schema`, `kill_if_commands_schema`) so the generated
  CRD enforces the same limits at the kube API level.
- `src/metrics.rs`: replace `unreachable!()` in the metric-registration
  fallback constructors with `panic!()` carrying
  `FALLBACK_METRIC_BUG_MSG` — a pointed diagnostic that identifies the
  offending metric name instead of a generic "entered unreachable code"
  message.
- `deploy/admission/validatingadmissionpolicy.yaml`: add CEL expressions
  1b/1c/1d/1e mirroring the new runtime validators — `clusterName`
  length/charset, `killIfCommands` count/per-entry-length — so invalid
  specs are rejected at admission instead of only at reconciliation
  time.
- `src/reconcilers/helpers_tests.rs`: 20 new tests covering the grace-
  period clock-skew fix, `validate_cluster_name` (happy path, empty,
  over-cap, control chars, non-ASCII), and `validate_kill_if_commands`
  (None/empty/typical/cap-boundary/over-cap/empty-entry).
- `deploy/crds/scheduledmachine.yaml`: regenerated via `cargo run --bin
  crdgen` to pick up the new schema constraints.
- `docs/src/reference/api.md`: regenerated via `make crddoc`.

### Why
Findings from the deep security audit of the codebase (see
conversation transcript 2026-04-24). Three issues were actionable: a
clock-skew bypass of the graceful-drain window (critical; violates the
contract of graceful shutdown under SOX §404 / NIST AC-3), an
unbounded `killIfCommands` that could balloon Prometheus label
cardinality and pin reclaim-agent CPU (high; DoS vector), and
silent recovery from a poisoned retry-count mutex (medium; corrupts
the exponential-backoff schedule). Cluster-name was also unbounded;
the 63-char cap matches the effective CAPI constraint (cluster names
flow into DNS labels downstream). Metric `unreachable!()` is a
cosmetic bug-diagnostic improvement.

### Impact
- [ ] Breaking change
- [x] Requires cluster rollout (new CRD schema constraints; existing
      CRs that violate them will be rejected on next `kubectl apply`)
- [ ] Config change only
- [ ] Documentation only

---

## [2026-04-23 08:00] - Fix vexctl-install Linux path (wrong asset filename)

**Author:** Erick Bourgeois

### Changed
- `Makefile` (`vexctl-install` target): rewrote the Linux branch. The
  original was modelled on the `gitleaks-install` target, which
  assumes a `name_version_os_arch.tar.gz` tarball convention with a
  versioned `name_version_checksums.txt`. vexctl uses neither
  convention: releases ship **raw binaries** named
  `vexctl-<os>-<arch>` (no tarball, no version in the filename), and
  the checksums file is `vexctl_checksums.txt` (version-less). The
  branch now downloads the raw binary, downloads the correct
  checksums file, greps with a two-space-prefix + end-of-line
  anchor (standard `sha256sum` format), verifies, and `install`s
  directly to `/usr/local/bin/vexctl`. No tarball extraction step.
- Help string updated from "GitHub release tarball on Linux" to
  "pinned raw binary + sha256 on Linux" to match reality.

### Why
The first CI run that exercised the `vex-validate` job on a
GitHub-hosted Linux runner failed end-to-end:

```
Downloading vexctl_0.4.1_linux_amd64.tar.gz...
curl: (22) The requested URL returned error: 404
curl: (22) The requested URL returned error: 404
grep: vexctl_checksums.txt: No such file or directory
sha256sum: vexctl_checksum_file.txt: no properly formatted checksum lines found
tar (child): /tmp/vexctl_0.4.1_linux_amd64.tar.gz: Cannot open: No such file or directory
install: cannot stat '/tmp/vexctl': No such file or directory
✓ vexctl installed: /bin/sh: 41: vexctl: not found
make: *** [Makefile:498: vex-validate] Error 127
```

Root cause: I shipped the Makefile target in Phase 1 without
verifying the vexctl release asset naming — it differs from the
gitleaks convention I copied from. Checked the actual
`/releases/tags/v0.4.1` API response and confirmed the correct
asset names.

The macOS branch (`brew install vexctl`) was always correct and is
untouched; local `make vex-validate` on the maintainer's Mac
continued working throughout, which is why the bug didn't surface
until CI ran.

### Verification
- Re-downloaded `vexctl_checksums.txt` from the v0.4.1 release and
  confirmed the new `grep "  $${BINARY}$$" vexctl_checksums.txt`
  pattern matches the `vexctl-linux-amd64` line with its SHA
  (`51753968975448c521b693e91a52f7e30b7d427da668375218fa64392ffb93c5`).
- Local `make vex-validate` on macOS continues to pass — brew
  branch untouched.
- The Linux branch was eyeballed-verified against the actual asset
  URL and checksum-format; will be proven by the next CI run on
  this branch.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [x] Config change only — Makefile fix, no runtime or API change.
- [ ] Documentation only

---

## [2026-04-22 21:45] - Pin CodeQL languages (fallout from Phase 1 Python deletion)

**Author:** Erick Bourgeois

### Added
- `.github/workflows/codeql.yaml`: explicit CodeQL Advanced Setup
  workflow with a matrix over `rust`, `actions`, and
  `javascript-typescript`. `build-mode: none` for all three (Rust
  supports it per the CodeQL compiled-languages docs; the other two
  don't require a build). Runs on PR, push to main, and a weekly
  Sunday 07:17 UTC full re-scan. Uploads SARIF per-language category
  to Code Scanning.
- `.github/codeql/codeql-config.yml`: companion config file with
  `paths-ignore` for `target/**` and `docs/site/**` (the generic
  generated-output exclusions). No Python-adjacent paths — since
  `python` isn't in the workflow's language matrix, the analyzer
  never looks at the Poetry manifests in `docs/`, so explicit
  exclusions there would be dead weight sending a misleading "we
  still configure for Python" signal.

### Why
Phase 1 (`.claude/CHANGELOG.md` 2026-04-22 19:45) deleted the `tools/`
directory including `assemble_openvex.py` and `validate_vex.py`. After
that change, GitHub's **default setup** CodeQL run started failing on
the next push with:

```
CodeQL detected code written in GitHub Actions, Rust and
JavaScript/TypeScript, but not any written in Python. Confirm that
there is some source code for Python in the project.
```

Root cause: default setup auto-detected Python because
`docs/pyproject.toml` (a Poetry manifest for MkDocs tooling) still
exists — but no `.py` source files remain, so the Python database
finalize step errors with exit code 32.

The fix is to take over from default setup with an explicit workflow
that pins the language list, which is what this entry adds. Default
setup will be automatically superseded once any workflow calls
`github/codeql-action/init`, so no Settings-UI change is needed.

The single JS file in scope is `docs/src/javascripts/mermaid-init.js`
(MkDocs Mermaid diagram init); that's why JS/TS stays on the list.
Poetry manifests are explicitly path-ignored — they are metadata, not
source, and would produce false-empty Python databases.

### Verification
- `python3 -c "import yaml; yaml.safe_load(open('.github/workflows/codeql.yaml'))"`
  and `yaml.safe_load(open('.github/codeql/codeql-config.yml'))` both
  parse clean.
- Action pins match the rest of the repo
  (`github/codeql-action@95e58e9a2cdfd71adc6e0353d5c52f41a045d225 # v4.35.2`,
  `actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2`).
- Dependabot will continue updating the codeql-action pin
  (`.github/dependabot.yml:41` already lists `github/codeql-action`
  in the allow-list).

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [x] Config change only — adds a workflow file and a sibling config,
  no runtime change.
- [ ] Documentation only

### Operator follow-up
If the repo's Code Scanning default setup was previously enabled via
the UI, it will automatically defer to this workflow — no toggle
needed. If a partial status appears under *Settings → Code security →
Code scanning*, it's cosmetic: the advanced-setup workflow takes
precedence.

---

## [2026-04-22 21:15] - Drop AUTO_VEX_PRESENCE gate; Security team is the verifier

**Author:** Erick Bourgeois

### Changed
- `.github/workflows/build.yaml`:
  - `build-vex` job: removed the `vars.AUTO_VEX_PRESENCE == '1'`
    conditional on the auto-presence download step. The artifact is
    now downloaded unconditionally, and the merge command includes
    `auto/vex.auto-presence.json` on every build. Added an explicit
    `test -f auto/vex.auto-presence.json || exit 1` guard so a
    missing artifact fails the job (rather than silently producing
    hand-authored-only VEX).
  - Comment headers on both `auto-vex-presence` and `build-vex`
    rewritten to describe the new trust model: CI emits aggressively,
    Security team verifies downstream and counter-signs.
- `.vex/README.md`: removed the "parallel-run gate / review-only"
  language. Auto-presence statements now merge into the signed VEX on
  every build.
- `docs/src/security/vex.md`:
  - CI flow step 3: removed the `AUTO_VEX_PRESENCE=1` gate clause.
  - Automation-split section: the "Automated" bullet no longer
    mentions a feature flag.
  - New top-level "Trust model" section describing the two-signature
    workflow: CI-side Cosign attestation (keyless OIDC via GitHub)
    first, Security-team Cosign attestation (their own OIDC identity)
    second. Each sits in the Sigstore transparency log; downstream
    consumers can require both via twin
    `--certificate-identity-regexp` invocations.
- `~/dev/roadmaps/5spot-automated-vex-generation.md`:
  - Top-of-file status line updated: Phase 1 ✅, Phase 2 ✅ (always-on),
    Phase 3 ⏳ scoped.
  - New "Trust model (load-bearing)" section at the top — replaces
    the implicit assumption in the earlier draft that our side would
    be the verification gate.
  - Phase 2 "Status" line updated to reflect the gate removal.
  - Phase 3 tradeoff section simplified: "two independent call-graph
    analyses must agree" and "multi-release parallel-run" removed;
    the Security team's re-verification is the second check.
  - Rollout section rewritten: no per-phase feature flags; each
    phase activates on merge.

### Why
Policy flip requested by the user. The original Phase 2 rollout plan
had an `AUTO_VEX_PRESENCE` repo variable that gated whether
auto-generated statements flowed into the signed VEX document; the
intent was a parallel-run validation window where maintainers could
diff the auto-set against `.vex/` before authorizing it. The revised
trust model moves that validation downstream to the Security team,
which re-derives each machine-authored claim from the signed evidence
(SBOM, call-graph attestations, SLSA provenance) and counter-signs if
they agree. That makes our side's gate redundant — if we get it
wrong, the VEX ships with only one attestation instead of two, which
is a discoverable condition for any downstream consumer running a
two-signature verify gate.

The trust model extends to the not-yet-implemented Phase 3
(reachability-based auto-VEX): it also ships unconditionally when
built, with call-graph attestations providing the evidence for
Security to verify.

### Verification
- `python3 -c "import yaml; yaml.safe_load(open('.github/workflows/build.yaml'))"`
  → valid YAML.
- `grep "AUTO_VEX_PRESENCE" .github/workflows/build.yaml` → no
  residual references.
- No Rust code touched; `cargo` checks unchanged from the previous
  entry.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [x] Config change only — specifically, removes a config knob that
  existed for ~1 hour and was never documented as user-facing.
- [x] Documentation only

---

## [2026-04-22 20:30] - Presence-based auto-VEX (roadmap Phase 2)

**Author:** Erick Bourgeois

### Added
- `src/auto_vex_presence.rs` (library module, ~180 LOC) + `src/auto_vex_presence_tests.rs`
  (20 unit tests, 100% positive/negative/exception coverage per project rule):
  - `compute_presence_vex` — pure function that diffs Grype findings against
    SBOMs and the already-triaged CVE set, emitting
    `not_affected + component_not_present` statements for CVEs whose purl
    is absent from every SBOM.
  - `build_document` — wraps statements in the OpenVEX envelope.
  - `load_triaged_from_vex_dir` — reads hand-authored `.vex/*.json` and
    collects the set of `vulnerability.name` values (permissive on
    missing dir, strict on malformed JSON).
  - Typed deserializers for Grype JSON and CycloneDX SBOM subsets —
    tolerant of unknown upstream fields.
  - Output sorted by CVE id for deterministic, diffable artifacts.
- `src/bin/auto_vex_presence.rs` — thin clap-driven CLI wrapping the
  library. Reads `--grype-json`, one or more `--sbom` files, a
  `--vex-dir`, a `--product-purl`, `--id`, optional `--author` and
  `--timestamp`, writes an OpenVEX JSON to `--output` or stdout.
- `Cargo.toml`: `[[bin]] auto-vex-presence` entry.
- `Makefile`: `vex-auto-presence` target (defaults `GRYPE_JSON=grype.json`
  and `SBOM_FILES=target/release/*.cdx.json docker-sbom-*.json`; both
  overridable). Added to `.PHONY`.
- `.github/workflows/build.yaml`: two new jobs gated on
  `github.event_name != 'pull_request'`:
  - `grype-triage` — runs Grype against each image variant without VEX
    suppression, uploads per-variant JSON (pinned GRYPE_VERSION 0.87.0
    matching the existing `grype` job).
  - `auto-vex-presence` — builds the new bin, downloads both triage
    scans + Docker SBOMs, runs the bin once per variant, merges the
    per-variant outputs via `vexctl merge`, uploads
    `vex-auto-presence` artifact.

### Changed
- `.github/workflows/build.yaml`:
  - Docker SBOM generation (`Generate Docker SBOM for ${{ matrix.variant.name }}`):
    gate loosened from `if: github.event_name == 'release'` to
    `if: github.event_name != 'pull_request'` so push-to-main also
    produces SBOMs for auto-vex-presence to consume.
  - `build-vex` job: `needs` extended to include `auto-vex-presence`.
    Added a feature-flagged download step (`if: vars.AUTO_VEX_PRESENCE == '1'`)
    and conditional merge inclusion. When the repo variable is
    `AUTO_VEX_PRESENCE=1`, `vex.auto-presence.json` is merged alongside
    `.vex/*.json`; otherwise hand-authored only. Artifact is produced
    regardless — only the merge is gated.
- `src/lib.rs`: re-export `pub mod auto_vex_presence`.
- `docs/src/security/vex.md`: CI flow section renumbered to 6 steps;
  added step 2 for auto-presence. Rewrote "Why we did not auto-generate
  statements" section as "What we automate, and what stays human" — the
  policy now distinguishes `component_not_present` (mechanical,
  automatable) from every other triage decision (human-authored).
- `.vex/README.md`: added "Automated statements (roadmap Phase 2)"
  section explaining the review-artifact vs flag-gated-merge split.

### Why
Phase 2 of the automated VEX generation roadmap
(`~/dev/roadmaps/5spot-automated-vex-generation.md`). Most of the
hand-authored statements under `.vex/` exist because Grype flags
CVEs on libc / glibc / zlib code paths that the Rust binary never
reaches — i.e. the vulnerable component is present in the image
filesystem but never invoked. A subset of those (plus most
base-image CVEs going forward) can be suppressed mechanically by
observing that the flagged package's purl is not in the SBOM at
all: the SBOM is authoritative for what's in the product, so
"component_not_present" is the one OpenVEX justification with a
purely verifiable definition. Automating that specific case shrinks
the `.vex/` maintenance queue without compromising the trust model
for any other justification.

The implementation is strictly SBOM-driven — no reachability analysis,
no source-code inspection. Reachability is Phase 3.

### Rollout / feature-flag
Per the roadmap rollout plan, the auto-presence artifact is
produced and uploaded on every push + release, but it is **only
merged into the signed VEX document when `vars.AUTO_VEX_PRESENCE=1`
is set** at the repository or organization level. This gives
maintainers at least one release of parallel-run validation: the
artifact is visible for review, but consumers see only the
hand-authored VEX until the flag is flipped. This matches the
roadmap's explicit guidance ("Run in parallel with hand-authored VEX
for one release; diff the auto-set against .vex/ and confirm no
surprises before flipping it on").

### Verification
- `cargo fmt --all -- --check`: clean.
- `cargo clippy --all-targets --all-features -- -D warnings`: clean.
- `cargo test --lib`: 349 tests pass (329 pre-existing + 20 new for
  `auto_vex_presence`).
- Smoke test: ran the bin against a synthetic Grype JSON (3 matches),
  synthetic SBOM (covering 1 of the 3 purls), and the current `.vex/`
  directory (covering 1 of the 3 CVEs). The bin emitted exactly the
  one statement for the remaining CVE whose purl was absent, and
  `vexctl merge` of the auto-presence output + `.vex/*.json`
  produced a 16-statement document with the auto statement correctly
  included.

### Impact
- [ ] Breaking change
- [x] Requires cluster rollout (no — CI only)
- [ ] Config change only
- [x] Documentation only

*(Rollout note: when maintainers want to flip `AUTO_VEX_PRESENCE=1`,
that's a repo-variable change in GitHub settings, no code deploy.)*

---

## [2026-04-22 19:45] - Replace Python VEX tooling with vexctl (roadmap Phase 1)

**Author:** Erick Bourgeois

### Removed
- `tools/` — entire directory deleted. Removed `assemble_openvex.py`,
  `validate_vex.py`, `validate-vex.sh`, `tests/assemble-openvex-tests.sh`,
  `tests/validate-vex-tests.sh`, and all 18 fixture directories under
  `tests/fixtures/`. ~400 LOC of bespoke Python + shell tests replaced
  by upstream `vexctl` invocations.

### Changed
- `.vex/*.toml` → `.vex/*.json` (15 files): migrated every statement
  from the bespoke TOML dialect to the native OpenVEX v0.2.0 JSON
  shape. Each file is now a single-statement OpenVEX document that
  `vexctl merge` can consume directly; no translation layer.
- `Makefile`: added `VEXCTL_VERSION ?= 0.4.1`, `vexctl-install` target
  (brew on macOS, pinned GitHub release tarball with checksum
  verification on Linux; errors on other OSes), `vex-validate`
  (parses every `.vex/*.json` via `vexctl merge` — successful parse
  is the validation), and `vex-assemble` (prints the merged document
  for local preview). `.PHONY` list updated.
- `.github/workflows/build.yaml`:
  - `validate-vex` job: dropped Python setup + custom validator calls;
    now runs `make vexctl-install` then `make vex-validate`.
  - `build-vex` job: dropped Python setup + `assemble_openvex.py`;
    now runs `vexctl merge --id <per-event-id> --author <actor>
    .vex/*.json` directly. Per-event `@id` logic (release / push /
    PR) preserved verbatim. Cosign attestation, artifact upload,
    and `attest-build-provenance` steps unchanged.
  - PR `paths:` filter: removed `tools/**` (directory no longer
    exists); `.vex/**` retained.
- `.vex/README.md`: rewritten to document native OpenVEX JSON
  authoring and `make vex-validate` / `make vex-assemble` instead of
  the old TOML schema and shell validator.
- `docs/src/security/vex.md`: CI flow section updated — step 1
  ("validate") and step 2 ("assemble") now describe `vexctl merge`
  behaviour; dropped the obsolete "cross-check with `vexctl validate`"
  step (vexctl has no `validate` subcommand). Maintainer-authoring
  section rewritten with the JSON shape in place of the TOML shape.
- `.claude/SKILL.md`: release checklist item for VEX updated from the
  old three-script check to `make vex-validate`.

### Why
Phase 1 of the automated VEX generation roadmap
(`~/dev/roadmaps/5spot-automated-vex-generation.md`). The custom
Python toolchain duplicated functionality that `vexctl` — the official
OpenVEX CLI — provides upstream. Maintaining a bespoke TOML dialect,
its validator, its assembler, and two fixture-based test suites
added ~400 LOC of surface area with no offsetting benefit once we
accept OpenVEX JSON as the authoring format. Replacing the whole
stack with `vexctl merge` (which doubles as the validator via
successful parse) removes the Python runtime from CI entirely,
eliminates a class of drift between our validator and the upstream
spec, and prepares the pipeline for Phases 2–3 (automated VEX
generation from SBOM and call-graph reachability).

The TOML-over-JSON ergonomics hit is real but small: 15 files in
`.vex/`, each ~15 lines of JSON, with no inline comments. In
exchange, future maintainers work with the same format upstream
tooling speaks and any OpenVEX-aware tool can consume the source
directory directly.

### Verification
- `make vex-validate` parses all 15 `.vex/*.json` successfully.
- `vexctl merge --id <x> --author <y> .vex/*.json` produces a
  15-statement document structurally identical to what the Python
  assembler would have produced (confirmed via key-by-key diff of
  `@context`, `@id`, `author`, `version`, and every statement's
  `products`, `status`, `justification`, `impact_statement`,
  `action_statement`, `timestamp`).

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [x] Documentation only

*(Not really "documentation only" — code deletion is substantive —
but no runtime, CRD, or API-surface change. Closest pre-existing
box.)*

---

## [2026-04-21 15:00] - Model emergency-reclaim in CALM — ADD retrofit

**Author:** Erick Bourgeois

### Added
- `docs/architecture/calm/architecture.json`: 4 new nodes — `service-workload-kubernetes-api` (workload cluster API server as a distinct trust boundary from the management cluster API), `service-reclaim-agent` (node-side DaemonSet), `data-asset-reclaim-request-annotation` (agent→controller signalling surface on the Node), `data-asset-reclaim-agent-configmap` (per-node projection in `5spot-system`).
- `docs/architecture/calm/architecture.json`: 4 new relationships — `rel-controller-workload-kube-api` (controller writes into the workload cluster: ConfigMap projection, Node label, Node annotation clear; audit-attribution control NIST AU-2/AU-10 via distinct field manager `5spot-controller-reclaim-agent`), `rel-reclaim-agent-workload-kube-api` (agent watches its ConfigMap + annotates its own Node; node-scoped-credentials control NIST AC-6), `rel-reclaim-configmap-stored-in-workload-api` (composed-of), `rel-reclaim-annotation-on-node` (composed-of).
- `docs/architecture/calm/architecture.json`: new `flow-emergency-reclaim` with 7 transitions mapping the load-bearing ordering contract (drain → Machine delete → `spec.schedule.enabled=false` → annotation clear → phase Disabled). Flow-level `ordering-idempotence` control (NIST SI-7) documents why step 5 must precede step 6 and how the `PHASE_EMERGENCY_REMOVE` match arm closes the crash window.
- Per-node controls on `service-reclaim-agent`: `least-privilege` (NIST AC-3/AC-6, evidence `deploy/node-agent/rbac.yaml`) and `container-hardening` (NIST SP 800-190 CM-6/SI-7, evidence `deploy/node-agent/daemonset.yaml` + `.trivyignore` architectural-necessity suppressions).

### Changed
- `docs/architecture/calm/architecture.json`: extended `rel-workload-cluster-contains-node` to include `service-workload-kubernetes-api` and `service-reclaim-agent` alongside the existing `network-physical-node`.
- `docs/src/architecture/{flows.md,system.md}`: regenerated by `make calm-diagrams` from the updated model (gitignored — CI regenerates).

### Why
The user just committed the ADD (Architecture Driven Development) rule: any architectural change must land in CALM before code. An audit against the two most-recent features found the **emergency-reclaim work is architecturally material** (new node-side component, new trust boundary, new cross-cluster signalling protocol, new `EmergencyRemove` phase, new RBAC scope in `5spot-system`) but was not reflected in CALM. This commit closes that gap — it is the shakedown test for the ADD rule, done now so the workflow is proven before the rule applies to future features. The separately-audited node-taints feature is not architectural under ADD (CRD-field + existing Node-watch pattern only) and is intentionally **not** retrofitted here.

## [2026-04-21 17:00] - Integration test for node-taint reconcile on a real cluster (Phase 8)

**Author:** Erick Bourgeois

### Changed
- `tests/integration_node_taints.rs`: New integration test file. Two `#[tokio::test] #[ignore]` cases exercise `five_spot::reconcilers::reconcile_node_taints` against a real kube cluster (kind). Case 1 (`apply_update_shrink_cycle`) applies two taints, shrinks to one, asserts only the removed one is gone and the kept one survives. Case 2 (`admin_conflict_is_reported_and_admin_taint_preserved`) seeds an admin-owned taint with the same `(key, effect)` as a desired one, then asserts the reconcile returns `Conflict` and leaves the admin-owned taint untouched. Skips gracefully (not fails) when no cluster is reachable, so `cargo test` stays hermetic. All test-applied taints use the `integration-test.5spot.local/` key prefix so the cleanup helper can scrub on both entry and exit without ever touching admin state. Deliberately bypasses CAPI/k0smotron entirely — we pick an already-Ready Node from the cluster rather than provisioning one, because the only piece this test needs to exercise is the Node-side patch/annotate path.
- `src/reconcilers/mod.rs`: Re-exported `reconcile_node_taints`, `ReconcileNodeTaintsInput`, `NodeTaintReconcileOutcome` from the `helpers` submodule at the `reconcilers` level so integration tests (which can only see the public API) can call them.

### Why
Phase 8 of the user-defined Node taints roadmap — real-cluster verification. The unit tests in `helpers_tests.rs` already cover the pure logic in `diff_node_taints` and drive `reconcile_node_taints` through a mocked kube service; the integration test proves the same function works end-to-end against a real API server, including server-side apply semantics, annotation round-tripping, and admin-conflict detection against live `spec.taints`. Per the user's decision earlier in this session, we deliberately mock the `status.nodeRef` side (by picking an existing Node) rather than stand up CAPI + k0smotron + RemoteMachine + SSH-reachable hardware — that's out of scope for this roadmap and the Node-patch code path is the only new behavior.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [ ] Documentation only

New tests are gated behind `#[ignore]`; running `cargo test` without a cluster still shows 328/328 lib tests plus `0 passed; 2 ignored` on the integration target. Developers who want to exercise Phase 8 run `cargo test --test integration_node_taints -- --ignored --test-threads=1` against a kind cluster (single-threaded because both cases mutate the same Node). `cargo fmt` ✓ · `cargo clippy --all-targets --all-features -- -D warnings` ✓ · `cargo test` 328/328 ✓.

---

## [2026-04-21 16:00] - Docs + examples + crddoc regen for nodeTaints (Phase 7)

**Author:** Erick Bourgeois

### Changed
- `docs/src/concepts/scheduled-machine.md`: Added a `nodeTaints` row to the spec-fields table, a new "Node Taints" subsection (shape, ownership model, `NodeTainted` condition vocabulary, what-it-does-not-manage), and an `appliedNodeTaints` row in the status-fields table. Links from both directions so operators reading the status field land on the ownership explainer.
- `docs/src/operations/troubleshooting.md`: Added a new "Node Taints" section with "Taints not appearing on Node" walkthrough, one subsection per `NodeTainted` condition reason (`NoNodeYet`, `NodeNotReady`, `TaintOwnershipConflict`, `PatchFailed`) with the exact `kubectl` commands operators need and the fix for each.
- `README.md`: Added a one-line bullet in the features list referencing `spec.nodeTaints` and the ownership/drift-reconcile story.
- `src/bin/crddoc.rs`: Added `nodeTaints` to the example YAML in the generated API doc, a `#### nodeTaints` section with the full NodeTaint field schema + reserved-prefix + uniqueness rules, and a `#### appliedNodeTaints` section in Status Fields explaining the ownership-record-of-truth role.
- `docs/src/reference/api.md`: Regenerated via `cargo run --bin crddoc` (LAST step per CLAUDE.md order-of-operations) — now 231 lines, with spec and status taint fields documented.

### Why
Phase 7 of the user-defined Node taints roadmap — docs last. Operators need to find the condition vocabulary *before* they hit a production issue, and the troubleshooting entry is keyed off the condition reason so a `kubectl get scheduledmachine ... -o jsonpath='{.status.conditions[?(@.type=="NodeTainted")]}'` walk leads directly to the fix. The crddoc regen runs last per the CLAUDE.md mandate because it incorporates everything upstream.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [x] Documentation only

`cargo fmt` ✓ · `cargo clippy --all-targets --all-features -- -D warnings` ✓ · `cargo test --lib` 328/328 ✓ · `cargo run --bin crddoc` ✓ (regenerated `docs/src/reference/api.md`).

---

## [2026-04-21 15:30] - ValidatingAdmissionPolicy CEL rules for nodeTaints (Phase 6)

**Author:** Erick Bourgeois

### Changed
- `deploy/admission/validatingadmissionpolicy.yaml`: Added six validation rules (14–19) covering `spec.nodeTaints`: RFC-1123 qualified-name regex on `key`, 253-char cap on `key`, 63-char cap on `value`, reserved `5spot.finos.org/` prefix rejection, reserved kubelet prefixes (`kubernetes.io/`, `node.kubernetes.io/`, `node-role.kubernetes.io/`) rejection, and uniqueness of `(key, effect)` pairs via O(n²) `filter().size() == 1` scan. Each rule guards `!has(object.spec.nodeTaints)` so empty/absent fields pass trivially.
- `examples/scheduledmachine-bad-taint.yaml`: New worked-negative example — four taints, each one tripping a different rule (reserved 5spot prefix, reserved kubelet prefix, duplicate key+effect). Lets operators verify the VAP is actually installed by running `kubectl apply -f` and watching the rejection fire.

### Why
Phase 6 of the user-defined Node taints roadmap — defense in depth. The Rust-side validator in `src/crd.rs` is the authoritative guard (it runs in every cluster regardless of whether the VAP is installed); the VAP is the faster-feedback layer that rejects bad input at API-server admission before the reconciler queues a hopeless work item. Both layers intentionally cover the same rules so a cluster without the VAP still has correct semantics, and a cluster with the VAP gets the sharper error message earlier.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [x] Config change only (VAP is re-applied via `kubectl apply -f deploy/admission/validatingadmissionpolicy.yaml`; existing CRs are untouched because the new rules only fire on CREATE/UPDATE)
- [ ] Documentation only

No Rust changes. YAML parse verified via `python3 -c "import yaml; yaml.safe_load_all(...)"` — 19 validations, 1 doc. End-to-end test (kubectl reject) requires a cluster and is covered by Phase 8.

---

## [2026-04-21 15:00] - Termination guards on node-taint reconcile (Phase 5)

**Author:** Erick Bourgeois

### Changed
- `src/reconcilers/scheduled_machine.rs`: Added two defensive guards at the top of `reconcile_node_taints_best_effort` — one short-circuits on `metadata.deletion_timestamp.is_some()` (the Node is about to be drained and deleted; applying taints mid-delete leaves a confusing post-mortem state), the other on `spec.kill_switch` (kill-switch routes to `handle_kill_switch`, and taints are not a supported shortcut for eviction — drain is the sanctioned path). Both guards log at debug and return without touching the apiserver.
- `src/reconcilers/scheduled_machine_tests.rs`: Added 5 tests using `tower_test::mock::Handle` with a `next_request()` vs 50ms-sleep race — the watcher task panics if any HTTP request arrives. Covers deletion_timestamp, kill_switch, absent nodeRef, empty nodeRef name, and empty desired+applied. The "no HTTP call" watcher pattern mirrors `test_patch_machine_refs_status_both_none_is_noop_no_http_call`.

### Why
Phase 5 of the user-defined Node taints roadmap. The natural reconcile flow already short-circuits on deletion (deletion_timestamp check at the top of `reconcile` routes to `handle_deletion`) and on kill_switch (routes to `handle_kill_switch` before the Active phase handler). These guards are defense-in-depth: they pin the contract at the function boundary so a future caller that invokes `reconcile_node_taints_best_effort` from a new phase handler cannot accidentally apply taints mid-termination. The tests prove the guards block the apiserver call, not just the return value.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [x] Config change only (no runtime behaviour change for the Active path; termination paths now have explicit guards instead of relying on dispatch-level short-circuits)
- [ ] Documentation only

`cargo fmt` ✓ · `cargo clippy --all-targets --all-features -- -D warnings` ✓ · `cargo test --lib` 328/328 ✓.

---

## [2026-04-21 14:30] - Wire node-taint reconcile into `Active` phase (Phase 4)

**Author:** Erick Bourgeois

### Changed
- `src/reconcilers/scheduled_machine.rs`: Added `reconcile_node_taints_best_effort` and call it from `handle_active_phase` right after `provision_reclaim_agent_best_effort` (so both side-effects key off the same resolved `nodeRef`). Persists the new applied list via `patch_applied_node_taints_status` only when it differs from `status.applied_node_taints`. `NoNodeYet` / `NodeNotReady` log at debug; `Conflict` logs at warn; all failures are logged-and-returned so taint reconcile cannot block machine scheduling. Early-returns on empty `nodeRef`, empty desired ∧ empty previously-applied, and missing namespace.
- `src/reconcilers/helpers.rs`: Added `ReconcileNodeTaintsInput<'a>` struct, `NodeTaintReconcileOutcome { NoNodeYet | NodeNotReady | Applied { applied } | Conflict { conflicts } }`, `reconcile_node_taints` (GET node → check Ready → diff → apply) and `patch_applied_node_taints_status` (merge-patch wrapper mirroring `patch_machine_refs_status`).
- `src/reconcilers/helpers_tests.rs`: Added 5 tests using `tower_test::mock::pair` — 404 → `NoNodeYet`, Ready=False → `NodeNotReady`, Ready + no-op → `Applied` with zero PATCHes, Ready + add → two GETs + one PATCH + `Applied`, admin collision → `Conflict`.

### Why
Phase 4 of the user-defined Node taints roadmap. With Phase 3's pure `diff_node_taints` + `apply_node_taints` in place, this phase plumbs them into the actual reconcile loop behind the `Active` phase's existing `nodeRef` resolution. Best-effort semantics match `provision_reclaim_agent_best_effort`: taint drift is a workload-scheduling concern, not an availability concern, and must never wedge the reconcile.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [x] Config change only (new controller behaviour triggers only when a CR declares `spec.nodeTaints`; CRs without the field see no change)
- [ ] Documentation only

`cargo fmt` ✓ · `cargo clippy --all-targets --all-features -- -D warnings` ✓ · `cargo test --lib` 323/323 ✓.

---

## [2026-04-21 14:00] - Node-taint diff helper + SSA apply IO (Phase 3)

**Author:** Erick Bourgeois

### Changed
- `src/reconcilers/helpers.rs`: Added `NodeTaintPlan { to_add, to_update, to_remove, unchanged, conflicts }` with `is_noop()`, pure `diff_node_taints(current, desired, previously_applied)` that identifies taints by `(key, effect)`, treats `value` as mutable, and routes admin-owned collisions into `conflicts` instead of overwriting, and `apply_node_taints(client, node_name, plan)` which GETs the Node, rebuilds `spec.taints` (controller-owned minus `to_remove`, plus admin-owned, plus `to_add`/`to_update`), and SSA-patches under field manager `5spot-controller-node-taints` with the `5spot.finos.org/applied-taints` ownership annotation.
- `src/reconcilers/helpers_tests.rs`: Added 14 tests — 11 pure-diff cases (empty/no-op/add/update-value/remove-previously-applied/keep-admin/admin-collision-on-add/admin-collision-on-remove-value-drift/duplicate-desired-key-across-effects/full-cycle/remove-many) plus 3 IO cases via `tower_test::mock::pair` covering no-op (no PATCH), add (GET + PATCH with correct field manager + annotation payload), and 404-on-GET (error bubbles, no PATCH).

### Why
Phase 3 of the user-defined Node taints roadmap. The diff logic is isolated from IO so that the ownership model — identity on `(key, effect)`, value mutable, no cross-tenant overwrites — is verified exhaustively in pure tests before it meets a real kube client. The apply side uses SSA + annotation so a later `kubectl get node -o json` can trace every controller-owned taint back to the CR.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [x] Config change only (new helpers are dead code until Phase 4 wires them into `handle_active_phase`; no runtime behaviour change yet)
- [ ] Documentation only

`cargo fmt` ✓ · `cargo test --lib` 318/318 ✓. (Clippy deferred to Phase 4 — the new `pub fn`s are unused until wired, which trips `-D warnings` `dead_code`; Phase 4 wiring resolves it.)

---

## [2026-04-19 22:30] - Node-taint status surface + `NodeTainted` condition (Phase 2)

**Author:** Erick Bourgeois

### Changed
- `src/crd.rs`: Added `applied_node_taints: Vec<NodeTaint>` to `ScheduledMachineStatus` (defaults to empty, omitted from JSON when empty). This is the controller's record of truth for ownership — only entries it listed here are eligible for removal; admin-added taints colliding on `(key, effect)` surface as a condition instead of being overwritten.
- `src/constants.rs`: Added `CONDITION_TYPE_NODE_TAINTED = "NodeTainted"` and five condition reasons covering every state machine transition for the `NodeTainted` condition: `REASON_NODE_TAINTS_APPLIED` ("Applied"), `REASON_NODE_NOT_READY` ("NodeNotReady"), `REASON_NODE_TAINT_PATCH_FAILED` ("PatchFailed"), `REASON_NO_NODE_YET` ("NoNodeYet"), and `REASON_TAINT_OWNERSHIP_CONFLICT` ("TaintOwnershipConflict").
- `src/crd_tests.rs`: Added five new tests — empty-default behaviour, serialisation-omit-when-empty, round-trip with two taints, deserialise-from-absent, and a hard-coded enforcement of all six condition constant string values so a future rename cannot silently ship.
- `deploy/crds/scheduledmachine.yaml`: Regenerated via `cargo run --bin crdgen` — now publishes the `appliedNodeTaints` status array.

### Why
Phase 2 of the user-defined Node taints roadmap. Separating the status type + condition vocabulary from the reconciler IO (Phase 3) means the diff helper and the apply IO can be built against a stable status contract. The condition reasons are constants (not strings inline in the reconciler) so that the dashboard / alerting layer has a fixed vocabulary to match on.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [x] Config change only
- [x] Documentation-only code change (CALM model + rendered mermaid only; no Rust, no deploy manifests touched)

### Verification
- `make calm-validate` → 0 errors, 0 warnings, 0 info/hints.
- `make calm-diagrams` → regenerates `docs/src/architecture/flows.md` (now 3 flows: activation, deactivation, emergency-reclaim) and `docs/src/architecture/system.md` (14 nodes including the new 4, workload-cluster subgraph now contains API server + reclaim agent).
- `make docs` → `mkdocs build` passes; every hand-authored mermaid component in `docs/src/concepts/emergency-reclaim.md` has a corresponding CALM node.
- [x] Config change only (CRD re-apply picks up the `appliedNodeTaints` status schema; existing CRs are unaffected because the field defaults to empty)
- [ ] Documentation only

`cargo fmt` ✓ · `cargo clippy --all-targets --all-features -- -D warnings` ✓ · `cargo test --lib` 304/304 ✓.

---

## [2026-04-19 22:00] - Add `spec.nodeTaints` CRD schema + validator (Phase 1)

**Author:** Erick Bourgeois

### Changed
- `src/crd.rs`: Added `NodeTaint { key, value?, effect }` struct and `TaintEffect { NoSchedule | PreferNoSchedule | NoExecute }` enum. Added `node_taints: Vec<NodeTaint>` to `ScheduledMachineSpec` (defaults to empty, omitted from serialisation when empty). Added `validate_node_taints()` plus private `validate_taint_key` / `validate_taint_value` / `is_qualified_name` / `is_dns_subdomain` helpers, with reserved-prefix checks for `5spot.finos.org/`, `kubernetes.io/`, `node.kubernetes.io/`, and `node-role.kubernetes.io/`.
- `src/crd_tests.rs`: Added 24 new tests covering serde round-trip (with/without `value`), `TaintEffect` variants, invalid-variant rejection, hash/eq identity, empty-list default, spec omission when empty, and every rejection path of `validate_node_taints` (empty key, leading/trailing hyphen, invalid char, 64-char key, 64-char value, duplicate `(key, effect)`, and each reserved prefix).
- `src/reconcilers/helpers_tests.rs`, `src/reconcilers/scheduled_machine_tests.rs`: Added `node_taints: vec![]` to existing `ScheduledMachineSpec` constructors so they compile against the new field.
- `deploy/crds/scheduledmachine.yaml`: Regenerated via `cargo run --bin crdgen`; now includes the `nodeTaints` array schema with `key`/`value`/`effect` properties and the enum restriction on `effect`.
- `docs/src/reference/api.md`: Regenerated via `cargo run --bin crddoc` (picked up pre-existing `killIfCommands` doc drift in the static generator; `nodeTaints` prose is deferred to Phase 7 per the roadmap's docs-last policy).
- `examples/scheduledmachine-tainted.yaml`: New worked example showing two taints (`workload=batch:NoSchedule`, `dedicated=ml:NoExecute`).

### Why
Phase 1 of the user-defined Node taints roadmap (`~/dev/roadmaps/5spot-user-defined-node-taints.md`). This lands the schema + CR-level validator in isolation so the reconciler work in Phases 2–4 has a stable type to build against. Validation runs at the reconciler boundary for clusters without a ValidatingAdmissionPolicy; the VAP (Phase 6) is the defense-in-depth layer.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [x] Config change only (CRD re-apply to pick up new `nodeTaints` schema — absent = empty, so existing CRs are unaffected)
- [ ] Documentation only

`cargo fmt` ✓ · `cargo clippy --all-targets --all-features -- -D warnings` ✓ · `cargo test --lib` 299/299 ✓.

---

## [2026-04-21 13:00] - Defer Phase-2-rung-2 netlink proc connector to GitHub issue

**Author:** Erick Bourgeois

### Changed
- `/Users/erick/dev/roadmaps/5spot-emergency-reclaim-by-process-match.md`: Updated the Phase-2 rung-2 (netlink proc connector) status row from "⏳ Not started" to "⏳ Deferred — tracked as [finos/5-spot#40](https://github.com/finos/5-spot/issues/40)". Row body rewritten to name the concrete tradeoff (detection latency ~1s → <10ms; lower idle CPU; deterministic worst case) and note the counter-tradeoff (rung 1 is cheaper under heavy-exec workloads) so future readers don't re-re-evaluate from scratch.
- Opened [finos/5-spot#40](https://github.com/finos/5-spot/issues/40) (label: `enhancement`) carrying the full scope, dependency-choice tradeoff (`nix` vs `neli` vs `netlink-proto`), deployment delta (`CAP_NET_ADMIN` add + suppression-rationale updates), out-of-scope markers (eBPF, cross-node), and acceptance criteria.

### Why
Rung 2 is optimization, not correctness — rung 1 already meets the <5s SLA and integration/stopwatch testing will be the actual critical-path work. Keeping it in the roadmap as "⏳ Not started" was misleading because the decision is already made (defer), not open. Moving the live tracking to a GitHub issue gets it in front of contributors who may want to pick it up, and keeps the roadmap narrative truthful about what's "active work" vs "filed follow-up".

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [x] Documentation only

No code or manifest changes. `cargo-quality` not re-run (no Rust touched).

---

## [2026-04-21 12:30] - Phase-2.5 async-orchestrator mock-API tests + roadmap alignment

**Author:** Erick Bourgeois

### Changed
- `src/reconcilers/helpers_tests.rs`: 4 new async-orchestrator tests for `reconcile_reclaim_agent_provision` using the existing `tower_test::mock` harness. Non-empty commands → Node label merge-patch then ConfigMap server-side apply (asserting `fieldManager=5spot-controller-reclaim-agent` + `force=true` query params, the `enabled` label string in the patch body, and the `data.reclaim.toml` key present in the apply body). Empty commands → Node label patch (JSON-null value) then ConfigMap DELETE. 404-on-delete → benign `Ok(())` so a re-run after partial tear-down completes. Label-PATCH 500 → `ReconcilerError::KubeError` propagation with no second request issued. Also fixed one stale test comment in `test_build_reclaim_agent_configmap_data_key_is_reclaim_toml` that still referenced the old `/etc/5spot/reclaim.toml` mount path — rewrote to describe the watch-based contract.
- `/Users/erick/dev/roadmaps/5spot-emergency-reclaim-by-process-match.md`: Flipped Phase-2.5 status row from 🟡 Partial to ✅ Shipped (agent-side watch consumption + new mock-API tests close the loop; the DaemonSet ConfigMap mount is gone entirely, so the "residual" that was the blocker no longer exists). Flipped Phase-4 status row from 🟡 Partial to ✅ Shipped for unit scope. Updated test count to **275 green** (was 271). Promoted the three runtime-dependent Phase-4 items (kind integration, manual stopwatch, re-enable loop protection) to explicit `TODO-*` follow-ups with their own names so they stop reading as "open Phase 4 unit gaps". Header status line and cumulative-commits reference updated to include the 2026-04-21 increments.

### Why
Two intents bundled: (1) close the last tractable Phase-4 unit-test gap that was pure-Rust and macOS-viable — the async orchestrator paths of `reconcile_reclaim_agent_provision` had pure-helper tests but no request-sequence pinning, which meant a refactor could reverse the label-PATCH → ConfigMap-apply order (or drop force=true, or rename the field manager) without any red test. (2) Align the roadmap narrative with what's actually shipped: with the agent now watching the per-node ConfigMap via kube API and the DaemonSet mount removed, the Phase-2.5 "residual blocker" narrative is genuinely resolved — leaving it as 🟡 Partial would mislead the next reader.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [ ] Documentation only

**Test count:** 275 green (was 271). `cargo clippy --all-targets --all-features -- -D warnings` clean.

**Still open (runtime-dependent, explicit follow-ups):** Phase 2 rung-2 netlink proc connector (Linux-only, multi-session), kind-cluster integration test, real-hardware stopwatch verification. These are not Phase 4 unit-test gaps — they need infrastructure that isn't appropriate to stub.

---

## [2026-04-21 12:00] - Suppress Trivy KSV-0023 / KSV-0121 / KSV-0012 / KSV-0020 / KSV-0021 / KSV-0105 on reclaim-agent DaemonSet

**Author:** Erick Bourgeois

### Changed
- `.trivyignore`: New `AVD-KSV-0023` entry. The DaemonSet mounts host `/proc` read-only at `/host/proc` so the agent can read `/proc/<pid>/{comm,cmdline}` for every host process — this is the detection contract. There is no Kubernetes-native substitute (Downward API, projected volumes, and CSI drivers do not expose kernel `/proc`). Scope is already minimal: single path (`/proc`), `readOnly: true`, `type: Directory`, and the agent never writes back. Durable: Phase 2 rung-2 (netlink proc connector) still needs host `/proc` for the initial match fan-out.
- `.trivyignore`: New `AVD-KSV-0121` entry. HIGH-severity variant of KSV-0023 that names the disallowed host paths explicitly (`/proc` in our case). Same architectural justification — the detection contract is built around reading host `/proc`, which is exactly what the rule flags.
- `.trivyignore`: New `AVD-KSV-0012` entry. Container-level variant of the existing KSV-0118 root-user finding. Running as UID 0 is required so the agent can read `/proc/<pid>/{comm,cmdline}` under hardened `hidepid=2` /proc mounts. A non-root build would need `CAP_SYS_PTRACE` added back, which is strictly broader than running as root with every other capability dropped, `readOnlyRootFilesystem: true`, `allowPrivilegeEscalation: false`, and seccomp `RuntimeDefault`. Scope is bounded by the opt-in `5spot.finos.org/reclaim-agent: enabled` nodeSelector.
- `.trivyignore`: New `AVD-KSV-0021` entry. Follows directly from KSV-0012 — UID 0 implies GID 0, so `runAsGroup > 10000` cannot be satisfied without abandoning the root requirement already justified. The rule's purpose (avoid a shared high-GID supplementary group leaking filesystem access across workloads) is not a risk here: the container mounts only host `/proc` read-only, has `readOnlyRootFilesystem: true`, and drops all capabilities.
- `.trivyignore`: New `AVD-KSV-0105` entry. Identical root cause to KSV-0012 / KSV-0118 — different rule wording, same finding (UID 0 is required for `hidepid=2` /proc reads; the non-root alternative needs `CAP_SYS_PTRACE`, which is strictly broader than root + all-caps-dropped + read-only rootfs + no-privilege-escalation + seccomp `RuntimeDefault`).
- `.trivyignore`: New `AVD-KSV-0020` entry. "High-UID" strictness variant of KSV-0105 (rule wants UID > 10000, not just > 0). Moot at UID 0: the collision with `root` is deliberate, not accidental. Same architectural justification as the rest of this block.

### Why
Six Trivy findings all flow from the same architectural root cause: the reclaim-agent must read host process state, and the only mechanism Kubernetes offers is a hostPath `/proc` mount + UID 0. Scanner can't express "root is required, everything else is hardened" as a single rule, so each sub-finding needs its own suppression with the shared rationale. Same pattern as the pre-existing KSV-0010 (`hostPID: true`) and KSV-0118 (pod-level default context) suppressions.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [x] Documentation only (security-policy document — `.trivyignore` is input to the CI scanner, not a runtime artifact)

---

## [2026-04-21 11:30] - Reclaim-agent: watch per-node ConfigMap via kube API (Phase 2.5 residual)

**Author:** Erick Bourgeois

### Changed
- `src/constants.rs`: New `RECLAIM_CONFIG_DATA_KEY = "reclaim.toml"` — the literal was previously scattered across `helpers.rs` and the binary; centralising it removes the silent-drift risk if one side is renamed.
- `src/reclaim_agent.rs`: New pure helper `configmap_to_config(&ConfigMap) -> Result<Option<Config>, ConfigError>`. Three disciplined outcomes — `Ok(Some)` → arm scanner, `Ok(None)` → idle (CM exists but data key missing), `Err` → malformed payload, caller keeps last-good config. Also re-exports `RECLAIM_CONFIG_DATA_KEY` under the module path so tests and the bin can address it without reaching through `crate::constants::…`.
- `src/reclaim_agent_tests.rs`: 4 new tests covering the three outcomes above plus the `data: None` edge case. Brings reclaim-agent module test count to 33 (library total: **271 tests green**, up from 267).
- `src/bin/reclaim_agent.rs`: Rewrite. The agent no longer loads config from a file at startup — it runs a `kube::runtime::watcher` scoped by field selector `metadata.name=reclaim-agent-<NODE_NAME>` in `RECLAIM_AGENT_NAMESPACE` and bridges every `Event::Apply` / `Event::InitApply` / `Event::Delete` into a `tokio::sync::watch<Option<Config>>` channel. The scanner loop reads the current value each tick: `None` blocks on `rx.changed()` (with a 30s idle safety wakeup); `Some` scans `/proc` every `poll_interval_ms`. Hot-reload is automatic — a controller or operator edit to the ConfigMap propagates in at most one scanner tick. On a malformed CM edit the watcher logs and holds the previous config rather than disarming. `--config` flag removed; `--proc-root`, `--node-name`, `--oneshot` preserved.
- `src/reconcilers/helpers.rs`: `build_reclaim_agent_configmap` now writes under `RECLAIM_CONFIG_DATA_KEY` instead of the bare literal. Behaviour unchanged; the output CM is byte-identical.
- `deploy/node-agent/daemonset.yaml`: Removed the `config` ConfigMap volume and its `/etc/5spot` mount; removed the `--config=/etc/5spot/reclaim.toml` arg. The DaemonSet pod template no longer references any ConfigMap, which sidesteps the K8s limitation (`configMap.name` cannot be downward-API-templated per replica) that blocked the per-node mount previously.
- `deploy/node-agent/rbac.yaml`: New namespaced `Role` + `RoleBinding` (`5spot-reclaim-agent-configmaps` in `5spot-system`) granting the agent SA `get,list,watch` on ConfigMaps in that one namespace. Cluster-wide grant deliberately avoided — the agent only ever needs to see its own node's CM. Existing `ClusterRole` (nodes: get/patch) is unchanged.

### Why
The previous shape shared a single cluster-wide `reclaim-agent-config` ConfigMap across all nodes because the DaemonSet pod template cannot reference a per-replica ConfigMap name. That defeated the per-node specificity the controller was already projecting in the 24:15 increment: the operator could have `node-A` armed to kill `java` and `node-B` armed to kill `idea`, but both agents would see the same config. Moving the CM consumption into the agent (watch API instead of file mount) unblocks per-node arming end-to-end. User-confirmed direction over the alternative (DaemonSet-per-ScheduledMachine) to preserve the kata-deploy-style opt-in pattern and keep N agents at 1 DaemonSet regardless of fleet size.

### Impact
- [ ] Breaking change
- [x] Requires cluster rollout (new RoleBinding + DaemonSet spec change; kubelet rolls the agent pods on apply)
- [ ] Config change only
- [ ] Documentation only

**Operator notes:**
- The legacy shared `reclaim-agent-config` ConfigMap (if present from a prior install) is no longer mounted and can be deleted by hand; nothing in this tree references it any more.
- Agent arming is now entirely driven by per-node `reclaim-agent-<NODE_NAME>` ConfigMaps — either projected automatically by the controller from `spec.killIfCommands`, or created by hand for manual drills. Deleting the ConfigMap puts the agent back to idle on the next scanner tick.

---

## [2026-04-21 10:15] - Explicit pod-level securityContext on reclaim-agent DaemonSet (Trivy KSV-0118)

**Author:** Erick Bourgeois

### Changed
- `deploy/node-agent/daemonset.yaml`: Added a pod-level `spec.securityContext` block on the DaemonSet pod template. Fields: `runAsNonRoot: false`, `runAsUser: 0`, `runAsGroup: 0`, `seccompProfile.type: RuntimeDefault`. Preceded by a comment explaining why each field is the value it is — in particular that `runAsNonRoot: false` is architecturally required because reading `/proc/<pid>/{comm,cmdline}` for every host pid under hardened `hidepid=2` mounts needs UID 0, and that the non-root alternative (adding back `CAP_SYS_PTRACE`) is a broader privilege than "drop ALL caps + run as root". The per-container `securityContext` on the `agent` container is unchanged (still drops ALL caps, read-only root FS, no privilege escalation, RuntimeDefault seccomp).

### Why
Trivy KSV-0118 ("Default security context configured") fires when a pod does not declare a pod-level `spec.securityContext`, regardless of how locked-down the per-container context is. The DaemonSet had a thorough container-level block but nothing at pod scope, so Trivy reported the pod as "using the default security context, which allows root privileges." An explicit pod-level block satisfies the rule by making the operator's intent auditable at pod scope, without weakening any privilege — it enumerates the same choices the container already made. Keeps parity with the Phase 2.5 architectural constraints documented in the emergency-reclaim roadmap.

### Impact
- [ ] Breaking change
- [x] Requires cluster rollout (DaemonSet spec change — kubelet will restart the agent pods on rollout)
- [ ] Config change only
- [ ] Documentation only

**Follow-up watch:** if a future Trivy version flags the `runAsNonRoot: false` value itself (some versions tighten this under a different rule ID), the fallback is a `.trivyignore` entry with the same rationale already captured in this comment.

---

## [2026-04-21 10:00] - Suppress Trivy KSV-0010 for reclaim-agent DaemonSet (hostPID is required)

**Author:** Erick Bourgeois

### Changed
- `.trivyignore`: Added a new `DaemonSet — deploy/node-agent/daemonset.yaml` section with a single entry `AVD-KSV-0010` ("Access to host PID namespace"). The block comment explains the architectural rationale: the reclaim-agent watches host processes by scanning `/proc` for argv matches and signals host PIDs to execute graceful stops — both capabilities depend on the agent sharing the host's PID namespace, and pod-scoped `/proc` would make the feature inoperable. The suppression is intentionally narrow (one rule, one workload, one block comment) and follows the same pattern already established for `AVD-KSV-0046` / `AVD-KSV-0048` / `AVD-KSV-0041` on the RBAC `ClusterRole` and `AVD-KSV-0125` on the controller Deployment.

### Why
The IaC security scan job in `.github/workflows/build.yaml` runs `aquasecurity/trivy-action` in `config` mode against the whole repo and uploads SARIF to GitHub Code Scanning. Without an entry in `.trivyignore`, the `hostPID: true` on the reclaim-agent DaemonSet (required by design — see Phase 2.5 in the emergency-reclaim roadmap) would surface as a recurring HIGH finding in every PR and push. The workflow already references `./.trivyignore` (build.yaml:717), so the file is the right lever.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [x] Config change only
- [ ] Documentation only

---

## [2026-04-20 24:15] - Phase 2.5 remainder: controller-side label stamp + per-node ConfigMap projection

**Author:** Erick Bourgeois

### Changed
- `src/reconcilers/helpers.rs`: Added four pure helpers — `render_reclaim_toml(&[String]) -> String` (emits a commented TOML body that round-trips through `reclaim_agent::parse_config`), `per_node_configmap_name(&str) -> String` (`reclaim-agent-<node>` from `RECLAIM_AGENT_CONFIGMAP_PREFIX`), `build_reclaim_agent_configmap(node, commands) -> ConfigMap` (carries `app.kubernetes.io/component=reclaim-agent` for `kubectl get -l` discovery; `data["reclaim.toml"]` is the projected body), and `build_reclaim_agent_label_patch(enable: bool) -> serde_json::Value` (merge-patch that sets the label to `enabled` or to JSON `null` for delete). Async orchestrator `reconcile_reclaim_agent_provision(&Client, node, &[String])` drives the full projection: label PATCH first (load-bearing — the DaemonSet's `nodeSelector` depends on it), then ConfigMap apply on non-empty commands or delete on empty (404 treated as benign for idempotent tear-down). Server-side apply uses field manager `5spot-controller-reclaim-agent` distinct from the main reconciler so audit logs can attribute writes.
- `src/reconcilers/scheduled_machine.rs`: New `provision_reclaim_agent_best_effort` helper runs from `handle_active_phase` right after `patch_machine_refs_status` — this is where we first hold a real `nodeRef` for the current Machine. Projection is deliberately best-effort with respect to the reconcile loop: a failed label PATCH or ConfigMap apply degrades the emergency-reclaim path but must not block day-to-day scheduling. An empty `spec.killIfCommands` still invokes the orchestrator, which hits the tear-down arm (label=null, delete CM) so clearing the spec cleans up past projections idempotently.
- `src/reconcilers/helpers_tests.rs`: 11 new unit tests — 3 for `render_reclaim_toml` (round-trip via `parse_config`, empty-commands round-trip still parses, quote-in-command is escaped), 2 for `per_node_configmap_name` (uses prefix constant, preserves node name verbatim without re-sanitisation), 3 for `build_reclaim_agent_configmap` (namespace + name contract, `reclaim.toml` data key required by DaemonSet volume mount, operator-discovery labels present), and 3 for `build_reclaim_agent_label_patch` (enable writes the `enabled` constant, disable writes JSON null rather than empty string, strategic-merge safety — only `metadata.labels.<one key>` is touched).

### Why
Phase 2.5 MVP landed on 2026-04-20 with the CRD field (`spec.killIfCommands`) and an in-pod DaemonSet, but projection was deferred — operators had to hand-label nodes and share a single cluster-wide `reclaim-agent-config` ConfigMap, which both defeats the opt-in intent and loses per-node pattern specificity. The increment here wires the controller to do that projection automatically: a `ScheduledMachine` with `killIfCommands = ["java"]` now gets the opt-in label + a named `ConfigMap` carrying that list appearing on its backing Node without operator action, and clearing the list cleanly tears both down. Matches the 2026-04-20 ShipDocs "As-implemented" contract in the roadmap's Phase 2.5 follow-up block.

### Impact
- [ ] Breaking change
- [x] Requires cluster rollout (controller acquires PATCH on Nodes + CRUD on ConfigMaps in `5spot-system` — RBAC update required; tracked separately from this entry)
- [ ] Config change only
- [ ] Documentation only

**Note on the DaemonSet YAML switch:** `deploy/node-agent/daemonset.yaml` still references the single shared `reclaim-agent-config` ConfigMap. Switching the mount to the per-node `reclaim-agent-<node-name>` shape is blocked by a Kubernetes limitation (volume-level `configMap.name` cannot be templated by downward API); a future agent-side rewrite to fetch its own ConfigMap at startup will unblock that switch. The per-node ConfigMaps projected by this change are present in-cluster today — the DaemonSet will consume them once the agent side lands.

---

## [2026-04-20 23:45] - Phase 4 tests: pin check_emergency_reclaim early-return contract

**Author:** Erick Bourgeois

### Changed
- `src/reconcilers/scheduled_machine_tests.rs`: Added 4 regression tests covering the Phase-3 dispatch guard's early-return paths. Three cover `check_emergency_reclaim` returning `Ok(None)` without any apiserver call when (a) `status` is absent, (b) `status.node_ref` is `None`, or (c) `node_ref.name` is empty. The fourth pins `handle_emergency_remove_phase`'s fast-fail `InvalidConfig` guard for non-namespaced resources. Harness trick: the existing `le_mock_client()` has no response handle attached, so any outbound API request hangs forever — each test wraps the call in a 100ms `tokio::time::timeout` so a hang fails cleanly with a clear message rather than blocking CI.

### Why
Phase-3 dispatch shipped at 23:15 without unit-test coverage of the guard itself. The guard's contract — "do not touch the apiserver when there is no node to reclaim" — is load-bearing for two reasons: (1) a Node.get("") call 400s and would surface as a CAPI error rather than the benign None the caller expects, and (2) every pending `ScheduledMachine` (before CAPI populates `status.nodeRef`) would otherwise hit the apiserver on every reconcile. These tests pin that contract so a future refactor cannot regress the guard into an unconditional fetch.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [x] Documentation only (test-only change; no production code touched)

---

## [2026-04-20 23:30] - kind: tag locally built image as `local-dev`; override Deployment image via `kubectl set image`

**Author:** Erick Bourgeois

### Changed
- `Makefile`: `KIND_IMAGE` default changed from `ghcr.io/finos/5-spot:v0.1.0` (which matched the pinned tag in `deploy/deployment/deployment.yaml`) to `ghcr.io/finos/5-spot:local-dev`. The tag unambiguously signals a developer build and decouples the local image from the Deployment manifest's release pin. `kind-deploy` now runs `kubectl -n 5spot-system set image deployment/5spot-controller controller=$(KIND_IMAGE)` immediately after `apply -R -f deploy/deployment/` to redirect the Deployment at the locally loaded image.

### Why
User preference: the locally built image should be labeled as a developer build, not impersonate the pinned release tag. The post-apply `kubectl set image` patch keeps `deploy/deployment/deployment.yaml` untouched (and therefore safe to check in / reflect production) while still letting `kind-setup` run end-to-end against the local build.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [x] Config change only (build tooling only)
- [ ] Documentation only

---

## [2026-04-20 23:15] - Phase 3 dispatch: wire handle_emergency_remove into reconcile loop

**Author:** Erick Bourgeois

### Changed
- `src/constants.rs`: Added `PHASE_EMERGENCY_REMOVE = "EmergencyRemove"`, `REASON_EMERGENCY_RECLAIM_DISABLED_SCHEDULE`, and `EMERGENCY_DRAIN_TIMEOUT_SECS = 60` (deliberately shorter than the 300s graceful drain — the agent's process-match has already decided the node must leave).
- `src/reconcilers/helpers.rs`: New pure builders — `build_emergency_reclaim_event`, `build_emergency_disable_schedule_event`, `emergency_reclaim_message` — plus the orchestrating `handle_emergency_remove` handler that executes the seven-step ordering contract (event → phase EmergencyRemove → drain → delete Machine → PATCH enabled=false → clear annotations → phase Disabled). Load-bearing step (5) propagates errors so the loop-breaker can retry; steps 1, 3, 6 are best-effort. Removed `#[allow(dead_code)]` from `ReclaimRequest`, `node_reclaim_request`, `build_clear_reclaim_patch`, `build_disable_schedule_patch` — all now have callers.
- `src/reconcilers/scheduled_machine.rs`: New `check_emergency_reclaim` guard runs after `kill_switch` but before schedule evaluation; fetches the Node, calls `node_reclaim_request`, and drives the full handler when annotated. New `handle_emergency_remove_phase` match arm catches a crash between annotation-clear (step 6) and phase-flip (step 7), finishing the transition to Disabled idempotently. Updated state-diagram doc comment with the EmergencyRemove → Disabled edge.
- `src/crd_tests.rs`: 3 new tests pinning the phase name, the disable-schedule event reason, and the drain-timeout bound (via `const` block).
- `src/reconcilers/helpers_tests.rs`: 10 new unit tests covering the three pure builders — warning severity, reason-constant use, note formatting with and without the optional reason/timestamp, and message-floor behaviour.
- Pre-existing clippy cleanups: backtick-wrapped `DaemonSet`/`ConfigMap` in rustdoc across `src/constants.rs`, `src/crd.rs`, `src/bin/reclaim_agent.rs`; collapsed duplicate `MatchSource` match arm to `|`; converted `match` to `let-else` in `scan_proc`; swapped a `Default::default()` for explicit `BTreeMap::default()` in reclaim_agent_tests.

### Why
The Phase-3 primitives landed in the previous session (helpers shipped behind `#[allow(dead_code)]`) — they had no caller. This change wires them into the reconcile loop so the annotation actually triggers the reclaim. Without this step, the node-side agent could paint the reclaim annotation but the controller would ignore it and keep following the schedule, re-adding the node at the next window. The seven-step ordering contract (disable-schedule *before* annotation-clear) ensures a crash at any point leaves state that the next reconcile can replay idempotently from the top.

### Impact
- [ ] Breaking change
- [x] Requires cluster rollout (controller must ship the new dispatch path; agent manifests unchanged)
- [ ] Config change only
- [ ] Documentation only

---

## [2026-04-20 21:20] - kind-load: bypass `cross`, native cargo cross-compile + plain `docker build`; kind-deploy: apply namespace first

**Author:** Erick Bourgeois

### Changed
- `Makefile`: `kind-load` is now self-contained and no longer calls into the `docker-build-*` → `prepare-binaries-linux-*` → `build-linux-*` chain. New recipe: host-arch → triple/linker/docker-arch mapping; checks the cross-linker is on PATH (prints `brew tap messense/macos-cross-toolchains && brew install <triple>` hint if missing); ensures the rustup target is installed via `rustup target add`; runs `cargo build --release --target <triple>` (leveraging the `linker = "<triple>-gcc"` entry already in `.cargo/config.toml`); stages the binary at `binaries/<docker-arch>/5spot`; builds the image with plain `docker build --build-arg TARGETARCH=<docker-arch> -t $(KIND_IMAGE) .` (no buildx → no `~/.docker/buildx/buildkitd.toml` dependency, no `cross` Docker image pull); loads into kind.
- `Makefile`: `kind-deploy` now applies `deploy/deployment/namespace.yaml` first, polls for namespace existence (up to 10s), then runs `kubectl apply -R -f deploy/deployment/`. Prevents the `NotFound: namespaces "5spot-system" not found` race where the ConfigMap/Deployment admission requests arrive before the kube-apiserver's namespace controller has fully registered the namespace even though the Namespace object was already created in the same `apply -R` batch.

### Why
1. **cross bypass**: `cross 0.2.5` on Apple Silicon fails for both `aarch64-unknown-linux-gnu` and `x86_64-unknown-linux-gnu` targets with `toolchain 'stable-x86_64-unknown-linux-gnu' may not be able to run on this system` — rustup rejects installing a non-native-host toolchain even though the requested *target* is correct. The `build-linux-arm64` recipe tries `cross` first and only falls back to the Homebrew `messense/macos-cross-toolchains` path if `cross` is absent. Rather than editing that existing target (which CI relies on), `kind-load` now does its own simpler build that uses the Homebrew toolchain directly.
2. **namespace race**: `kubectl apply -R` does sort by install-order (namespaces before namespaced resources) but kind's admission stack can briefly NotFound a freshly-created namespace for namespace-scoped resources submitted in the same batch. An explicit namespace-first apply plus short readiness poll eliminates the race.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [x] Config change only (build tooling only; `kind-load` now requires the Homebrew cross toolchain on macOS — prints an install hint if missing)
- [ ] Documentation only

---

## [2026-04-20 20:55] - kind-load: build image for host arch, not hardcoded amd64

**Author:** Erick Bourgeois

### Changed
- `Makefile`: `kind-load` no longer depends on `docker-build` (which forces `linux/amd64`). The recipe now inspects `uname -m`, dispatches to `docker-build-amd64` on `x86_64` and `docker-build-arm64` on `arm64`/`aarch64`, and retags the matching per-arch local image (`$(IMAGE_NAME):$(IMAGE_TAG)-$$ARCH`) as `$(KIND_IMAGE)` before `kind load`.

### Why
On Apple Silicon, `cross build --target x86_64-unknown-linux-gnu` invoked by `docker-build` → `build-linux-amd64` tries to install the `stable-x86_64-unknown-linux-gnu` host toolchain via rustup, which rejects it on an arm64 host (`toolchain may not be able to run on this system`). Even if the build succeeded via `--force-non-host`, kind nodes on Apple Silicon run arm64, so an amd64 image would fail at pod start with exec-format errors. Building for the host arch matches kind's node arch and sidesteps the cross-toolchain install entirely.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [x] Config change only (build tooling only)
- [ ] Documentation only

---

## [2026-04-20 20:30] - Makefile targets for local kind-cluster testing

**Author:** Erick Bourgeois

### Changed
- `Makefile`: New `kind-*` target group — `kind-install` (downloads kind v0.24.0 with SHA-256 checksum verification, platform-detected), `kind-create` / `kind-delete` (idempotent lifecycle for a cluster named `$(KIND_CLUSTER_NAME)`, default `5spot-dev`, using `$(KIND_NODE_IMAGE)`, default `kindest/node:v1.31.0`), `kind-load` (docker-build + retag local image as `$(KIND_IMAGE)` → `ghcr.io/finos/5-spot:v0.1.0` to match the in-cluster `Deployment.spec.containers.image`, then `kind load docker-image`), `kind-deploy` (applies `deploy/crds/` + `deploy/deployment/` and waits on `rollout status deployment/5spot-controller -n 5spot-system` up to 180s), `kind-example` (applies `examples/scheduledmachine-basic.yaml`), `kind-status` (summary of cluster / controller pods / ScheduledMachines), `kind-setup` (meta target chaining `kind-create` → `kind-load` → `kind-deploy`). Added 5 new variables (`KIND_VERSION`, `KIND_CLUSTER_NAME`, `KIND_NODE_IMAGE`, `KIND_IMAGE`) and registered the 8 new targets in `.PHONY`.
- `docs/src/development/setup.md`: Local-Kubernetes/kind section now documents the new `make kind-*` targets (one-shot + individual steps + override examples) and keeps the raw `kind create cluster` invocation as a fallback for bespoke cluster topologies.

### Why
Developers testing `ScheduledMachine` locally currently have to hand-assemble the `kind create` → `docker build` → image-tag-align → `kind load` → `kubectl apply -f deploy/crds/` → `kubectl apply -R -f deploy/deployment/` sequence, and each step has a non-obvious gotcha (the deployment hardcodes `ghcr.io/finos/5-spot:v0.1.0`, `imagePullPolicy: IfNotPresent` only resolves if the loaded image matches, the controller namespace is `5spot-system`). The new targets encode that sequence so `make kind-setup` is the one command for a working test environment.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [x] Config change only (build tooling only; no source, CRD, or deploy-manifest changes)
- [x] Documentation only (updated `docs/src/development/setup.md`)

---

## [2026-04-20 22:30] - Emergency reclaim — disable-schedule patch + lifecycle docs

**Author:** Erick Bourgeois

### Changed
- `src/reconcilers/helpers.rs`: New `build_disable_schedule_patch()` helper — returns a merge-patch body that flips `spec.schedule.enabled=false` on the owning `ScheduledMachine`. Marked `#[allow(dead_code)]` pending Phase-3 reconciler dispatch (same pattern as the sibling `build_clear_reclaim_patch`). Rustdoc documents the ordering contract: disable-schedule PATCH must run *before* the annotation-clear PATCH so that a crash between the two steps retries idempotently from the top.
- `src/reconcilers/helpers_tests.rs`: 3 new tests — `enabled=false` is emitted as the JSON literal `false` (not null, which would delete the key under merge-patch semantics); strategic-merge safety asserting only `spec.schedule.enabled` is addressed and no siblings are touched; belt-and-braces guard that the function never emits `enabled=true` (re-enable is a human action, never a controller decision).
- `docs/src/concepts/emergency-reclaim.md` (new): Dedicated lifecycle page for the process-match kill switch. Five mermaid diagrams — trigger chain flowchart, lifecycle state diagram showing `EmergencyRemove` integrated into the existing phase machine, sequence diagram for the end-to-end eject flow, opt-in installation flow, re-enable flow showing what happens when the user re-enables with the matched process still running. Covers why the exit is `Disabled` (not `Pending`), the two-kill-switch family (`killSwitch` vs `killIfCommands`), matching semantics (comm exact vs cmdline substring, case-sensitive), observability (two distinct Events, the `EmergencyReclaimDisabledSchedule` condition, transient Node annotations, agent logs), and explicit out-of-scope items.
- `docs/src/concepts/machine-lifecycle.md`: Added `EmergencyRemove` to the state diagram (transitions in from `Pending` / `Active` / `ShuttingDown`; exits to `Disabled`) and to the phase descriptions. New "Emergency Reclaim Flow" subsection alongside the existing "Kill Switch Flow" so readers can compare the two mechanisms side by side.
- `docs/src/concepts/scheduled-machine.md`: Added `killIfCommands` to the Other Fields table; updated the phase enumeration in Status Fields; added an emergency-reclaim cross-link under Related.
- `docs/src/operations/troubleshooting.md`: New "Emergency Reclaim (Kill Switch)" section — four scenarios (stuck in `EmergencyRemove`; eject loop on every schedule window; agent never fires on a known-matching process; `EmergencyReclaim` event fires but schedule is not disabled) each with concrete `kubectl` / `jq` diagnostics and resolution steps.
- `docs/mkdocs.yml`: Registered `Emergency Reclaim (Kill Switch): concepts/emergency-reclaim.md` under the Concepts nav section. Verified with `mkdocs build` — page renders with all five mermaid blocks.
- `docs/roadmaps/5spot-emergency-reclaim-by-process-match.md` (note: lives in `~/dev/roadmaps`, not in git per user convention): Phase 3 section now mandates the `spec.schedule.enabled=false` flip + the `EmergencyReclaimDisabledSchedule` condition/Event; new Open Question 6 captures the "explicit re-enable vs. auto-resume on process exit" trade-off with MVP rationale; Phase 4 tests include the flip assertion and a re-enable-loop-protection test; user-facing re-enable flow documented verbatim so Phase 3 code + operator docs stay in sync.

### Why
Follow-through on the roadmap review question: when emergency reclaim ejects a node, does it set `spec.schedule.enabled=false`? The correct answer is yes — otherwise the next schedule window silently re-adds the node, the agent sees the still-running matched process, and the eject→re-add→re-eject loop repeats every schedule boundary forever. The `build_disable_schedule_patch` helper gives the Phase-3 reconciler-dispatch work the primitive it needs to break that loop. Documentation was the second half of this commit because the kill-switch semantics (two mechanisms, different triggers, different exits, different reset paths) are not obvious from the code alone — a lifecycle feature this consequential needs a dedicated concepts page with diagrams, a troubleshooting section for the field, and cross-links from every page that mentions either kill switch.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout (the new helper is `#[allow(dead_code)]` until Phase 3 wires it into the reconciler — no runtime behaviour change in this commit)
- [ ] Config change only
- [x] Documentation only (the one code change — `build_disable_schedule_patch` — is dormant until Phase 3 calls it)

### Follow-up (not in this commit)
- Phase 3 dispatch wiring: call `build_disable_schedule_patch` after drain + Machine delete, before `build_clear_reclaim_patch`; emit the `EmergencyReclaimDisabledSchedule` Event and condition documented on the new concepts page; move the `#[allow(dead_code)]` off the helper once it has a caller.
- Integration test (in `tests/`): kind cluster + stub k0smotron, annotate Node, assert `spec.schedule.enabled` flips to `false`, assert the condition / Event appear, assert re-enabling while the trigger is still present re-fires the eject and re-flips `enabled=false`.

---

## [2026-04-20 19:45] - Emergency reclaim by process match (Phases 1–2.5 MVP)

**Author:** Erick Bourgeois

### Changed
- `src/crd.rs`: New optional `kill_if_commands: Option<Vec<String>>` field on `ScheduledMachineSpec`. Opt-in semantics mirror kata-deploy — `None` / absent preserves existing behavior; `Some(vec![...])` opts the node in to the reclaim-agent DaemonSet. `Some(vec![])` is legal but inert (detector never matches) and reserved for future controller-side validation warnings.
- `src/crd_tests.rs`: 4 positive/negative tests for the new field + 5 tests for the reclaim annotation/label constants, plus `kill_if_commands: None` added to the existing spec-serialization test.
- `src/constants.rs`: 9 new constants — reclaim trigger / reason / timestamp annotation keys, literal `"true"` trigger value, the opt-in node label key + value, the agent namespace, the per-node ConfigMap prefix, and the `EmergencyReclaim` condition reason.
- `src/reclaim_agent.rs` (new): Library surface for the node-side agent — `Config` (TOML), `parse_config` / `load_config`, `scan_proc` (reads `/proc/<pid>/comm` exact + `/proc/<pid>/cmdline` substring), `Match` / `MatchSource`, `build_patch_body` (strategic-merge that only touches `metadata.annotations`), `already_requested` (strict literal-`"true"` check). Empty match lists return `None` by design; `poll_interval_ms = 0` is rejected at load time.
- `src/reclaim_agent_tests.rs` (new): 21 tests — config happy/error paths, synthetic `/proc` trees via `tempfile::TempDir`, case-sensitivity, mid-scan process-exit race tolerance, missing-`/proc`-root error, idempotence across `"true"` / `"false"` / `"0"` / empty values.
- `src/bin/reclaim_agent.rs` (new): `5spot-reclaim-agent` binary. Clap CLI with `--config`, `--proc-root`, `--node-name` (downward API, required), `--oneshot`. Reads NODE via downward API, builds in-cluster kube client, runs idempotence pre-check, then loops `scan_proc` at the configured interval and PATCHes the Node on first match. Field manager name is distinct (`5spot-reclaim-agent`) so audit logs can attribute writes.
- `src/reconcilers/helpers.rs`: `ReclaimRequest` struct + `node_reclaim_request(node)` detector (returns `None` unless the trigger annotation is literally `"true"`) + `build_clear_reclaim_patch()` (merge-patch that sets all three annotations to `null` — Kubernetes merge-patch semantics for key deletion).
- `src/reconcilers/helpers_tests.rs`: 7 tests covering the detector across all annotation shapes + clear-patch structure; added `kill_if_commands: None` to the existing test spec literal.
- `src/reconcilers/scheduled_machine_tests.rs`: Added `kill_if_commands: None` to `create_test_spec`.
- `src/lib.rs`: Exposed the new `reclaim_agent` module.
- `src/bin/crddoc.rs`: Added `killIfCommands` example snippet + full field description block (crddoc is static `println!`-driven, not schema-derived).
- `docs/reference/api.md`: Regenerated via `crddoc`; now documents `killIfCommands` (lines 55–57, 133–143).
- `deploy/crds/scheduledmachine.yaml`: Regenerated via `crdgen`; schema now exposes `killIfCommands` at line 92.
- `Cargo.toml`: Added `toml = "0.8"` dependency + a new `[[bin]]` block for `5spot-reclaim-agent` targeting `src/bin/reclaim_agent.rs`.
- `deploy/node-agent/daemonset.yaml` (new): Opt-in DaemonSet gated by `nodeSelector: { 5spot.finos.org/reclaim-agent: enabled }`. Runs as root (reads every pid) with `hostPID: true`, host `/proc` mounted read-only at `/host/proc`, `system-node-critical` priority, tolerates every taint, `terminationGracePeriodSeconds: 5`. Config comes from a ConfigMap-projected volume at `/etc/5spot/reclaim.toml`. Image is pinned to `v0.1.0` (no `:latest`).
- `deploy/node-agent/rbac.yaml` (new): Dedicated ServiceAccount + ClusterRole + ClusterRoleBinding. Scope is narrow by design — only `get` + `patch` on `nodes`, no `list` / `watch`, no access to Machine / Pod / ScheduledMachine. Separation of identity from the main controller (distinct SA) lets audit logs unambiguously attribute a PATCH to the node-side agent vs. the controller.
- `deploy/node-agent/reclaim.toml.example` (new): Reference config — the real config is projected per-node by the controller.
- `deploy/node-agent/kustomization.yaml` (new): `kubectl apply -k deploy/node-agent/` bundle.

### Why
Follow-through on the `docs/roadmaps/5spot-emergency-reclaim-by-process-match.md` roadmap: certain workloads on shared fleet nodes (interactive JVMs left overnight, game clients, IDE processes) can indefinitely delay graceful drain windows, blocking scheduled remove operations and stranding capacity. The emergency-reclaim path gives operators an opt-in, process-match trigger that bypasses `gracefulShutdownTimeout` / `nodeDrainTimeout` and moves the machine into a non-graceful remove phase immediately on first match. The MVP implements rung 1 of the two-rung detection ladder (`/proc` poll) with the same Node-annotation contract that rung 2 (netlink proc connector, future work) will reuse. The opt-in is doubly gated — a node must (a) be labeled `5spot.finos.org/reclaim-agent=enabled` by the controller (which only stamps when the parent `ScheduledMachine` has a non-empty `killIfCommands`) AND (b) have its per-node ConfigMap populated — so the feature has zero effect on clusters that do not configure it.

### Impact
- [ ] Breaking change (new field is opt-in; absent / `None` preserves all existing behavior)
- [x] Requires cluster rollout (`kubectl apply -f deploy/crds/scheduledmachine.yaml` to pick up `killIfCommands` in the CRD schema; `kubectl apply -k deploy/node-agent/` only if operators want the reclaim agent available — otherwise it remains inert)
- [ ] Config change only
- [ ] Documentation only

### Follow-up (not in this commit)
- Controller dispatch: wire `node_reclaim_request()` + `build_clear_reclaim_patch()` into the reconciler, add `Phase::EmergencyRemove`, emit a Kubernetes Event with `REASON_EMERGENCY_RECLAIM`, call `kubectl drain --grace-period=0 --force --disable-eviction`, delete the CAPI `Machine` immediately, clear the annotation last.
- Controller-side projection of the per-node `reclaim-agent-<node-name>` ConfigMap + label stamp from `spec.killIfCommands` — currently the manifests assume a single shared `reclaim-agent-config`; the controller work will switch this to per-node as the roadmap specifies.
- Rung 2 (netlink proc connector) — requires `CAP_NET_ADMIN`; out of MVP scope.
- OCI image build + release wiring for the `5spot-reclaim-agent` binary (tarball + SBOM + VEX path already exists for the controller; extend it).

---

## [2026-04-20 00:15] - Grant `update` on `scheduledmachines/finalizers` to unblock Machine creation

**Author:** Erick Bourgeois

### Changed
- `deploy/deployment/rbac/clusterrole.yaml`: New rule granting `update` on `scheduledmachines/finalizers` under the `5spot.finos.org` API group. Added as a separate rule (rather than widening the existing `scheduledmachines` / `scheduledmachines/status` rule) so the verb surface on the finalizers subresource stays exactly at the minimum the API-server admission check requires. Inline comment documents the link back to `src/reconcilers/helpers.rs` where `blockOwnerDeletion: true` is set, and quotes the exact API-server error that the missing rule produces.

### Why
`src/reconcilers/helpers.rs:859` stamps `blockOwnerDeletion: true` on the `ownerReference` it writes from every child CAPI `Machine` back to its parent `ScheduledMachine`. Kubernetes treats `blockOwnerDeletion` as functionally equivalent to holding a finalizer on the owner, so during admission it checks that the creating service account has `update` on `<owner-resource>/finalizers` — a separate subresource in RBAC terms from the main `scheduledmachines` resource and from `scheduledmachines/status`. The controller's ClusterRole has shipped since the initial commit with full CRUD on `scheduledmachines` and `scheduledmachines/status` but without any rule for the `finalizers` subresource, so every Machine creation is rejected by the API server with `machines.cluster.x-k8s.io "<name>" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on`. This is a latent RBAC bug, not a regression. The fix grants exactly the verb the API server checks (`update`); no broader verb set is needed and granting more would violate least-privilege.

### Impact
- [ ] Breaking change
- [x] Requires cluster rollout (re-apply the ClusterRole; no controller restart required — RBAC changes are picked up on the next API request)
- [ ] Config change only
- [ ] Documentation only

### Follow-up (optional, not in this commit)
- Integration test that creates a `ScheduledMachine` against a kind cluster with the shipped ClusterRole and asserts the child `Machine` is admitted. Would catch any future regression in RBAC vs. `ownerReference` shape without waiting for a prod cluster rollout to expose it.

---

## [2026-04-19 21:50] - Triage GHSA-cq8v-f236-94qc (rand soundness); widen VEX identifier shapes

**Author:** Erick Bourgeois

### Changed
- `osv-scanner.toml` (new, at repo root): Ignore entry for `GHSA-cq8v-f236-94qc` with `ignoreUntil = 2026-10-19` and a reason that mirrors the VEX statement. This is the file shape Scorecard's `Vulnerabilities` check (and osv-scanner itself) expect — per the remediation text in the Scorecard finding.
- `.vex/GHSA-cq8v-f236-94qc.toml` (new): OpenVEX statement with `status = not_affected`, `justification = vulnerable_code_not_in_execute_path`, explicit impact statement. Applies to both the Chainguard and Distroless product identifiers.
- `tools/validate_vex.py`: Widened the accepted identifier set from `CVE-YYYY-NNNN+` only to also accept `GHSA-xxxx-xxxx-xxxx` (case-insensitive) and `RUSTSEC-YYYY-NNNN`. The TOML field is still named `cve` for backward compatibility with the existing 11 statement files. New `_is_accepted_id()` helper composes the three regexes; error message updated to list all three shapes.
- `tools/assemble_openvex.py`: Dropped the `.upper()` on `doc["cve"]` when rendering `vulnerability.name`. GHSA ID segments are canonically lowercase and upper-casing them breaks round-tripping against osv.dev / github.com/advisories. CVE IDs in existing files are already uppercase, so this is a no-op for them.
- `tools/tests/validate-vex-tests.sh`: Added happy-path cases `valid-ghsa` and `valid-rustsec`, plus negative cases `invalid-ghsa-format`, `invalid-rustsec-format`, and `duplicate-ghsa` (uniqueness check across case variants of a GHSA ID).
- `tools/tests/fixtures/{valid-ghsa,valid-rustsec,invalid-ghsa-format,invalid-rustsec-format,duplicate-ghsa}/` (new fixture dirs).
- `tools/tests/assemble-openvex-tests.sh`: Added regression guard asserting `GHSA-cq8v-f236-94qc` is emitted verbatim (not upper-cased) into `vulnerability.name`.
- `.vex/README.md`: Documented the expanded identifier format.

### Why
OpenSSF Scorecard's `Vulnerabilities` rule flagged `GHSA-cq8v-f236-94qc` (rand soundness when a custom `log` logger calls `rand::rng()` mid-reseed) on `rand 0.8.6`, pulled transitively via `warp 0.3.7` → `tokio-tungstenite 0.21` → `tungstenite 0.21` where rand is used exclusively for websocket frame masking. The vulnerable path requires the `log` logger itself to call into rand during a reseed, which 5-Spot's tracing-subscriber stack never does. The advisory ships without a CVE ID, which the existing VEX validator (`CVE-YYYY-NNNN+` only) rejected — making it the first advisory the VEX pipeline could not represent. Widening the validator to accept GHSA + RUSTSEC identifiers unblocks this triage and every future non-CVE advisory without forcing a field rename of the existing 11 files. The full fix (rand >= 0.9.3) requires crossing a warp 0.3.x semver boundary (warp 0.3 hard-pins tokio-tungstenite ^0.21) and is tracked as a follow-up, not attempted here.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [x] Config change only (CI / supply-chain)
- [ ] Documentation only

### Follow-up (optional, not in this commit)
- Warp 0.3 → 0.4 or axum migration so `tokio-tungstenite` and the transitive `rand` can be bumped past the vulnerable range; remove `osv-scanner.toml` entry and the VEX statement when that lands.
- Consider renaming the TOML field from `cve` to `id` once every existing statement has been regenerated; not worth the churn today.

---

## [2026-04-19 21:25] - Remove bogus `vexctl validate` step from build-vex job

**Author:** Erick Bourgeois

### Changed
- `.github/workflows/build.yaml` (`build-vex` job): Deleted the `Install vexctl` step (VEXCTL_VERSION 0.3.0) and the `Run vexctl validate vex.openvex.json` step. Updated the inline comment on the adjacent `Install Grype` step to drop the now-dangling "mirrors the vexctl install step above" reference.

### Why
`vexctl` has no `validate` subcommand — it never has — and runs of `build-vex` on push-to-main were failing with `unknown command "validate" for "vexctl"`. The step was added on the assumption that vexctl mirrored the OpenVEX reference implementation's schema validator; it does not. vexctl's 0.3.0 subcommand surface is `attest / create / generate / merge / verify`. Since vexctl was not used anywhere else in the job (Cosign does the attestation, not vexctl), the install step was removed with it rather than left as dead weight. Input-side schema/enum/uniqueness validation is still performed up front by `tools/validate-vex.sh` (which runs `validate_vex.py` against `.vex/*.toml`) — that's the real defensive gate. No behaviour change on pull_request events: the whole `build-vex` job is gated by `if: github.event_name != 'pull_request'` and only runs on push-to-main + release.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [x] Config change only (CI workflow)
- [ ] Documentation only

### Follow-up (optional, not in this commit)
If a second-opinion schema check of the *assembled* `vex.openvex.json` is wanted, add a step that validates against the OpenVEX JSON Schema with a tool that actually ships a validator (e.g. `python -m jsonschema -i vex.openvex.json openvex-schema.json`). Record the dependency in `~/dev/roadmaps/5spot-vex-generation-and-signing.md § Dependencies`.

---

## [2026-04-19 21:10] - Add child-cluster MutatingAdmissionPolicy to label Nodes for kata-deploy

**Author:** Erick Bourgeois

### Changed
- `deploy/admission/child-cluster-kata-runtime-mutatingpolicy.yaml` (new): `MutatingAdmissionPolicy` (`admissionregistration.k8s.io/v1alpha1`) that matches `nodes` on `CREATE` and stamps `katacontainers.io/kata-runtime=true` onto `metadata.labels` via an `ApplyConfiguration` patch. `failurePolicy: Ignore` so a policy evaluation error can never block kubelet from registering a Node.
- `deploy/admission/child-cluster-kata-runtime-mutatingpolicybinding.yaml` (new): `MutatingAdmissionPolicyBinding` activating the policy cluster-wide on the child cluster. `matchResources: {}` covers every Node; an `objectSelector` can be added to scope to a specific pool.

### Why
The child (workload) cluster needs Nodes labelled for the upstream `kata-deploy` DaemonSet's default `nodeSelector`. Labelling at Node `CREATE` is the cleanest admission shape: kubelet updates the `Ready` condition via the `nodes/status` subresource, which the API server does not allow to mutate `metadata.labels`, so a true "on Ready" mutation is not possible. `kata-deploy` is a DaemonSet, and DaemonSet pod lifecycle already gates installation on Node readiness — matching on `CREATE` and letting the DaemonSet handle Ready yields the behaviour the user asked for without introducing a controller.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [x] Config change only (child-cluster manifests; not wired into the 5-spot controller)
- [ ] Documentation only

### Follow-up (optional, not in this commit)
- Promote to `admissionregistration.k8s.io/v1beta1` once the child-cluster floor is Kubernetes >= 1.34.
- If a subset of Nodes should opt out of kata, add an `objectSelector` to the binding rather than conditions in the policy expression.

---

## [2026-04-19 20:45] - Fix vexctl installer 404 in build.yaml

**Author:** Erick Bourgeois

### Changed
- `.github/workflows/build.yaml` (Install vexctl step): Changed the download URL from the non-existent `vexctl_${VEXCTL_VERSION}_linux_amd64.tar.gz` tarball to the raw binary `vexctl-linux-amd64`. Removed the `tar -xzf` step accordingly. Kept the version pin at `0.3.0` and the inline comment, expanded to describe the actual asset layout (raw binaries + detached Cosign `.sig`/`.pem` per platform).

### Why
The OpenVEX project does not publish versioned tarballs for vexctl. Their release assets are raw platform binaries named `vexctl-<os>-<arch>` (hyphens, no version, no `.tar.gz`). The workflow was copy-pasted from the Grype install step — which *does* ship tarballs in the format `grype_<version>_<os>_<arch>.tar.gz` — so the pattern looked reasonable but 404ed on the first CI run. Verified the correct asset name against `api.github.com/repos/openvex/vexctl/releases/tags/v0.3.0`. Grype's own installer was audited and is correct.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [x] Config change only (CI workflow)
- [ ] Documentation only

### Follow-up (optional, not in this commit)
vexctl ships detached Cosign signatures (`vexctl-linux-amd64.sig` + `.pem`). A follow-up could verify the binary with `cosign verify-blob` before installing, closing a small supply-chain gap on the installer itself.

---

## [2026-04-19 20:30] - Add Community section to README (FINOS Slack #5-spot)

**Author:** Erick Bourgeois

### Changed
- `README.md`: New `## Community` section between `## Contributing` and `## Security`, pointing to `#5-spot` on the FINOS Slack workspace (<https://finos-lf.slack.com/channels/5-spot>, join at <https://finos.org/slack>), plus GitHub Issues and GitHub Discussions as the other two canonical contact surfaces. Security-sensitive reports are explicitly redirected back to the Security section so they do not land in public Issues.

### Why
New contributors and users landing on the repo had no documented way to reach the project maintainers for usage questions short of opening an Issue. Publishing the FINOS Slack channel — now that the project lives under the FINOS org — gives a low-friction conversation surface and matches what every other FINOS project does.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [x] Documentation only

---

## [2026-04-19 20:15] - Rename Service to `controller` (RFC 1035 compliance)

**Author:** Erick Bourgeois

### Changed
- `deploy/deployment/service.yaml`: Renamed `metadata.name` from `5spot-controller` to `controller`. Kubernetes Services require **RFC 1035** DNS labels (`[a-z]([-a-z0-9]*[a-z0-9])?`) — stricter than the RFC 1123 rule used by Namespace/Deployment/ConfigMap/RBAC — and labels must start with a letter, not a digit. The cluster API server rejects `5spot-controller` with `metadata.name: Invalid value: "5spot-controller": a DNS-1035 label…`. Added a header comment explaining the constraint so nobody renames it back.
- `docs/src/installation/controller.md`: Updated the port-forward example from `svc/5spot-controller` to `svc/controller`.

### Why
`kubectl apply --dry-run=client` did not catch this — client-side dry-run skips the Service-specific admission validator. Only `--dry-run=server` (or an actual apply) surfaces the RFC 1035 rule. Renaming to `controller` is safe here because the Service is already scoped by the `5spot-system` namespace; cluster DNS resolves it as `controller.5spot-system.svc.cluster.local`. The ServiceMonitor selector matches on label `app: 5spot-controller` (not on Service name), so monitoring continues to work without change.

### Audit: every other kube resource name

Server-side validated against a live cluster — all resources other than the Service were already compliant. Kubernetes applies different name rules per kind; this matrix confirms each one:

| Kind | Name | Rule | Result |
|---|---|---|---|
| Namespace | `5spot-system` | RFC 1123 label (digit-start OK) | ✅ |
| ConfigMap | `5spot-controller-config` | RFC 1123 subdomain | ✅ |
| Deployment | `5spot-controller` | RFC 1123 subdomain | ✅ |
| Service | `controller` | **RFC 1035 label** (letter-start required) | ✅ (renamed) |
| NetworkPolicy | `5spot-controller` | RFC 1123 subdomain | ✅ |
| PodDisruptionBudget | `5spot-controller` | RFC 1123 subdomain | ✅ |
| ClusterRole | `5spot-controller` | RFC 1123 subdomain | ✅ |
| ClusterRoleBinding | `5spot-controller` | RFC 1123 subdomain | ✅ |
| ServiceAccount | `5spot-controller` | RFC 1123 subdomain | ✅ |
| ServiceMonitor | `5spot-controller` | CR default = RFC 1123 subdomain | ✅ |
| CustomResourceDefinition | `scheduledmachines.5spot.finos.org` | `<plural>.<group>`, each label RFC 1123 (digit-start OK) | ✅ |
| ValidatingAdmissionPolicy | `scheduledmachine-validation` | RFC 1123 subdomain | ✅ |
| ValidatingAdmissionPolicyBinding | `scheduledmachine-validation-binding` | RFC 1123 subdomain | ✅ |
| ScheduledMachine | `business-hours-worker`, `weekend-worker` | RFC 1123 subdomain | ✅ |

### Unrelated issue surfaced during validation
`examples/scheduledmachine-weekend.yaml` fails server-side admission with `unknown field "spec.schedule.cron"` — the example is out of date against the current CRD schema (which uses day/hour ranges, not cron expressions). Tracked separately; not fixed in this commit.

### Impact
- [x] Breaking change (anyone referencing `5spot-controller.5spot-system.svc.cluster.local` — e.g. in ServiceMonitor `endpoints[*].port` lookups by Service name, Ingress backends, or custom dashboards — must use `controller.5spot-system.svc.cluster.local`)
- [x] Requires cluster rollout (old Service object must be `kubectl delete`d; the new one will be created on re-apply)
- [ ] Config change only
- [ ] Documentation only

---

## [2026-04-19 19:45] - Resolve Trivy IaC findings in GitHub Code Scanning

**Author:** Erick Bourgeois

### Changed
- `deploy/deployment/deployment.yaml`:
  - Replaced `image: ghcr.io/RBC/5-spot:latest` with `image: ghcr.io/finos/5-spot:v0.1.0` (Trivy **KSV-0013** — image `:latest` tag; also removes a stale RBC-internal reference that violated the CLAUDE.md internal-references rule and pointed at a non-existent org).
  - Added `seccompProfile: { type: RuntimeDefault }` at both the pod `spec.securityContext` and the container `securityContext` (Trivy **KSV-0104** — seccomp policies disabled, **KSV-0030** — runtime/default seccomp profile not set).
  - Added `runAsGroup: 65534` to the container `securityContext` (Trivy **KSV-0021** — runs with low GID; 65534 is the `nogroup`/`nobody` GID on distroless and Chainguard images).
- `Dockerfile`:
  - Removed "RBC Capital Markets" from the copyright header (CLAUDE.md internal-references rule).
  - Pinned the distroless base image by digest: `gcr.io/distroless/cc-debian13:nonroot@sha256:8f960b7fc6a5d6e28bb07f982655925d6206678bd9a6cde2ad00ddb5e2077d78`. Dependabot (docker ecosystem) will open a re-pin PR when Google publishes a patched image.
- `Dockerfile.chainguard`:
  - Removed "RBC Capital Markets" from the copyright header.
  - Pinned the Chainguard glibc-dynamic base image by digest: `cgr.dev/chainguard/glibc-dynamic:latest@sha256:fa0d07a6a352921b778c4da11d889b41d9ef8e99c69bc2ec1f8c9ec46b2462e9` (Trivy **DS-0001** — `:latest` tag used). Chainguard rebuilds this tag daily with security patches and Dependabot picks up the new digest on each rebuild.
- `.trivyignore` (new): Six architecturally-justified suppressions, each with a written rationale so an auditor can answer "why is this ignored" without reading code:
  - `AVD-KSV-0046` — RBAC wildcard on `bootstrap.cluster.x-k8s.io` and `infrastructure.cluster.x-k8s.io` API groups is required by the provider-agnostic design (controller must support any CAPI provider installed in the cluster).
  - `AVD-KSV-0048` — Pod/eviction permissions are required for node drain during machine shutdown.
  - `AVD-KSV-0041` — Secret access is **read-only** (get/list/watch), required to resolve SSH keys and bootstrap-data references.
  - `AVD-KSV-0125` — Registry allow-listing is a cluster-level admission policy (Kyverno / OPA / VAP), not a workload field.
  - `AVD-KSV-01010` — ConfigMap "sensitive content" finding is a false positive; the ConfigMap holds only log-level strings and port numbers.
  - `AVD-DS-0026` — Kubernetes uses `livenessProbe`/`readinessProbe` on `:8081` for health; Dockerfile `HEALTHCHECK` would be dead code, and distroless/Chainguard have no shell to run one.
- `.github/workflows/build.yaml`: `Run Trivy config scan` step now explicitly passes `trivyignores: ./.trivyignore` so the suppressions apply deterministically in CI (Trivy auto-discovers the file by default, but the explicit path documents intent and survives future action-version bumps).

### Why
GitHub Code Scanning was showing 10 open Trivy IaC findings against 5-Spot's deploy manifests and Dockerfiles, including two **error-severity KSV-0046** alerts. Most were real hardening gaps (seccomp, low GID, `:latest` tags, stale RBC image path); a handful (CAPI wildcards, pod eviction, read-only secret access) are load-bearing architectural choices that belong in an explicit suppression list with written justification rather than being fixed away. A local `trivy config` run after the changes now reports **0 misconfigurations** across all 17 scanned files. Separately, this fixes two CLAUDE.md policy violations — the `ghcr.io/RBC/5-spot` image reference and the "RBC Capital Markets" copyright headers — that had been quietly sitting in the tree.

### Impact
- [ ] Breaking change
- [x] Requires cluster rollout (deployment.yaml securityContext + image tag change)
- [x] Config change only (Dockerfile digest pins, `.trivyignore`, CI workflow input)
- [ ] Documentation only

### Operational note
The deployment image is now pinned to `v0.1.0` — the current latest GitHub release. When a new release is cut, either edit `deploy/deployment/deployment.yaml` or override the tag via your kustomize/Helm overlay. Do **not** revert to `:latest`.

---

## [2026-04-19 18:00] - Add OpenVEX generation, validation, and signing pipeline

**Author:** Erick Bourgeois

### Changed
- `.vex/README.md` (new), `.vex/.gitkeep` (new): repository convention for hand-authored per-CVE triage. One TOML file per CVE with `status`, `justification` (enum), `products`, `author`, and `timestamp`. Documents the workflow, the allowed enum values, and what is required per status (`not_affected` → justification; `affected`/`under_investigation` → `action_statement`).
- `tools/validate-vex.sh` (new), `tools/validate_vex.py` (new): shell + Python (`tomllib`, stdlib) validator. Checks per-file schema, enum membership (status, justification), non-empty `products`, RFC-3339 UTC timestamp, and CVE uniqueness across files. Deterministic error output.
- `tools/assemble_openvex.py` (new): assembler that reads `.vex/*.toml` and emits a single OpenVEX v0.2.0 JSON document with a canonical `@id`. Runs `validate_dir` first so malformed input never yields a document. Normalizes TOML datetimes to RFC-3339 UTC `Z`-suffix strings, sorts keys for diffable output.
- `tools/tests/validate-vex-tests.sh` + 18 fixture dirs: 19 cases covering every positive/negative/exception branch (empty dir, missing dir, missing each required field, malformed TOML, bad CVE format, invalid status/justification, empty products, bad timestamp, missing-justification-when-required, missing-action-statement-when-required, duplicate CVE across files, valid-single / valid-multiple / valid-affected).
- `tools/tests/assemble-openvex-tests.sh`: 6 cases covering happy path (file + stdout), JSON validity, validator-gate negative path, CLI argument errors, default-timestamp RFC-3339 shape.
- `.github/workflows/build.yaml`:
  - Added `.vex/**` and `tools/**` to the PR `paths:` filter so changes to either directory trigger the workflow.
  - New PR-only `validate-vex` job runs the validator unit tests and then validates the live `.vex/` directory. Gates PRs touching `.vex/**` or `tools/**`.
  - New release-only `build-vex` job (`needs: [docker, extract-version]`): defensively re-runs the validator, calls `assemble_openvex.py` with a canonical `@id = https://github.com/<repo>/releases/tag/<tag>/vex`, installs pinned `vexctl` and runs `vexctl validate` as a second-opinion check, Cosign-attests the document to both image digests (`cosign attest --type openvex` for Chainguard + Distroless), runs `actions/attest-build-provenance` on the document, and uploads `vex.openvex.json` + `.bundle` as a workflow artifact.
  - `upload-release-assets` now `needs: build-vex`, downloads the `openvex` artifact, copies the document to `release/` and the attestation bundle to `signatures/`, includes both in `checksums.sha256`, and publishes the document as a release asset.
- `docs/src/security/vex.md` (new): user-facing page covering what VEX is, what gets published, how to verify (Cosign + `gh attestation verify`), how to consume (`grype --vex`, `trivy --vex`), and how maintainers add statements. Linked from `docs/src/security/index.md` and `docs/mkdocs.yml` nav.
- `README.md`: Security section now lists OpenVEX under the publication surface with links to `docs/src/security/vex.md` and `.vex/README.md`.
- `.claude/SKILL.md`: `pre-commit-checklist` skill gains a "If preparing a release" block requiring every open Trivy finding to have a statement in `.vex/`, plus explicit commands for the validator and the two test scripts.

### Why
Without VEX, every Trivy CVE surfaced on 5-Spot images re-triages itself at every downstream consumer. Publishing a signed OpenVEX document once — per release, bound to image digests via Cosign, attached to the GitHub Release — pushes the triage decision to exactly one place (`.vex/<cve>.toml`, PR-reviewed) and lets scanners (Grype, Trivy, Harbor) suppress already-triaged findings without repeating the analysis. Keeps 5-Spot honest: only human-authored statements ship, and CI won't emit a document from a malformed source tree.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [x] Config change only (CI/CD workflow + new repo convention)
- [ ] Documentation only

### Operational note
- **vexctl pin:** `build-vex` installs vexctl by downloading a tagged release tarball (`VEXCTL_VERSION: '0.3.0'`). Per the roadmap's "Dependencies to pin" section this should be replaced with SHA-pinned asset verification (`cosign verify-blob`) before the first release actually flows through this workflow; re-pin quarterly with the rest of the signing toolchain.
- **First release after merge:** add at least one hand-authored statement in `.vex/` (even `under_investigation` on a known low-severity CVE) so the signing chain is exercised end-to-end and the release contains a non-empty `vex.openvex.json`.
- **Phase 4 of the roadmap (CycloneDX-VEX co-emission) is intentionally deferred** — kept gated behind an explicit consumer ask per the roadmap's decision gate. Phases 1/2/3/5 are now in place.

---

## [2026-04-19 17:15] - Add Dependabot config for Actions, Cargo, and Docker

**Author:** Erick Bourgeois

### Changed
- `.github/dependabot.yml` (new): SPDX-headered Dependabot v2 config with three ecosystems:
  - **`github-actions`** — weekly Monday 09:00 `America/Toronto`, limit 10 PRs. Grouped update `actions-routine` bundles low-risk first-party + tooling bumps (`actions/*`, `docker/*`, `github/codeql-action`, `softprops/action-gh-release`, `anchore/sbom-action`, `aquasecurity/trivy-action`) into one PR. Security-sensitive actions (`sigstore/*`, `EmbarkStudios/cargo-deny-action`, `ossf/*`) are left ungrouped so each one opens as an individual PR for per-change review. `dtolnay/rust-toolchain` explicitly ignored (branch-tracking ref, no tags — re-pinned manually on quarterly cadence).
  - **`cargo`** — weekly Monday, limit 5 PRs. Groups patch + minor bumps; majors open individually. cargo-audit + cargo-deny CI gates block regressions.
  - **`docker`** — weekly Monday, limit 3 PRs. Picks up `FROM ...@sha256:...` digest bumps once Dockerfiles adopt digest pinning (currently no-op; future-proof).
- Commit-message prefix: `ci` for Actions, `chore` for Cargo and Docker. Labels applied to every PR for easy filtering.

### Why
The previous commit SHA-pinned all external GitHub Actions across the workflows, which is correct for supply-chain hygiene but creates stagnation risk — pins do not auto-update, so known-vulnerable action versions can sit in CI indefinitely. Dependabot solves the stagnation side of the trade-off: it opens a PR per new release with the full changelog diff, and the same CI gates (Semgrep, Trivy, cargo-deny, Scorecard) that guard any other PR guard the bump itself. This closes the loop between "pin everything" and "keep pins current" without inviting humans to cowboy-update SHAs.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [x] Config change only (Dependabot)
- [ ] Documentation only

### Operational note
- **PR volume:** expect 1–3 Actions PRs and 1–2 Cargo PRs per week initially, tapering as the tree stabilizes. The `actions-routine` group keeps the routine noise in one PR.
- **Review rule of thumb:** security-sensitive bumps (sigstore, cargo-deny, ossf) → read the full release notes before merging. Routine Actions group → verify Scorecard Pinned-Dependencies still passes, spot-check the diff for any behaviour change flags, merge if CI is green.
- **Future-proof Docker:** `Dockerfile` and `Dockerfile.chainguard` use tag-based `FROM` lines today. If we move to digest pinning (`FROM image@sha256:...`), Dependabot starts opening PRs for base-image digest bumps automatically — no further config change needed.

---

## [2026-04-19 17:00] - Pin all GitHub Actions to full commit SHAs

**Author:** Erick Bourgeois

### Changed
- `.github/workflows/build.yaml`, `calm.yaml`, `calm-test.yaml`, `docs.yaml` and `.github/actions/prepare-docker-binaries/action.yaml`: replaced every `@<tag>` reference with `@<40-char-sha> # <tag>` for all external actions. 77 replacements across 5 files covering `actions/*`, `docker/*`, `sigstore/cosign-installer`, `anchore/sbom-action`, `aquasecurity/trivy-action`, `EmbarkStudios/cargo-deny-action`, `softprops/action-gh-release`, `github/codeql-action/upload-sarif`, `dtolnay/rust-toolchain`, and all `firestoned/github-actions/*` sub-actions. `scorecard.yaml` was already SHA-pinned and is unchanged.
- `.github/workflows/build.yaml`: the SLSA reusable workflow at `slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@v2.1.0` stays on a semver tag — pinning to SHA would break SLSA provenance verification because `slsa-verifier` validates against approved released versions. Added an inline multi-line comment explaining this so future reviewers don't "fix" it by SHA-pinning.
- `aquasecurity/trivy-action` was previously pinned to the nonexistent tag `0.28.0` (missing `v` prefix — the tag is `v0.28.0`). Bumped to the current latest `v0.35.0` at SHA `57a97c7e7821a5776cebc9bb87c984fa69cba8f1`.
- `dtolnay/rust-toolchain@stable` resolved to SHA `29eef336d9b2848a0b548edc03f92a220660cdb8` with an inline comment noting it is a moving ref that should be re-pinned quarterly to keep Rust current.

### Why
Closes the OpenSSF Scorecard `Pinned-Dependencies` finding across the entire CI surface. Unpinned action tags are a supply-chain risk: a compromised or silently-force-pushed tag would silently execute attacker code in our workflows with access to `GITHUB_TOKEN`, `SNYK_TOKEN`-equivalent secrets, and GHCR push permissions. SHA pinning freezes behaviour to a specific reviewed commit; Dependabot can later send PRs to bump pins with a full diff.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [x] Config change only (CI/CD workflow)
- [ ] Documentation only

### Operational note
- **Dependabot for Actions is strongly recommended** as a follow-up so pins don't rot. Add `.github/dependabot.yml` with a `github-actions` ecosystem entry; Dependabot will open one PR per outdated action SHA with a changelog link.
- **Re-pin cadence:** security-sensitive actions (`sigstore/*`, `ossf/*`, `EmbarkStudios/cargo-deny-action`) should be bumped within a week of a new release. Build tooling (`docker/*`, `actions/*`) is less time-sensitive. `dtolnay/rust-toolchain` should be bumped quarterly to keep the Rust toolchain current.
- All pin comments include the resolved semver tag (e.g. `# v4.3.1`) so a human reader can see at a glance what version is in use without clicking through to the SHA.

---

## [2026-04-19 17:30] - Phase 5 docs finish: regenerate stale api.md (mdBook source)

**Author:** Erick Bourgeois

### Changed
- `docs/src/reference/api.md`: regenerated via `make crddoc`. Now includes the Phase 1 status-schema additions — `providerID`, extended `nodeRef` (with `uid`, `apiVersion`, `kind` alongside `name`), `machineRef`, `bootstrapRef`, `infrastructureRef`, and refreshed condition/observedGeneration docstrings. Previously this file was stale; Phase 1's regen wrote only to `docs/reference/api.md` (a legacy path referenced by `.claude/SKILL.md`), not to the mdBook source path that actually renders on the doc site (`docs/src/reference/api.md`, per the `Makefile` `crddoc` target).

### Why
The Phase 5 checklist in the roadmap required "`docs/src/reference/api.md` — regenerated by `regen-api-docs` skill." Spot-checking after the earlier Phase 5 commit revealed that the canonical Makefile target output path and the SKILL.md instruction disagreed, and the mdBook source copy was still on the pre-Phase-1 schema. Regenerating closes that gap so the doc site reflects what consumers actually get on `kubectl get scheduledmachine -o yaml`.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [x] Documentation only

### Operational note
- **SKILL.md drift:** `.claude/SKILL.md` currently instructs `cargo run --bin crddoc > docs/reference/api.md`, which writes to the wrong path. The `Makefile` `crddoc` target is correct. A small follow-up should reconcile the skill instructions with the Makefile so future regens aren't silently written to the wrong path.

---

## [2026-04-19 16:30] - Phase 5 docs: watch-topology diagram in architecture.md

**Author:** Erick Bourgeois

### Changed
- `docs/src/concepts/architecture.md`: New top-level section "Watch Topology" with a Mermaid diagram showing the primary `ScheduledMachine` watch plus the two new secondary watches on CAPI `Machine` (label-filtered) and core `Node` (cluster-wide) feeding into pure reverse-mapper functions (`machine_to_scheduled_machine`, `node_to_scheduled_machines`) that enqueue the owning `ScheduledMachine`. Updated the "Controller" component-detail bullet list to enumerate the three watches.
- `docs/roadmaps/5spot-event-driven-watches-and-status-enrichment.md` (copy at `~/dev/roadmaps/`): status header updated to `Phase 5 ✅` — roadmap closed.

### Why
Closes Phase 5 of the event-driven-watches roadmap. Until this commit, the concepts doc described a controller that only watched its own CR; readers had no way to understand why a Node cordon now triggers an immediate reconcile instead of waiting for the next requeue. The watch-topology diagram makes the event-driven claim concrete and auditable.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [x] Documentation only

---

## [2026-04-19 16:00] - Replace misleading Snyk claim with real OSS security tooling (Semgrep + Trivy config + cargo-deny)

**Author:** Erick Bourgeois

### Changed
- `.github/workflows/build.yaml`: Three new PR + push-to-main jobs (release events skip these — the release pipeline relies on scans that already ran on main):
  - `semgrep-sast` — runs `semgrep scan` in the `returntocorp/semgrep` container with the community `p/rust`, `p/security-audit`, `p/secrets`, and `p/owasp-top-ten` rulesets; `--metrics=off`; uploads SARIF to Code Scanning under the `semgrep` category. No token required.
  - `iac-scan` — `aquasecurity/trivy-action@0.28.0` with `scan-type: config` against the repo root (picks up `deploy/**/*.yaml` + `Dockerfile*` + `Dockerfile.chainguard`); uploads SARIF under the `trivy-iac` category.
  - `cargo-deny` — `EmbarkStudios/cargo-deny-action@v2` running `check --all-features` against the new `deny.toml`.
- `deny.toml` (new, repo root): SPDX-headered cargo-deny config. License allow-list covers Apache-2.0, MIT, BSD-2/3-Clause, ISC, Unicode-DFS-2016, Unicode-3.0, Zlib, CC0-1.0, MPL-2.0, OpenSSL. Advisories: `yanked = "deny"`. Bans: `multiple-versions = "warn"` (to avoid breaking CI on transitive pulls we do not control), `wildcards = "deny"`. Sources: only official crates.io registry, no unknown git URLs.
- `README.md`: Removed the misleading `Snyk` SAST badge (we never actually ran Snyk). Added badges for **Semgrep**, **Trivy** (Container + IaC), **cargo-deny**, **cargo-audit**, **Cosign**, and **SLSA** — each reflecting tooling that actually runs in the pipeline.
- `docs/architecture/calm/architecture.json`: The `supply-chain-scanning` control description now reads: "Repository is scanned by Semgrep OSS (SAST), Trivy (container image + IaC config), cargo-audit + cargo-deny (RustSec advisories, license allow-list, source restrictions), and Gitleaks (secrets); OpenSSF Scorecard publishes supply-chain posture; SPDX license identifiers on source files." Old wording referenced Snyk and Aqua which were never actually wired up.

### Why
The README's Security & Compliance section and the CALM architecture JSON both claimed "Snyk (SAST)" but `rg -i snyk` returned zero hits outside those two strings — the repo has never had Snyk configured. That is a compliance posture lie for a project in a regulated banking context. This commit closes the three real gaps with free OSS tooling: SAST (Semgrep), IaC misconfig scanning (Trivy config), and dependency license/advisory enforcement (cargo-deny). All three are token-free, run on every PR, and write SARIF to the existing Code Scanning dashboard — same surface consumers already use for Scorecard and Trivy container results.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [x] Config change only (CI/CD workflow + cargo-deny config)
- [ ] Documentation only

### Operational note
- **First run of `cargo-deny` may fail** if an existing transitive dep carries a license not in the allow-list. Treat the first failure as signal: either add the license to `deny.toml` (with a justification comment) or file an issue to swap the dep. Do not weaken `unknown-registry = "deny"` — that is a supply-chain boundary, not a nuisance.
- **Semgrep findings initially expected**: the first run will surface Rust and OWASP-style findings that were never triaged. Review in Code Scanning → filter by tool `semgrep` → triage-or-suppress with justification (Semgrep supports `// nosemgrep: rule-id — reason` inline comments).
- **Trivy IaC findings**: scope is the `deploy/` manifests plus both Dockerfiles. Findings you cannot fix (e.g., a base-image constraint) can be suppressed via a `.trivyignore` file at repo root with a comment.
- All three jobs run as `needs: [verify-commits]` so they parallelize with `extract-version`/`build` and do not serialize the critical path.

---

## [2026-04-19 15:45] - Add OpenSSF Scorecard badge to README

**Author:** Erick Bourgeois

### Changed
- `README.md`: Added `[![OpenSSF Scorecard]...]` badge as the first entry in the "Security & Compliance" section, linking to `https://scorecard.dev/viewer/?uri=github.com/finos/5-spot`.

### Why
The `scorecard.yaml` workflow already publishes results to the OpenSSF REST API (`api.securityscorecards.dev`) and to GitHub Code Scanning, but without a visible badge the score is invisible to anyone reading the README. The badge is the canonical consumer-facing signal that the project runs Scorecard and auto-updates with each `scorecard.yaml` run.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [x] Documentation only

---

## [2026-04-19 15:30] - Sign binaries and generate SLSA provenance on push-to-main (not only on release)

**Author:** Erick Bourgeois

### Changed
- `.github/workflows/build.yaml`:
  - `sign-artifacts` job: `if` flipped from `github.event_name == 'release'` to `!= 'pull_request'`. Added `attestations: write` and `contents: read` permissions. New step runs `actions/attest-build-provenance@v2` on each signed tarball (amd64 + arm64) — GitHub-native attestation in addition to the Cosign signature.
  - `generate-provenance-subjects` job: `if` flipped to `!= 'pull_request'` so SLSA subject hashes are computed on push-to-main too.
  - `slsa-provenance` job: `if` flipped to `!= 'pull_request'`; the reusable SLSA generator now runs on every push-to-main. `upload-assets` parameterised to `${{ github.event_name == 'release' }}` so the `.intoto.jsonl` only attaches to the GitHub Release on release events; on push-to-main it lands as a workflow artifact (verifiable via `slsa-verifier` or `gh attestation verify`).
- `upload-release-assets` unchanged — stays `if: github.event_name == 'release'`; its `needs:` on `sign-artifacts`/`slsa-provenance` resolves cleanly whether those jobs ran (push, release) or were skipped (PR).

### Why
Every merge to `main` produces a buildable, distributable artifact — it should carry the same supply-chain assurances as a release. With these changes, a `main` build now has: Cosign keyless signature on the container image (already in place), GitHub Artifact Attestation on the container image (already in place), Cosign signature on each binary tarball (new on `main`), GitHub Artifact Attestation on each binary tarball (new), and SLSA Level 3 provenance for the binaries (new on `main`). Consumers pulling a `main-YYYY-MM-DD` build can now cryptographically verify it the same way they would a release.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [x] Config change only (CI/CD workflow)
- [ ] Documentation only

### Operational note
- Push-to-main runs now take longer (extra ~3–5 min for the SLSA generator + attestations). Expected trade-off.
- `sign-artifacts` signed-tarball artifact retention is the repo default (90 days) — long enough for verification and audit, well under the release-asset lifetime.
- Binary SLSA generator dependency pinned at `@v2.1.0` (unchanged) — verify this is the latest patched version periodically.

---

## [2026-04-19 15:00] - Harden OpenSSF Scorecard workflow: SPDX header + pin remaining action

**Author:** Erick Bourgeois

### Changed
- `.github/workflows/scorecard.yaml`: Prepended the project's standard two-line SPDX/Copyright header (`Copyright (c) 2025 Erick Bourgeois, firestoned` + `SPDX-License-Identifier: Apache-2.0`) to match every other workflow in `.github/workflows/`. Pinned the previously unpinned `github/codeql-action/upload-sarif@v3` to the full commit SHA for v3.35.2 (`ce64ddcb0d8d890d2df4a9d1c04ff297367dea2a`) with an inline version comment — brings the last step into line with the other three actions in the file which were already SHA-pinned.

### Why
OpenSSF Scorecard's own `Pinned-Dependencies` check flags float-tagged actions (`@v3`) as a supply-chain risk, so the scorecard workflow itself should score well on that check. The SPDX header is the repo-wide convention; every other workflow has it. Both changes are compliance housekeeping — no behavior change.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [x] Config change only
- [ ] Documentation only

---

## [2026-04-19 14:00] - Phases 3 + 4: Event-driven `.watches()` on CAPI Machine and Node

**Author:** Erick Bourgeois

### Changed
- `src/reconcilers/helpers.rs`: Added two pure mapper functions used as secondary-watch reverse-maps on the Controller builder:
  - `machine_to_scheduled_machine(&DynamicObject) -> Vec<ObjectRef<ScheduledMachine>>` — reads the `5spot.eribourg.dev/scheduled-machine` label (constant `LABEL_SCHEDULED_MACHINE`) and emits at most one `ObjectRef`. Guards against missing/empty/whitespace label values, missing namespace, and imposter labels with similar prefixes.
  - `node_to_scheduled_machines<'a, I: IntoIterator<Item = &'a ScheduledMachine>>(&Node, I) -> Vec<ObjectRef<ScheduledMachine>>` — returns all SMs whose `status.nodeRef.name` matches `node.metadata.name`. Returns multiple refs on conflict so the reconciler can surface the issue. O(N) per Node event; documented as acceptable for current scale.
- `src/reconcilers/mod.rs`: Re-exports both mappers alongside the existing `error_policy`, `evaluate_schedule`, `should_process_resource`.
- `src/main.rs`: Extended the `Controller` builder with `.watches_with(...)` on CAPI `Machine` (`ApiResource` from GVK, label-filtered `watcher::Config`) and `.watches(...)` on `Node`. The Node mapper closure captures a clone of `controller.store()` (`reflector::Store<ScheduledMachine>`) and calls `state()` on each event to avoid out-of-band API lists.
- `src/reconcilers/helpers_tests.rs`: Added 14 new unit tests — 7 for `machine_to_scheduled_machine` (positive match, missing label, no labels, empty value, whitespace value, missing namespace, wrong-prefix imposter) and 7 for `node_to_scheduled_machines` (single match, multi-match conflict, no match, empty list, SM-without-status, Node-without-name, empty `nodeRef.name`). Covers positive, negative, and exception paths per the project's 100%-coverage rule.

### Why
Closes roadmap Phases 3 + 4 of `event-driven-watches-and-status-enrichment.md`. The `ScheduledMachine` controller previously only watched its own CR — all downstream state was polled during reconciles, which CLAUDE.md explicitly forbids ("ALWAYS use event-driven programming… as opposed to polling"). With these two watches, CAPI setting `status.nodeRef` or a Node being drained now enqueues an immediate reconcile via kube-rs's watch stream (<1s debounce) instead of waiting for the next periodic requeue. The label-based reverse map on Machines uses the label we already stamp on every child — zero new ownership semantics, no `controller: true` owner references introduced, no contention with CAPI's own controllers.

### Impact
- [ ] Breaking change
- [x] Requires cluster rollout — controller now opens additional watches (CAPI `Machine` with label selector; `Node` cluster-wide)
- [ ] Config change only
- [ ] Documentation only

### Operational note
- RBAC: the controller already has `get`/`list`/`watch` on CAPI `Machine` (used for drain lookup) and needs `list`/`watch` on core `Node`. Verify `get,list,watch` on `nodes` is present in the ClusterRole before rollout; if not, add it in the same rollout.
- Observability: the existing `reconcile_queue_depth` metric will show enqueues driven by Machine and Node events in addition to SM events.

---

## [2026-04-19 10:00] - Phase 2: Close test-coverage gap for async CAPI helpers

**Author:** Erick Bourgeois

### Changed
- `src/reconcilers/helpers_tests.rs`: Added 14 `tower-test`-backed unit tests covering positive, negative, and exception paths for the three async helpers introduced in the previous Phase 2 entry:
  - `fetch_capi_machine` — 200 returns `Some`, 404 returns `Ok(None)`, 500 and 403 map to `CapiError`.
  - `patch_machine_refs_status` — both-fields patch asserts full body shape, single-field patches assert the other key is **omitted** (not null) so merge-patch never clears existing values, both-`None` case asserts **zero HTTP traffic**, 500 and 404 map to `KubeError`.
  - `get_node_from_machine` — success returns the node name, machine-404 and nodeRef-missing both return `Ok(None)`, 500 propagates as `CapiError`.

### Why
Per durable user guidance: every function (public **and** private) must have unit tests covering the happy path, negative cases, and exception/error paths. The original Phase 2 change landed with tests only for the pure `extract_machine_refs` function — the three async helpers, which actually touch the Kubernetes API surface, were undertested. This entry closes that gap so the coverage floor now matches the project rule (not just CLAUDE.md's "public function" minimum).

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [x] Documentation only (test-only change — no runtime behaviour modified)

---

## [2026-04-18 12:00] - Phase 2: Populate providerID and nodeRef from CAPI Machine

**Author:** Erick Bourgeois

### Changed
- `src/reconcilers/helpers.rs`: Added three new helpers — `extract_machine_refs` (pure function pulling `providerID` + full `NodeRef` out of a CAPI Machine `DynamicObject`), `fetch_capi_machine` (typed 404 → `Ok(None)` wrapper around the dynamic GET), and `patch_machine_refs_status` (merge-patches both fields onto `ScheduledMachine.status`). Refactored `get_node_from_machine` to delegate to the first two helpers for the drain path.
- `src/reconcilers/scheduled_machine.rs`: `handle_active_phase` happy path now fetches the CAPI Machine, extracts refs, and patches them onto the SM status. Failures are logged and ignored — status enrichment must never block reconciliation.
- `src/reconcilers/helpers_tests.rs`: Added 6 TDD cases for `extract_machine_refs` covering fully populated, empty, providerID-only, nodeRef-without-uid, incomplete-nodeRef-returns-None, and malformed-providerID-ignored.

### Why
Phase 2 of the event-driven watches + status enrichment roadmap. The schema landed in Phase 1; this change actually populates the new fields every time the reconciler visits an active machine, so `kubectl get sm -o jsonpath='{.status.providerID}{"\t"}{.status.nodeRef.name}'` returns meaningful values.

### Impact
- [ ] Breaking change
- [x] Requires cluster rollout — controller image carries the new reconcile logic
- [ ] Config change only
- [ ] Documentation only

---

## [2026-04-16 17:00] - Phase 1: Enrich ScheduledMachine status with providerID and full nodeRef

**Author:** Erick Bourgeois

### Changed
- `src/crd.rs`: Added `provider_id: Option<String>` (serialized as `providerID`) to `ScheduledMachineStatus`. Replaced the thin `LocalObjectReference { name }` node reference with a new `NodeRef { apiVersion, kind, name, uid }` struct mirroring CAPI's `Machine.status.nodeRef`. Removed the now-unused `LocalObjectReference` type.
- `src/crd_tests.rs`: Added 6 TDD cases covering providerID round-trip, full `nodeRef` deserialization, optional `uid`, serialization omission, old-shape rejection, and `NodeRef` round-trip.
- `src/bin/crddoc.rs`: Documented new `providerID` and `nodeRef` status fields.
- `deploy/crds/scheduledmachine.yaml`: Regenerated from the updated Rust types.
- `docs/reference/api.md`: Regenerated to reflect new status schema.

### Why
Phase 1 of the event-driven watches + status enrichment roadmap. Surfacing `providerID` and a full Node reference (with UID) on `ScheduledMachine.status` lets operators correlate a scheduled machine to a specific VM and Node from `kubectl get sm -o jsonpath=...`, without manual lookups across CAPI Machines and the Node API. This is the schema foundation that Phase 2 (reconciler populates the fields) and Phases 3–4 (event-driven watches on CAPI Machine and Node) build upon.

### Impact
- [x] Breaking change — `status.nodeRef` shape changed from `{ name }` to `{ apiVersion, kind, name, uid }`. Existing CRs with the old shape must clear `status.nodeRef` before rollout, or the controller will report deserialization errors on that field.
- [x] Requires cluster rollout — CRD must be re-applied alongside the new controller image.
- [ ] Config change only
- [ ] Documentation only

---

## [2026-04-17] - Bump base image to cc-debian13 and fix GLIBC_2.39 crash (issue #17)

**Author:** Daniel Guns

### Changed
- `Dockerfile`: Base image bumped from `gcr.io/distroless/cc-debian12:nonroot` (glibc 2.36) to `gcr.io/distroless/cc-debian13:nonroot` (glibc 2.41)
- `.github/workflows/build.yaml`: Pinned Linux x86_64 runner from `ubuntu-latest` to `ubuntu-24.04` for CI stability
- `Cargo.lock`: Updated transitive dependency `rustls-webpki` from `0.103.11` to `0.103.12`

### Why
`ubuntu-latest` now resolves to Ubuntu 24.04 (glibc 2.39), producing binaries that require `GLIBC_2.39` at runtime. The previous `cc-debian12` base only provides glibc 2.36, causing a hard crash at container startup. Bumping to `cc-debian13` (glibc 2.41) resolves the mismatch. Runners are explicitly pinned to `ubuntu-24.04` so CI doesn't break silently when `ubuntu-latest` moves to 26.04. Additionally, two CVEs in `rustls-webpki 0.103.11` (RUSTSEC-2026-0098, RUSTSEC-2026-0099) were patched by bumping to `0.103.12`. Fixes issue #17.

### Impact
- [ ] Breaking change
- [x] Requires cluster rollout — new base image
- [ ] Config change only
- [ ] Documentation only

---

## [2026-04-10 08:50] - Extend Cosign image signing to main-branch pushes

**Author:** Erick Bourgeois

### Changed
- `.github/workflows/build.yaml`: Changed Cosign signing step condition from `github.event_name == 'release'` to `github.event_name != 'pull_request'`

### Why
Main-branch images are tagged `latest` and `main-YYYY-MM-DD` and may be deployed to staging. Signing them allows `cosign verify` to work on staging images, not just production releases. PR images remain unsigned — they are ephemeral, tagged `pr-{number}`, and not deployed anywhere.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [ ] Documentation only

---

## [2026-04-10 08:45] - Fix attest job: add GHCR login before push-to-registry

**Author:** Erick Bourgeois

### Changed
- `.github/workflows/build.yaml`: Added `docker/login-action@v3` step to the `attest` job before `actions/attest-build-provenance@v2`

### Why
`push-to-registry: true` in `actions/attest-build-provenance` pushes the attestation bundle as an OCI artifact to GHCR, which requires registry credentials. Each job runs in a fresh environment — the Docker login performed by `firestoned/github-actions/docker/setup-docker` in the `docker` job does not carry over to the `attest` job.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [ ] Documentation only

---

## [2026-04-10 08:30] - Consolidate pr.yaml, main.yaml, release.yaml into single build.yaml

**Author:** Erick Bourgeois

### Changed
- `.github/workflows/build.yaml`: New consolidated workflow replacing the three separate files; triggers on `pull_request`, `push` to main, and `release: published`; uses `if:` at job and step level to gate event-specific behaviour
- `.github/workflows/pr.yaml`: Deleted
- `.github/workflows/main.yaml`: Deleted
- `.github/workflows/release.yaml`: Deleted

### Why
Three workflows shared the same build matrix, env vars, and most job logic, requiring the same fix to be applied in three places (e.g. the linker override, the attest job). A single file is easier to maintain and gives a complete picture of CI behaviour in one place.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [ ] Documentation only

> **Key design decisions:**
> - `extract-version` serves as the quality gate: it runs after all checks that apply to the current event (`verify-commits`, `license-check`, `format`) and all downstream jobs depend on it.
> - Docker metadata uses three separate `docker/metadata-action` steps gated by `if:`; `docker/build-push-action` concatenates all outputs and filters empty lines.
> - Cosign signing, Docker SBOM generation, `sign-artifacts`, SLSA provenance, and `upload-release-assets` are guarded by `if: github.event_name == 'release'`.
> - `test` and `format`/`clippy` are guarded by `if: github.event_name == 'pull_request'`.
> - `trivy` is guarded by `if: github.event_name != 'pull_request'`.
> - Artifact retention: PR/push = 1 day (two upload steps); release = default.

---

## [2026-04-10 07:45] - Add GitHub artifact attestation job to all three CI/CD workflows

**Author:** Erick Bourgeois

### Changed
- `.github/workflows/pr.yaml`: Added `id: docker_build` to docker build step; added `outputs:` block to docker job (per-variant digests); added `export-digest` step; added `attest` job depending on `docker` and `extract-version`
- `.github/workflows/main.yaml`: Same additions to docker job and new `attest` job
- `.github/workflows/release.yaml`: Same additions to `docker-release` job and new `attest` job (depends on `docker-release`)

### Why
GitHub's `actions/attest-build-provenance` generates a signed SLSA provenance attestation stored natively in GitHub Artifact Attestations and optionally pushed to the OCI registry alongside the image. This is queryable with `gh attestation verify` and complements the existing Cosign signatures in the release workflow. Requires the matrix digest-export pattern to pass per-image digests from a matrix job to a downstream job.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [x] Documentation only

---

## [2026-04-08 17:30] - Add documentation build and GitHub Pages deployment workflow

**Author:** Erick Bourgeois

### Changed
- `.github/workflows/docs.yaml`: New workflow — builds MkDocs documentation (including `make docs` which runs `cargo run --bin crddoc`) and deploys to GitHub Pages on push to main; runs link checks on PRs

### Why
The project has a full MkDocs documentation site under `docs/` but no automated build or publishing pipeline. This workflow closes that gap by building on every relevant change, checking for broken links on PRs, and publishing to GitHub Pages on every merge to main.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [x] Documentation only

> **Note:** GitHub Pages must be enabled in the repository settings (`Settings → Pages → Source: GitHub Actions`) for the deploy job to succeed. The `poetry.lock` file should be committed after the first `poetry install` run to improve cache efficiency and build reproducibility.

---

## [2026-04-09 11:00] - Fix CI linker error caused by .cargo/config.toml on Linux runners

**Author:** Erick Bourgeois

### Changed
- `.github/workflows/pr.yaml`: Added `CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER: cc` and `CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER: cc` to the top-level `env:` block
- `.github/workflows/main.yaml`: Same
- `.github/workflows/release.yaml`: Same

### Why
`.cargo/config.toml` specifies `linker = "x86_64-unknown-linux-gnu-gcc"` and `linker = "aarch64-unknown-linux-gnu-gcc"` for the respective targets — Homebrew cross-compilers needed when a macOS developer uses `cargo build --target <linux-triple>` locally (the Makefile fallback path). On Linux CI runners, `cargo build --release` with no explicit target resolves to the **native** triple (e.g., `x86_64-unknown-linux-gnu` on `ubuntu-latest`), which picks up the same override and fails because the cross-compiler is not installed on GitHub Actions runners. `CARGO_TARGET_*_LINKER` environment variables take precedence over `config.toml`, restoring `cc` (the system linker) in CI without modifying the config file.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [ ] Documentation only

---

## [2026-04-09 10:00] - Replace cross-compilation with native cargo builds in CI

**Author:** Erick Bourgeois

### Changed
- `.github/workflows/pr.yaml`: Replaced `firestoned/github-actions/rust/setup-rust-build@v1.3.6` + `build-binary@v1.3.6` + `generate-sbom@v1.3.6` with `dtolnay/rust-toolchain@stable` + `cargo build --release` + `cargo-cyclonedx`; switched ARM64 build to `ubuntu-24.04-arm` native runner; updated artifact paths from `target/$target/release/` to `target/release/`; fixed `license-id: "MIT"` → `"Apache-2.0"`
- `.github/workflows/main.yaml`: Same build and license-check changes
- `.github/workflows/release.yaml`: Same build and license-check changes; replaced `setup-rust-build` in `package-deploy-manifests` job with `dtolnay/rust-toolchain@stable`

### Why
The `firestoned/github-actions/rust/build-binary@v1.3.6` action internally sets `-C linker=x86_64-unknown-linux-gnu-gcc`, which is not installed on GitHub Actions `ubuntu-latest` runners, causing all build jobs to fail. Native `cargo build --release` on arch-appropriate runners eliminates cross-compilation entirely. License-id was stale after the FINOS Apache-2.0 migration.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [x] Documentation only

---

## [2026-04-09 03:00] - Phase 2 (P2-4): leader election via kube-lease-manager

**Author:** Erick Bourgeois

### Changed
- `Cargo.toml`: Added `kube-lease-manager = "0.11"` dependency
- `src/constants.rs`: Added `DEFAULT_LEASE_NAME`, `DEFAULT_LEASE_DURATION_SECS`, `DEFAULT_LEASE_RENEW_DEADLINE_SECS`, `DEFAULT_LEASE_RETRY_PERIOD_SECS`, `DEFAULT_LEASE_GRACE_SECS`, `DEFAULT_LEASE_NAMESPACE` constants
- `src/reconcilers/scheduled_machine.rs`: Added `is_leader: Arc<AtomicBool>` to `Context` (defaults to `true` for backward-compatible single-instance mode); added leader guard in `reconcile_guarded` — non-leaders return `Action::await_change()` immediately
- `src/main.rs`: Added `enable_leader_election`, `lease_name`, `lease_namespace`, `lease_duration_secs`, `lease_renew_deadline_secs` CLI args; when enabled, sets `is_leader = false` at startup and spawns a background `kube-lease-manager` task that flips `is_leader` on acquisition/loss
- `deploy/deployment/deployment.yaml`: Fixed `POD_NAME` → `CONTROLLER_POD_NAME` env var (aligns with `Context::new` and leader election holder identity)
- `src/reconcilers/scheduled_machine_tests.rs`: Added 2 TDD tests — `test_context_new_defaults_is_leader_to_true` and `test_reconcile_guarded_awaits_change_when_not_leader`
- `docs/src/operations/configuration.md`: Added all leader election env vars, CLI args, Leader Election section, Lease RBAC rules

### Why
Basel III HA (P2-4): a single-replica controller is a single point of failure. With `ENABLE_LEADER_ELECTION=true` and `replicas: 2`, only the lease holder reconciles resources. Standby replicas react within one `LEASE_DURATION_SECONDS` window on leader failure. `Context::is_leader` defaults to `true` so existing single-replica deployments continue without any config change.

### Impact
- [ ] Breaking change
- [x] Requires cluster rollout — set `ENABLE_LEADER_ELECTION=true` and `replicas: 2`; RBAC for `leases` already in `clusterrole.yaml`
- [ ] Config change only
- [ ] Documentation only

---

## [2026-04-09 02:00] - Add SPDX license headers to all .github YAML files

**Author:** Erick Bourgeois

### Changed
- `.github/ISSUE_TEMPLATE/bug_report.yml`: Added `# Copyright (c) 2025 Erick Bourgeois, finos` + `# SPDX-License-Identifier: Apache-2.0` header
- `.github/ISSUE_TEMPLATE/feature_request.yml`: Same
- `.github/ISSUE_TEMPLATE/meeting_minutes.yml`: Same
- `.github/ISSUE_TEMPLATE/support_question.yml`: Same

### Why
Supply-chain provenance and automated license scanning (NIST SA-4) require SPDX headers on all project-owned files. The three workflow files and both composite actions already had headers from P2-10; these four issue templates were the remaining `.github/` YAML files without them. `dco.yml` was intentionally left untouched — it is managed by FINOS and carries an explicit "Do not edit" notice.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [x] Documentation only

---

## [2026-04-09 01:00] - Phase 2 (P2-5): exponential back-off in error policy

**Author:** Erick Bourgeois

### Changed
- `src/constants.rs`: Added `MAX_RECONCILE_RETRIES: u32 = 10` constant
- `src/reconcilers/scheduled_machine.rs`: Added `retry_counts: Arc<Mutex<HashMap<String, u32>>>` to `Context`; updated `Context::new` to initialise it; updated `reconcile_guarded` to clear the retry count on successful reconciliation
- `src/reconcilers/helpers.rs`: Added `compute_backoff_secs(retry_count: u32) -> u64` (pure, capped exponential); replaced fixed-delay `error_policy` with retry-count-aware implementation that increments the per-resource counter and computes `ERROR_REQUEUE_SECS * 2^n` capped at `MAX_BACKOFF_SECS`
- `src/reconcilers/helpers_tests.rs`: Added 5 TDD tests for `compute_backoff_secs` (base, doubling, cap at retry 4, cap at MAX_RECONCILE_RETRIES, large count)

### Why
Basel III HA resilience (P2-5): a fixed 30 s retry interval can cause thundering-herd pressure when many resources fail simultaneously. Bounded exponential back-off distributes retry load while ensuring eventual recovery. Retry counts are cleared on success so transient failures do not permanently elevate delay. Aligns with NIST SI-2 flaw remediation by limiting retry storms.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [ ] Documentation only

---

## [2026-04-09 00:00] - Phase 2 (P2-3/P2-7): reconciliation correlation IDs and Condition status enum

**Author:** Erick Bourgeois

### Changed
- `src/reconcilers/scheduled_machine.rs`: Added `generate_reconcile_id()` — derives a short correlation ID from the resource's UID last segment + nanosecond hex timestamp; refactored `reconcile_scheduled_machine` into `reconcile_guarded` wrapped in a `tracing::info_span!` carrying `reconcile_id`, `resource`, and `namespace` — every log line in a reconciliation now carries these fields in JSON output (NIST AU-3 / SOX §404 P2-3)
- `src/reconcilers/scheduled_machine_tests.rs`: Added 5 TDD tests for `generate_reconcile_id()` covering non-empty output, UID-last-segment prefix, hex timestamp suffix, unknown-fallback when no UID, and uniqueness across calls
- `src/crd.rs`: Added `condition_status_schema()` and wired it to `Condition.status` via `#[schemars(schema_with = "...")]` — constrains the CRD field to `enum: [True, False, Unknown]` (NIST CM-5 / P2-7)
- `src/crd_tests.rs`: Added 5 TDD tests for `Condition.status` schema enum: constraint exists, all three values present, and runtime `Condition::new()` still accepts string status unchanged
- `deploy/crds/scheduledmachine.yaml`: Regenerated — `Condition.status` now has `enum: [True, False, Unknown]` in the CRD OpenAPI schema
- `docs/reference/api.md`: Regenerated to reflect schema change

### Why
- **P2-3**: Every reconciliation now emits a unique `reconcile_id` on all log lines via a `tracing` span, enabling full end-to-end correlation in a SIEM or log aggregation platform. Closes the NIST AU-3 / SOX §404 correlation ID gap.
- **P2-7**: The `Condition.status` field previously accepted any string; the CRD schema now enforces the Kubernetes-standard `True`/`False`/`Unknown` enum as required by NIST CM-5 configuration change control. Runtime behaviour is unchanged — the constraint is schema-only.

### Impact
- [ ] Breaking change
- [x] Requires cluster rollout — CRD must be reapplied (`kubectl apply -f deploy/crds/scheduledmachine.yaml`); existing CRs with valid status values are unaffected
- [ ] Config change only
- [ ] Documentation only

---

## [2026-04-08 15:00] - Add Security section to MkDocs with Admission Validation guide

**Author:** Erick Bourgeois

### Added
- `docs/src/security/index.md`: New security section landing page — security posture at a glance table, compliance mapping summary, and links to sub-pages
- `docs/src/security/admission-validation.md`: Comprehensive user-facing guide for the `ValidatingAdmissionPolicy` covering: VAP vs. webhook comparison table, Mermaid admission flow sequence diagram, full 13-rule reference table with per-rule detail and examples, deployment instructions, rollout strategy (Audit → Deny → AuditAndDeny), four concrete kubectl test examples, namespace scoping guidance, and Kubernetes version compatibility table
- `docs/mkdocs.yml`: Added `Security` top-level nav section (between Advanced Topics and Developer Guide) containing Overview, Admission Validation, and Threat Model pages

### Why
The `ValidatingAdmissionPolicy` deployed in the previous entry had no user-facing documentation. Operators need to know what is validated, how to deploy it, how to do a safe rollout, and how to test it. The new Security section also surfaces the threat model in the main navigation — previously it existed only in the repo but was not reachable from the docs site.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [x] Documentation only

---

## [2026-04-08 14:00] - Phase 2 (P2-6/P2-8/P2-9/P2-10): eviction correctness, JSON logging, supply-chain provenance

**Author:** Erick Bourgeois

### Changed
- `src/reconcilers/helpers.rs`: Fixed P2-6 — `evict_pod` 429 PDB-blocked arm now returns `Err(ReconcilerError::CapiError(...))` instead of silently returning `Ok(())`; log level raised from `info` to `warn`; doc comment updated to remove the incorrect "429 is not an error" statement
- `src/reconcilers/helpers_tests.rs`: Added 5 TDD mock API tests for `evict_pod` covering: success (200), already-deleted (404 → Ok), PDB-blocked (429 → CapiError), server error (500 → CapiError), and forbidden (403 → CapiError)
- `src/main.rs`: Wired P2-8 — added `--log-format` CLI arg mapped to `RUST_LOG_FORMAT` env var (default `"json"`); tracing subscriber now uses `.json()` layer for `json` and plain text layer for `text`/anything else
- `deploy/deployment/deployment.yaml`: Changed `RUST_LOG_FORMAT` default from `"text"` to `"json"` so production pods emit structured JSON for SIEM ingestion
- `src/**/*.rs` (all 18 files): Added P2-10 SPDX supply-chain provenance headers to every Rust source file:
  ```
  // Copyright (c) 2025 Erick Bourgeois, RBC Capital Markets
  // SPDX-License-Identifier: Apache-2.0
  ```

### Why
- **P2-6**: A PDB-blocked eviction (HTTP 429) was silently treated as success, causing the drain loop to believe the pod was evicted when it wasn't — a data-integrity bug that could leave a node non-empty. Now propagated as `CapiError` so the caller can decide to retry or abort.
- **P2-8**: Structured JSON logging is required for SIEM ingestion and NIST AU-3 compliance; `text` format was only appropriate for local development.
- **P2-9**: `cargo-audit 0.22.0` was already running via `firestoned/github-actions/rust/security-scan@v1.3.6` on all PRs and main — no code change required, marked ✅ in roadmap.
- **P2-10**: SPDX headers enable automated license scanning and supply-chain provenance tracking per NIST SA-4.

### Impact
- [ ] Breaking change
- [x] Requires cluster rollout — `RUST_LOG_FORMAT=json` default; existing log parsers expecting plain text must be updated
- [ ] Config change only
- [ ] Documentation only

---

## [2026-04-08 13:00] - Add ValidatingAdmissionPolicy for ScheduledMachine (NIST CM-5)

**Author:** Erick Bourgeois

### Added
- `deploy/admission/validatingadmissionpolicy.yaml`: `ValidatingAdmissionPolicy` with 13 CEL validation rules covering: `clusterName` non-empty; `gracefulShutdownTimeout`/`nodeDrainTimeout` duration format (`^\d+[smh]$`); `cron` XOR `daysOfWeek`/`hoursOfDay` mutual exclusivity; `daysOfWeek` day-name/range item format; `hoursOfDay` hour/range item format; `bootstrapSpec`/`infrastructureSpec` apiVersion namespaced-group requirement; bootstrap/infrastructure provider API group allowlist (mirrors `ALLOWED_BOOTSTRAP_API_GROUPS` / `ALLOWED_INFRASTRUCTURE_API_GROUPS` in `src/constants.rs`); `bootstrapSpec.kind`/`infrastructureSpec.kind` non-empty
- `deploy/admission/validatingadmissionpolicybinding.yaml`: `ValidatingAdmissionPolicyBinding` with `validationActions: [Deny]` applied cluster-wide

### Changed
- `docs/roadmaps/compliance-sox-basel3-nist.md`: Marked P3-4, CM-5, and all CRD schema validation gaps as resolved; updated gap table and compliance control mapping
- `docs/roadmaps/project-roadmap-2026.md`: Updated Phase 3.1 Admission Webhooks → Admission Validation; checked off all implemented rules; noted future mutating webhook and reference-existence check as separate items
- `docs/src/security/threat-model.md`: Updated Deployment-Layer Controls table to reflect VAP deployed (was a recommendation)

### Why
`ValidatingAdmissionPolicy` (Kubernetes ≥ 1.26) enforces spec constraints at API-server admission time without requiring a separate webhook server, TLS certificate, or additional binary. Closes the NIST CM-5 gap: invalid specs that previously reached the reconciler are now rejected before being persisted to etcd.

### Impact
- [ ] Breaking change
- [x] Requires cluster rollout — apply `deploy/admission/` manifests; requires Kubernetes ≥ 1.26 (alpha), ≥ 1.28 (beta), ≥ 1.30 (GA)
- [ ] Config change only
- [ ] Documentation only

---

## [2026-04-08 12:00] - Complete rustdoc coverage across all Rust source files

**Author:** Erick Bourgeois

### Changed
- `src/reconcilers/helpers.rs`: Expanded thin one-liner docs on all remaining functions — `add_finalizer`, `handle_deletion`, `handle_kill_switch`, `check_grace_period_elapsed`, `update_phase_with_last_schedule`, `update_phase_with_grace_period`, `bootstrap_resource_name`, `infrastructure_resource_name`, `machine_resource_name`, `create_dynamic_resource`, `parse_api_version`, `remove_machine_from_cluster`, `should_evict_pod`, `evict_pod`, and `error_policy` — with full `///` docs covering purpose, behaviour details, and `# Errors` sections
- `src/bin/crdgen.rs`: Replaced `//` comment header with `//!` module doc explaining purpose, usage, and regeneration requirement; added `///` on `main()` with `# Panics` note
- `src/bin/crddoc.rs`: Added `//!` module doc explaining purpose, usage, and implementation note about static-println generation vs schema-driven approach; added `///` on `main()`

### Why
All public items and binary entry points now have complete rustdoc coverage to satisfy the project's documentation standard (CLAUDE.md §Code Comments) and to provide clear in-IDE guidance for future contributors.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [x] Documentation only

---

## [2026-04-08 00:03] - Phase 2 (P2-1/P2-2): Kubernetes Event audit trail and before/after phase logging

**Author:** Erick Bourgeois

### Changed
- `src/reconcilers/scheduled_machine.rs`: Added `Recorder` field to `Context` struct (created from `Reporter` with controller name and pod name); removed separate `client`/`recorder` args from all `update_phase*` call sites — now pass `&ctx` directly
- `src/reconcilers/helpers.rs`: Added `build_phase_transition_event()` pure function that constructs a `KubeEvent` from phase transition parameters (`Warning` for `Error`/`Terminated`, `Normal` otherwise); updated `update_phase()`, `update_phase_with_last_schedule()`, and `update_phase_with_grace_period()` to accept `&Context` (replacing separate `&Client` + `&Recorder` params), log `from → to` phase transition at `INFO` level, and publish an immutable Kubernetes Event via the recorder (best-effort — failures emit `WARN` but do not abort the transition)
- `deploy/deployment/rbac/clusterrole.yaml`: Added `events.k8s.io` / `events` create+patch rule alongside the existing core `""` events rule (kube-rs Recorder uses the `events.k8s.io/v1` API)
- `src/reconcilers/helpers_tests.rs`: Added 7 unit tests for `build_phase_transition_event()` covering Normal/Warning event types, note format, unknown from-phase fallback, action field, and reason field

### Why
P2-1 and P2-2 from the SOX/Basel III/NIST compliance roadmap. Every machine phase transition now writes an immutable Kubernetes Event visible via `kubectl describe scheduledmachine <name>`, providing an auditable record of state changes required by SOX §404 (immutable audit trail) and NIST AU-2/AU-3 (event recording and audit record content). Before/after logging closes the gap against AU-3 by making the previous phase explicit in each log line.

### Impact
- [ ] Breaking change
- [x] Requires cluster rollout
- [ ] Config change only
- [ ] Documentation only

---

## [2026-04-08 00:02] - Phase 1 Compliance Remediation (SOX/Basel III/NIST SP 800-53)

**Author:** Erick Bourgeois

### Changed
- `src/reconcilers/scheduled_machine.rs`: Replaced 7 `unwrap()` calls with `ok_or_else` error propagation in all phase handlers (`reconcile_inner`, `handle_pending_phase`, `handle_active_phase`, `handle_shutting_down_phase`, `handle_inactive_phase`, `handle_disabled_phase`, `handle_error_phase`) — NIST SI-3, P1-1
- `src/reconcilers/helpers.rs`: Replaced 3 `unwrap()` calls with `ok_or_else` error propagation in `add_finalizer`, `handle_deletion`, and `handle_kill_switch` — NIST SI-3, P1-1
- `src/metrics.rs`: Replaced 11 `.expect()` panics with graceful fallback pattern using private helper functions (`fallback_counter_vec`, `fallback_gauge`, `fallback_gauge_vec`, `fallback_histogram_vec`); metrics initialization failures now log a warning and continue rather than crash — NIST SI-3, P1-2
- `deploy/deployment/networkpolicy.yaml`: Created Kubernetes NetworkPolicy implementing NIST SC-7 boundary protection — ingress restricted to Prometheus scrape (port 8080, monitoring namespace only) and kubelet probes (port 8081); egress restricted to DNS (port 53) and Kubernetes API server (port 6443) — NIST SC-7, P1-3
- `src/main.rs`: Explicitly applied `K8S_API_TIMEOUT_SECS` constant to `kube::Config` `read_timeout` and `write_timeout` fields to enforce connection timeouts against Kubernetes API server — Basel III operational resilience, P1-4

### Why
Phase 1 of the SOX/Basel III/NIST SP 800-53 compliance remediation roadmap (`docs/roadmaps/compliance-sox-basel3-nist.md`). All `unwrap()`/`expect()` calls in production code paths represent potential uncontrolled panics that violate NIST SI-3 (Malicious Code Protection) and operational resilience requirements. The NetworkPolicy enforces least-privilege network access per NIST SC-7. Explicit API timeouts align with Basel III operational resilience requirements for bounded failure modes.

### Impact
- [ ] Breaking change
- [x] Requires cluster rollout
- [ ] Config change only
- [ ] Documentation only

---

## [2026-04-08 00:01c] - Convert all remaining ASCII diagrams to Mermaid

**Author:** Erick Bourgeois

### Changed
- `docs/src/concepts/schedules.md`: Converted cron field reference ASCII art to `flowchart LR` Mermaid diagram (5 labelled field nodes)

### Why
Project standard requires all diagrams to use Mermaid. This was the last remaining ASCII diagram across all docs.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [x] Documentation only

---

## [2026-04-08 00:01b] - Convert threat model ASCII diagrams to Mermaid

**Author:** Erick Bourgeois

### Changed
- `docs/src/security/threat-model.md`: Converted Section 2 system overview ASCII art to `flowchart TB` Mermaid diagram; converted Section 4 trust boundaries text block to `flowchart LR` Mermaid diagram

### Why
Project standard is Mermaid for all diagrams (consistent with architecture.md and other docs/src files).

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [x] Documentation only

---

## [2026-04-08 00:01] - Add threat model document

**Author:** Erick Bourgeois

### Changed
- `docs/src/security/threat-model.md`: New STRIDE threat model covering all controller components, trust boundaries, threat actors, 30+ threats with likelihood/impact ratings, full mitigations matrix, and 6 residual risk items with remediation guidance

### Why
Regulatory requirement in a banking environment: all security-significant components must have a documented threat model traceable to identified controls. This also captures the rationale behind the security hardening changes made in the same session.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [x] Documentation only

---

## [2026-04-08 00:00] - Add GitHub Actions CI/CD Workflows

**Author:** Erick Bourgeois

### Changed
- `.github/workflows/pr.yaml`: Pull Request CI — lint, test, Linux binary builds, Docker build/push, security scan
- `.github/workflows/main.yaml`: Main branch CI/CD — builds, Docker push (latest + date tags), security scan, Trivy container scan
- `.github/workflows/release.yaml`: Release workflow — versioned Docker images with Cosign signing, SLSA provenance, binary signing, deploy manifest packaging, release asset upload

### Why
Establish baseline CI/CD pipeline for the 5-spot operator using the same firestoned GitHub Actions patterns as bindy. Linux-only builds (x86_64 + ARM64) since this is a Kubernetes operator.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [ ] Documentation only

---

## [2026-04-08 00:00] - Security hardening: namespace isolation, input validation, RBAC narrowing

**Author:** Erick Bourgeois

### Changed
- `src/crd.rs`: Removed `namespace` field from `EmbeddedResource` — bootstrap and infrastructure resources are now always created in the ScheduledMachine's own namespace, preventing cross-namespace attacks
- `src/crd.rs`: Added `timezone_schema()` with `maxLength: 64` and character-class pattern constraint to block log injection via the timezone field
- `src/reconcilers/helpers.rs`: Fixed integer overflow in `parse_duration()` — now uses `checked_mul` and rejects durations exceeding 24 hours (`MAX_DURATION_SECS`)
- `src/reconcilers/helpers.rs`: Added `validate_labels()` — rejects label/annotation keys using reserved prefixes (`kubernetes.io/`, `k8s.io/`, `cluster.x-k8s.io/`, `5spot.finos.org/`) before merging into CAPI Machine resources
- `src/reconcilers/helpers.rs`: Added `validate_api_group()` — enforces an allowlist of permitted API groups for bootstrap and infrastructure embedded resources; blocks core Kubernetes APIs (`v1`, `rbac.authorization.k8s.io/v1`, etc.)
- `src/reconcilers/helpers.rs`: Wrapped `remove_machine_from_cluster` in `tokio::time::timeout` inside `handle_deletion` — finalizer cleanup now has a hard 10-minute deadline, preventing indefinite namespace deletion blocks
- `src/reconcilers/scheduled_machine.rs`: Added `ValidationError` and `TimeoutError` variants to `ReconcilerError`
- `src/constants.rs`: Added `MAX_DURATION_SECS`, `MAX_TIMEZONE_LEN`, `FINALIZER_CLEANUP_TIMEOUT_SECS`, `RESERVED_LABEL_PREFIXES`, `ALLOWED_BOOTSTRAP_API_GROUPS`, `ALLOWED_INFRASTRUCTURE_API_GROUPS`
- `deploy/deployment/rbac/clusterrole.yaml`: Narrowed `k0smotron.io` resources from wildcard to explicit list (`k0sworkerconfigs`, `remotemachines` and their `/status` subresources)
- `src/reconcilers/helpers_tests.rs`: New test file — 25 security-focused tests covering overflow protection, reserved label rejection, and API group allowlist enforcement
- `src/crd_tests.rs`, `src/reconcilers/scheduled_machine_tests.rs`: Removed `namespace: None` from `EmbeddedResource` test fixtures (field removed)

### Why
Comprehensive security audit identified: cross-namespace resource creation via user-controlled namespace overrides, integer overflow in duration parsing, label injection into CAPI resources, unbounded apiVersion/kind inputs, and missing finalizer cleanup timeouts. These are now all addressed to meet zero-trust security requirements for a regulated banking environment.

### Impact
- [x] Breaking change — `EmbeddedResource.namespace` field removed from CRD schema (existing CRs with this field: Kubernetes ignores unknown fields, no action required)
- [x] Requires cluster rollout — CRDs must be regenerated (`regen-crds` skill) before deploying
- [ ] Config change only
- [ ] Documentation only

---

## [2026-03-21 12:00] - Adopt .claude Skills Structure

**Author:** Erick Bourgeois

### Changed
- Created `.claude/` directory with SKILL.md and CHANGELOG.md
- Adopted skills-based workflow from bindy project
- Updated documentation structure for better organization

### Why
Standardize project instructions and skills across projects, improving consistency and making procedures reusable and discoverable.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [x] Documentation only

---

## [2026-01-18 12:00] - Add VMware cloud-init preparation script

**Author:** Unknown

### Added
- `scripts/install-cloud-init.sh`: Linux-only script to convert VMDK→raw, mount LVM with conflict-safe handling, chroot to install `cloud-init` and `open-vm-tools`, optional initramfs rebuild, raw→streamOptimized VMDK, and import as vSphere template via `govc`.

### Why
Enable automated preparation and deployment of a cloud-init-enabled RHEL image on a VMware VM. Credentials and vSphere target configuration are provided via environment variables to avoid storing secrets in code.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [x] Config change only
- [ ] Documentation only

---

## [2026-01-18 17:45] - Harden govc VM existence check in upload script

**Author:** Unknown

### Changed
- `scripts/install-cloud-init.sh`: Replaced fragile `govc vm.info`-based existence check with robust `govc find -type m -name <name>` logic; iterates over matched inventory paths, converts templates to VMs when needed, and destroys them before import.

### Why
`govc vm.info` can return exit code 0 with no output, leading to false positives. Using `govc find` and inspecting inventory paths provides reliable detection of existing VMs/templates with the target name and avoids confusing "not found" errors.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [x] Config change only
- [ ] Documentation only

---

## [2026-01-18 18:30] - Simplify LVM VG handling with isolated system directory

**Author:** Unknown

### Changed
- `scripts/install-cloud-init.sh`: Use `LVM_SYSTEM_DIR` to isolate loop device LVM metadata to a separate directory (`/tmp/lvm-loop-$$`); use temporary VG name (`vg00_loop`) if host has same VG name to avoid device-mapper conflicts in `/dev/mapper/`.

### Why
Device-mapper device names in `/dev/mapper/` are global at the kernel level, even with isolated LVM metadata via `LVM_SYSTEM_DIR`. If both host and loop device have `vg00` with LVs named `root`, `var`, etc., device-mapper refuses to create duplicate devices ("Device or resource busy"). By using `vgimportclone -n vg00_loop` when a conflict exists, we give the loop device VG a unique name for device-mapper while keeping metadata isolated. No rename needed after deactivation since the isolated metadata directory is simply deleted.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [x] Config change only
- [ ] Documentation only