Skip to content

CHANGELOG.md

Latest commit

 

History

History
967 lines (698 loc) · 71.1 KB

CHANGELOG.md

File metadata and controls

967 lines (698 loc) · 71.1 KB
Raw

Changelog

All notable changes to the 5spot project will be documented in this file.

The format is based on the regulated environment requirements:

  • Author attribution is MANDATORY for all entries
  • Changes are logged in reverse chronological order
  • Each entry must include impact assessment

[2026-04-19 20:15] - Rename Service to controller (RFC 1035 compliance)

Author: Erick Bourgeois

Changed

  • deploy/deployment/service.yaml: Renamed metadata.name from 5spot-controller to controller. Kubernetes Services require RFC 1035 DNS labels ([a-z]([-a-z0-9]*[a-z0-9])?) — stricter than the RFC 1123 rule used by Namespace/Deployment/ConfigMap/RBAC — and labels must start with a letter, not a digit. The cluster API server rejects 5spot-controller with metadata.name: Invalid value: "5spot-controller": a DNS-1035 label…. Added a header comment explaining the constraint so nobody renames it back.
  • docs/src/installation/controller.md: Updated the port-forward example from svc/5spot-controller to svc/controller.

Why

kubectl apply --dry-run=client did not catch this — client-side dry-run skips the Service-specific admission validator. Only --dry-run=server (or an actual apply) surfaces the RFC 1035 rule. Renaming to controller is safe here because the Service is already scoped by the 5spot-system namespace; cluster DNS resolves it as controller.5spot-system.svc.cluster.local. The ServiceMonitor selector matches on label app: 5spot-controller (not on Service name), so monitoring continues to work without change.

Audit: every other kube resource name

Server-side validated against a live cluster — all resources other than the Service were already compliant. Kubernetes applies different name rules per kind; this matrix confirms each one:

Kind Name Rule Result
Namespace 5spot-system RFC 1123 label (digit-start OK) ✅
ConfigMap 5spot-controller-config RFC 1123 subdomain ✅
Deployment 5spot-controller RFC 1123 subdomain ✅
Service controller RFC 1035 label (letter-start required) ✅ (renamed)
NetworkPolicy 5spot-controller RFC 1123 subdomain ✅
PodDisruptionBudget 5spot-controller RFC 1123 subdomain ✅
ClusterRole 5spot-controller RFC 1123 subdomain ✅
ClusterRoleBinding 5spot-controller RFC 1123 subdomain ✅
ServiceAccount 5spot-controller RFC 1123 subdomain ✅
ServiceMonitor 5spot-controller CR default = RFC 1123 subdomain ✅
CustomResourceDefinition scheduledmachines.5spot.finos.org <plural>.<group>, each label RFC 1123 (digit-start OK) ✅
ValidatingAdmissionPolicy scheduledmachine-validation RFC 1123 subdomain ✅
ValidatingAdmissionPolicyBinding scheduledmachine-validation-binding RFC 1123 subdomain ✅
ScheduledMachine business-hours-worker, weekend-worker RFC 1123 subdomain ✅

Unrelated issue surfaced during validation

examples/scheduledmachine-weekend.yaml fails server-side admission with unknown field "spec.schedule.cron" — the example is out of date against the current CRD schema (which uses day/hour ranges, not cron expressions). Tracked separately; not fixed in this commit.

Impact

  • Breaking change (anyone referencing 5spot-controller.5spot-system.svc.cluster.local — e.g. in ServiceMonitor endpoints[*].port lookups by Service name, Ingress backends, or custom dashboards — must use controller.5spot-system.svc.cluster.local)
  • Requires cluster rollout (old Service object must be kubectl deleted; the new one will be created on re-apply)
  • Config change only
  • Documentation only

[2026-04-19 19:45] - Resolve Trivy IaC findings in GitHub Code Scanning

Author: Erick Bourgeois

Changed

  • deploy/deployment/deployment.yaml:
    • Replaced image: ghcr.io/RBC/5-spot:latest with image: ghcr.io/finos/5-spot:v0.1.0 (Trivy KSV-0013 — image :latest tag; also removes a stale RBC-internal reference that violated the CLAUDE.md internal-references rule and pointed at a non-existent org).
    • Added seccompProfile: { type: RuntimeDefault } at both the pod spec.securityContext and the container securityContext (Trivy KSV-0104 — seccomp policies disabled, KSV-0030 — runtime/default seccomp profile not set).
    • Added runAsGroup: 65534 to the container securityContext (Trivy KSV-0021 — runs with low GID; 65534 is the nogroup/nobody GID on distroless and Chainguard images).
  • Dockerfile:
    • Removed "RBC Capital Markets" from the copyright header (CLAUDE.md internal-references rule).
    • Pinned the distroless base image by digest: gcr.io/distroless/cc-debian13:nonroot@sha256:8f960b7fc6a5d6e28bb07f982655925d6206678bd9a6cde2ad00ddb5e2077d78. Dependabot (docker ecosystem) will open a re-pin PR when Google publishes a patched image.
  • Dockerfile.chainguard:
    • Removed "RBC Capital Markets" from the copyright header.
    • Pinned the Chainguard glibc-dynamic base image by digest: cgr.dev/chainguard/glibc-dynamic:latest@sha256:fa0d07a6a352921b778c4da11d889b41d9ef8e99c69bc2ec1f8c9ec46b2462e9 (Trivy DS-0001 — :latest tag used). Chainguard rebuilds this tag daily with security patches and Dependabot picks up the new digest on each rebuild.
  • .trivyignore (new): Six architecturally-justified suppressions, each with a written rationale so an auditor can answer "why is this ignored" without reading code:
    • AVD-KSV-0046 — RBAC wildcard on bootstrap.cluster.x-k8s.io and infrastructure.cluster.x-k8s.io API groups is required by the provider-agnostic design (controller must support any CAPI provider installed in the cluster).
    • AVD-KSV-0048 — Pod/eviction permissions are required for node drain during machine shutdown.
    • AVD-KSV-0041 — Secret access is read-only (get/list/watch), required to resolve SSH keys and bootstrap-data references.
    • AVD-KSV-0125 — Registry allow-listing is a cluster-level admission policy (Kyverno / OPA / VAP), not a workload field.
    • AVD-KSV-01010 — ConfigMap "sensitive content" finding is a false positive; the ConfigMap holds only log-level strings and port numbers.
    • AVD-DS-0026 — Kubernetes uses livenessProbe/readinessProbe on :8081 for health; Dockerfile HEALTHCHECK would be dead code, and distroless/Chainguard have no shell to run one.
  • .github/workflows/build.yaml: Run Trivy config scan step now explicitly passes trivyignores: ./.trivyignore so the suppressions apply deterministically in CI (Trivy auto-discovers the file by default, but the explicit path documents intent and survives future action-version bumps).

Why

GitHub Code Scanning was showing 10 open Trivy IaC findings against 5-Spot's deploy manifests and Dockerfiles, including two error-severity KSV-0046 alerts. Most were real hardening gaps (seccomp, low GID, :latest tags, stale RBC image path); a handful (CAPI wildcards, pod eviction, read-only secret access) are load-bearing architectural choices that belong in an explicit suppression list with written justification rather than being fixed away. A local trivy config run after the changes now reports 0 misconfigurations across all 17 scanned files. Separately, this fixes two CLAUDE.md policy violations — the ghcr.io/RBC/5-spot image reference and the "RBC Capital Markets" copyright headers — that had been quietly sitting in the tree.

Impact

  • Breaking change
  • Requires cluster rollout (deployment.yaml securityContext + image tag change)
  • Config change only (Dockerfile digest pins, .trivyignore, CI workflow input)
  • Documentation only

Operational note

The deployment image is now pinned to v0.1.0 — the current latest GitHub release. When a new release is cut, either edit deploy/deployment/deployment.yaml or override the tag via your kustomize/Helm overlay. Do not revert to :latest.

[2026-04-19 18:00] - Add OpenVEX generation, validation, and signing pipeline

Author: Erick Bourgeois

Changed

  • .vex/README.md (new), .vex/.gitkeep (new): repository convention for hand-authored per-CVE triage. One TOML file per CVE with status, justification (enum), products, author, and timestamp. Documents the workflow, the allowed enum values, and what is required per status (not_affected → justification; affected/under_investigation → action_statement).
  • tools/validate-vex.sh (new), tools/validate_vex.py (new): shell + Python (tomllib, stdlib) validator. Checks per-file schema, enum membership (status, justification), non-empty products, RFC-3339 UTC timestamp, and CVE uniqueness across files. Deterministic error output.
  • tools/assemble_openvex.py (new): assembler that reads .vex/*.toml and emits a single OpenVEX v0.2.0 JSON document with a canonical @id. Runs validate_dir first so malformed input never yields a document. Normalizes TOML datetimes to RFC-3339 UTC Z-suffix strings, sorts keys for diffable output.
  • tools/tests/validate-vex-tests.sh + 18 fixture dirs: 19 cases covering every positive/negative/exception branch (empty dir, missing dir, missing each required field, malformed TOML, bad CVE format, invalid status/justification, empty products, bad timestamp, missing-justification-when-required, missing-action-statement-when-required, duplicate CVE across files, valid-single / valid-multiple / valid-affected).
  • tools/tests/assemble-openvex-tests.sh: 6 cases covering happy path (file + stdout), JSON validity, validator-gate negative path, CLI argument errors, default-timestamp RFC-3339 shape.
  • .github/workflows/build.yaml:
    • Added .vex/** and tools/** to the PR paths: filter so changes to either directory trigger the workflow.
    • New PR-only validate-vex job runs the validator unit tests and then validates the live .vex/ directory. Gates PRs touching .vex/** or tools/**.
    • New release-only build-vex job (needs: [docker, extract-version]): defensively re-runs the validator, calls assemble_openvex.py with a canonical @id = https://github.com/<repo>/releases/tag/<tag>/vex, installs pinned vexctl and runs vexctl validate as a second-opinion check, Cosign-attests the document to both image digests (cosign attest --type openvex for Chainguard + Distroless), runs actions/attest-build-provenance on the document, and uploads vex.openvex.json + .bundle as a workflow artifact.
    • upload-release-assets now needs: build-vex, downloads the openvex artifact, copies the document to release/ and the attestation bundle to signatures/, includes both in checksums.sha256, and publishes the document as a release asset.
  • docs/src/security/vex.md (new): user-facing page covering what VEX is, what gets published, how to verify (Cosign + gh attestation verify), how to consume (grype --vex, trivy --vex), and how maintainers add statements. Linked from docs/src/security/index.md and docs/mkdocs.yml nav.
  • README.md: Security section now lists OpenVEX under the publication surface with links to docs/src/security/vex.md and .vex/README.md.
  • .claude/SKILL.md: pre-commit-checklist skill gains a "If preparing a release" block requiring every open Trivy finding to have a statement in .vex/, plus explicit commands for the validator and the two test scripts.

Why

Without VEX, every Trivy CVE surfaced on 5-Spot images re-triages itself at every downstream consumer. Publishing a signed OpenVEX document once — per release, bound to image digests via Cosign, attached to the GitHub Release — pushes the triage decision to exactly one place (.vex/<cve>.toml, PR-reviewed) and lets scanners (Grype, Trivy, Harbor) suppress already-triaged findings without repeating the analysis. Keeps 5-Spot honest: only human-authored statements ship, and CI won't emit a document from a malformed source tree.

Impact

  • Breaking change
  • Requires cluster rollout
  • Config change only (CI/CD workflow + new repo convention)
  • Documentation only

Operational note

  • vexctl pin: build-vex installs vexctl by downloading a tagged release tarball (VEXCTL_VERSION: '0.3.0'). Per the roadmap's "Dependencies to pin" section this should be replaced with SHA-pinned asset verification (cosign verify-blob) before the first release actually flows through this workflow; re-pin quarterly with the rest of the signing toolchain.
  • First release after merge: add at least one hand-authored statement in .vex/ (even under_investigation on a known low-severity CVE) so the signing chain is exercised end-to-end and the release contains a non-empty vex.openvex.json.
  • Phase 4 of the roadmap (CycloneDX-VEX co-emission) is intentionally deferred — kept gated behind an explicit consumer ask per the roadmap's decision gate. Phases 1/2/3/5 are now in place.

[2026-04-19 17:15] - Add Dependabot config for Actions, Cargo, and Docker

Author: Erick Bourgeois

Changed

  • .github/dependabot.yml (new): SPDX-headered Dependabot v2 config with three ecosystems:
    • github-actions — weekly Monday 09:00 America/Toronto, limit 10 PRs. Grouped update actions-routine bundles low-risk first-party + tooling bumps (actions/*, docker/*, github/codeql-action, softprops/action-gh-release, anchore/sbom-action, aquasecurity/trivy-action) into one PR. Security-sensitive actions (sigstore/*, EmbarkStudios/cargo-deny-action, ossf/*) are left ungrouped so each one opens as an individual PR for per-change review. dtolnay/rust-toolchain explicitly ignored (branch-tracking ref, no tags — re-pinned manually on quarterly cadence).
    • cargo — weekly Monday, limit 5 PRs. Groups patch + minor bumps; majors open individually. cargo-audit + cargo-deny CI gates block regressions.
    • docker — weekly Monday, limit 3 PRs. Picks up FROM ...@sha256:... digest bumps once Dockerfiles adopt digest pinning (currently no-op; future-proof).
  • Commit-message prefix: ci for Actions, chore for Cargo and Docker. Labels applied to every PR for easy filtering.

Why

The previous commit SHA-pinned all external GitHub Actions across the workflows, which is correct for supply-chain hygiene but creates stagnation risk — pins do not auto-update, so known-vulnerable action versions can sit in CI indefinitely. Dependabot solves the stagnation side of the trade-off: it opens a PR per new release with the full changelog diff, and the same CI gates (Semgrep, Trivy, cargo-deny, Scorecard) that guard any other PR guard the bump itself. This closes the loop between "pin everything" and "keep pins current" without inviting humans to cowboy-update SHAs.

Impact

  • Breaking change
  • Requires cluster rollout
  • Config change only (Dependabot)
  • Documentation only

Operational note

  • PR volume: expect 1–3 Actions PRs and 1–2 Cargo PRs per week initially, tapering as the tree stabilizes. The actions-routine group keeps the routine noise in one PR.
  • Review rule of thumb: security-sensitive bumps (sigstore, cargo-deny, ossf) → read the full release notes before merging. Routine Actions group → verify Scorecard Pinned-Dependencies still passes, spot-check the diff for any behaviour change flags, merge if CI is green.
  • Future-proof Docker: Dockerfile and Dockerfile.chainguard use tag-based FROM lines today. If we move to digest pinning (FROM image@sha256:...), Dependabot starts opening PRs for base-image digest bumps automatically — no further config change needed.

[2026-04-19 17:00] - Pin all GitHub Actions to full commit SHAs

Author: Erick Bourgeois

Changed

  • .github/workflows/build.yaml, calm.yaml, calm-test.yaml, docs.yaml and .github/actions/prepare-docker-binaries/action.yaml: replaced every @<tag> reference with @<40-char-sha> # <tag> for all external actions. 77 replacements across 5 files covering actions/*, docker/*, sigstore/cosign-installer, anchore/sbom-action, aquasecurity/trivy-action, EmbarkStudios/cargo-deny-action, softprops/action-gh-release, github/codeql-action/upload-sarif, dtolnay/rust-toolchain, and all firestoned/github-actions/* sub-actions. scorecard.yaml was already SHA-pinned and is unchanged.
  • .github/workflows/build.yaml: the SLSA reusable workflow at slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@v2.1.0 stays on a semver tag — pinning to SHA would break SLSA provenance verification because slsa-verifier validates against approved released versions. Added an inline multi-line comment explaining this so future reviewers don't "fix" it by SHA-pinning.
  • aquasecurity/trivy-action was previously pinned to the nonexistent tag 0.28.0 (missing v prefix — the tag is v0.28.0). Bumped to the current latest v0.35.0 at SHA 57a97c7e7821a5776cebc9bb87c984fa69cba8f1.
  • dtolnay/rust-toolchain@stable resolved to SHA 29eef336d9b2848a0b548edc03f92a220660cdb8 with an inline comment noting it is a moving ref that should be re-pinned quarterly to keep Rust current.

Why

Closes the OpenSSF Scorecard Pinned-Dependencies finding across the entire CI surface. Unpinned action tags are a supply-chain risk: a compromised or silently-force-pushed tag would silently execute attacker code in our workflows with access to GITHUB_TOKEN, SNYK_TOKEN-equivalent secrets, and GHCR push permissions. SHA pinning freezes behaviour to a specific reviewed commit; Dependabot can later send PRs to bump pins with a full diff.

Impact

  • Breaking change
  • Requires cluster rollout
  • Config change only (CI/CD workflow)
  • Documentation only

Operational note

  • Dependabot for Actions is strongly recommended as a follow-up so pins don't rot. Add .github/dependabot.yml with a github-actions ecosystem entry; Dependabot will open one PR per outdated action SHA with a changelog link.
  • Re-pin cadence: security-sensitive actions (sigstore/*, ossf/*, EmbarkStudios/cargo-deny-action) should be bumped within a week of a new release. Build tooling (docker/*, actions/*) is less time-sensitive. dtolnay/rust-toolchain should be bumped quarterly to keep the Rust toolchain current.
  • All pin comments include the resolved semver tag (e.g. # v4.3.1) so a human reader can see at a glance what version is in use without clicking through to the SHA.

[2026-04-19 17:30] - Phase 5 docs finish: regenerate stale api.md (mdBook source)

Author: Erick Bourgeois

Changed

  • docs/src/reference/api.md: regenerated via make crddoc. Now includes the Phase 1 status-schema additions — providerID, extended nodeRef (with uid, apiVersion, kind alongside name), machineRef, bootstrapRef, infrastructureRef, and refreshed condition/observedGeneration docstrings. Previously this file was stale; Phase 1's regen wrote only to docs/reference/api.md (a legacy path referenced by .claude/SKILL.md), not to the mdBook source path that actually renders on the doc site (docs/src/reference/api.md, per the Makefile crddoc target).

Why

The Phase 5 checklist in the roadmap required "docs/src/reference/api.md — regenerated by regen-api-docs skill." Spot-checking after the earlier Phase 5 commit revealed that the canonical Makefile target output path and the SKILL.md instruction disagreed, and the mdBook source copy was still on the pre-Phase-1 schema. Regenerating closes that gap so the doc site reflects what consumers actually get on kubectl get scheduledmachine -o yaml.

Impact

  • Breaking change
  • Requires cluster rollout
  • Config change only
  • Documentation only

Operational note

  • SKILL.md drift: .claude/SKILL.md currently instructs cargo run --bin crddoc > docs/reference/api.md, which writes to the wrong path. The Makefile crddoc target is correct. A small follow-up should reconcile the skill instructions with the Makefile so future regens aren't silently written to the wrong path.

[2026-04-19 16:30] - Phase 5 docs: watch-topology diagram in architecture.md

Author: Erick Bourgeois

Changed

  • docs/src/concepts/architecture.md: New top-level section "Watch Topology" with a Mermaid diagram showing the primary ScheduledMachine watch plus the two new secondary watches on CAPI Machine (label-filtered) and core Node (cluster-wide) feeding into pure reverse-mapper functions (machine_to_scheduled_machine, node_to_scheduled_machines) that enqueue the owning ScheduledMachine. Updated the "Controller" component-detail bullet list to enumerate the three watches.
  • docs/roadmaps/5spot-event-driven-watches-and-status-enrichment.md (copy at ~/dev/roadmaps/): status header updated to Phase 5 ✅ — roadmap closed.

Why

Closes Phase 5 of the event-driven-watches roadmap. Until this commit, the concepts doc described a controller that only watched its own CR; readers had no way to understand why a Node cordon now triggers an immediate reconcile instead of waiting for the next requeue. The watch-topology diagram makes the event-driven claim concrete and auditable.

Impact

  • Breaking change
  • Requires cluster rollout
  • Config change only
  • Documentation only

[2026-04-19 16:00] - Replace misleading Snyk claim with real OSS security tooling (Semgrep + Trivy config + cargo-deny)

Author: Erick Bourgeois

Changed

  • .github/workflows/build.yaml: Three new PR + push-to-main jobs (release events skip these — the release pipeline relies on scans that already ran on main):
    • semgrep-sast — runs semgrep scan in the returntocorp/semgrep container with the community p/rust, p/security-audit, p/secrets, and p/owasp-top-ten rulesets; --metrics=off; uploads SARIF to Code Scanning under the semgrep category. No token required.
    • iac-scan — aquasecurity/trivy-action@0.28.0 with scan-type: config against the repo root (picks up deploy/**/*.yaml + Dockerfile* + Dockerfile.chainguard); uploads SARIF under the trivy-iac category.
    • cargo-deny — EmbarkStudios/cargo-deny-action@v2 running check --all-features against the new deny.toml.
  • deny.toml (new, repo root): SPDX-headered cargo-deny config. License allow-list covers Apache-2.0, MIT, BSD-2/3-Clause, ISC, Unicode-DFS-2016, Unicode-3.0, Zlib, CC0-1.0, MPL-2.0, OpenSSL. Advisories: yanked = "deny". Bans: multiple-versions = "warn" (to avoid breaking CI on transitive pulls we do not control), wildcards = "deny". Sources: only official crates.io registry, no unknown git URLs.
  • README.md: Removed the misleading Snyk SAST badge (we never actually ran Snyk). Added badges for Semgrep, Trivy (Container + IaC), cargo-deny, cargo-audit, Cosign, and SLSA — each reflecting tooling that actually runs in the pipeline.
  • docs/architecture/calm/architecture.json: The supply-chain-scanning control description now reads: "Repository is scanned by Semgrep OSS (SAST), Trivy (container image + IaC config), cargo-audit + cargo-deny (RustSec advisories, license allow-list, source restrictions), and Gitleaks (secrets); OpenSSF Scorecard publishes supply-chain posture; SPDX license identifiers on source files." Old wording referenced Snyk and Aqua which were never actually wired up.

Why

The README's Security & Compliance section and the CALM architecture JSON both claimed "Snyk (SAST)" but rg -i snyk returned zero hits outside those two strings — the repo has never had Snyk configured. That is a compliance posture lie for a project in a regulated banking context. This commit closes the three real gaps with free OSS tooling: SAST (Semgrep), IaC misconfig scanning (Trivy config), and dependency license/advisory enforcement (cargo-deny). All three are token-free, run on every PR, and write SARIF to the existing Code Scanning dashboard — same surface consumers already use for Scorecard and Trivy container results.

Impact

  • Breaking change
  • Requires cluster rollout
  • Config change only (CI/CD workflow + cargo-deny config)
  • Documentation only

Operational note

  • First run of cargo-deny may fail if an existing transitive dep carries a license not in the allow-list. Treat the first failure as signal: either add the license to deny.toml (with a justification comment) or file an issue to swap the dep. Do not weaken unknown-registry = "deny" — that is a supply-chain boundary, not a nuisance.
  • Semgrep findings initially expected: the first run will surface Rust and OWASP-style findings that were never triaged. Review in Code Scanning → filter by tool semgrep → triage-or-suppress with justification (Semgrep supports // nosemgrep: rule-id — reason inline comments).
  • Trivy IaC findings: scope is the deploy/ manifests plus both Dockerfiles. Findings you cannot fix (e.g., a base-image constraint) can be suppressed via a .trivyignore file at repo root with a comment.
  • All three jobs run as needs: [verify-commits] so they parallelize with extract-version/build and do not serialize the critical path.

[2026-04-19 15:45] - Add OpenSSF Scorecard badge to README

Author: Erick Bourgeois

Changed

  • README.md: Added [![OpenSSF Scorecard]...] badge as the first entry in the "Security & Compliance" section, linking to https://scorecard.dev/viewer/?uri=github.com/finos/5-spot.

Why

The scorecard.yaml workflow already publishes results to the OpenSSF REST API (api.securityscorecards.dev) and to GitHub Code Scanning, but without a visible badge the score is invisible to anyone reading the README. The badge is the canonical consumer-facing signal that the project runs Scorecard and auto-updates with each scorecard.yaml run.

Impact

  • Breaking change
  • Requires cluster rollout
  • Config change only
  • Documentation only

[2026-04-19 15:30] - Sign binaries and generate SLSA provenance on push-to-main (not only on release)

Author: Erick Bourgeois

Changed

  • .github/workflows/build.yaml:
    • sign-artifacts job: if flipped from github.event_name == 'release' to != 'pull_request'. Added attestations: write and contents: read permissions. New step runs actions/attest-build-provenance@v2 on each signed tarball (amd64 + arm64) — GitHub-native attestation in addition to the Cosign signature.
    • generate-provenance-subjects job: if flipped to != 'pull_request' so SLSA subject hashes are computed on push-to-main too.
    • slsa-provenance job: if flipped to != 'pull_request'; the reusable SLSA generator now runs on every push-to-main. upload-assets parameterised to ${{ github.event_name == 'release' }} so the .intoto.jsonl only attaches to the GitHub Release on release events; on push-to-main it lands as a workflow artifact (verifiable via slsa-verifier or gh attestation verify).
  • upload-release-assets unchanged — stays if: github.event_name == 'release'; its needs: on sign-artifacts/slsa-provenance resolves cleanly whether those jobs ran (push, release) or were skipped (PR).

Why

Every merge to main produces a buildable, distributable artifact — it should carry the same supply-chain assurances as a release. With these changes, a main build now has: Cosign keyless signature on the container image (already in place), GitHub Artifact Attestation on the container image (already in place), Cosign signature on each binary tarball (new on main), GitHub Artifact Attestation on each binary tarball (new), and SLSA Level 3 provenance for the binaries (new on main). Consumers pulling a main-YYYY-MM-DD build can now cryptographically verify it the same way they would a release.

Impact

  • Breaking change
  • Requires cluster rollout
  • Config change only (CI/CD workflow)
  • Documentation only

Operational note

  • Push-to-main runs now take longer (extra ~3–5 min for the SLSA generator + attestations). Expected trade-off.
  • sign-artifacts signed-tarball artifact retention is the repo default (90 days) — long enough for verification and audit, well under the release-asset lifetime.
  • Binary SLSA generator dependency pinned at @v2.1.0 (unchanged) — verify this is the latest patched version periodically.

[2026-04-19 15:00] - Harden OpenSSF Scorecard workflow: SPDX header + pin remaining action

Author: Erick Bourgeois

Changed

  • .github/workflows/scorecard.yaml: Prepended the project's standard two-line SPDX/Copyright header (Copyright (c) 2025 Erick Bourgeois, firestoned + SPDX-License-Identifier: Apache-2.0) to match every other workflow in .github/workflows/. Pinned the previously unpinned github/codeql-action/upload-sarif@v3 to the full commit SHA for v3.35.2 (ce64ddcb0d8d890d2df4a9d1c04ff297367dea2a) with an inline version comment — brings the last step into line with the other three actions in the file which were already SHA-pinned.

Why

OpenSSF Scorecard's own Pinned-Dependencies check flags float-tagged actions (@v3) as a supply-chain risk, so the scorecard workflow itself should score well on that check. The SPDX header is the repo-wide convention; every other workflow has it. Both changes are compliance housekeeping — no behavior change.

Impact

  • Breaking change
  • Requires cluster rollout
  • Config change only
  • Documentation only

[2026-04-19 14:00] - Phases 3 + 4: Event-driven .watches() on CAPI Machine and Node

Author: Erick Bourgeois

Changed

  • src/reconcilers/helpers.rs: Added two pure mapper functions used as secondary-watch reverse-maps on the Controller builder:
    • machine_to_scheduled_machine(&DynamicObject) -> Vec<ObjectRef<ScheduledMachine>> — reads the 5spot.eribourg.dev/scheduled-machine label (constant LABEL_SCHEDULED_MACHINE) and emits at most one ObjectRef. Guards against missing/empty/whitespace label values, missing namespace, and imposter labels with similar prefixes.
    • node_to_scheduled_machines<'a, I: IntoIterator<Item = &'a ScheduledMachine>>(&Node, I) -> Vec<ObjectRef<ScheduledMachine>> — returns all SMs whose status.nodeRef.name matches node.metadata.name. Returns multiple refs on conflict so the reconciler can surface the issue. O(N) per Node event; documented as acceptable for current scale.
  • src/reconcilers/mod.rs: Re-exports both mappers alongside the existing error_policy, evaluate_schedule, should_process_resource.
  • src/main.rs: Extended the Controller builder with .watches_with(...) on CAPI Machine (ApiResource from GVK, label-filtered watcher::Config) and .watches(...) on Node. The Node mapper closure captures a clone of controller.store() (reflector::Store<ScheduledMachine>) and calls state() on each event to avoid out-of-band API lists.
  • src/reconcilers/helpers_tests.rs: Added 14 new unit tests — 7 for machine_to_scheduled_machine (positive match, missing label, no labels, empty value, whitespace value, missing namespace, wrong-prefix imposter) and 7 for node_to_scheduled_machines (single match, multi-match conflict, no match, empty list, SM-without-status, Node-without-name, empty nodeRef.name). Covers positive, negative, and exception paths per the project's 100%-coverage rule.

Why

Closes roadmap Phases 3 + 4 of event-driven-watches-and-status-enrichment.md. The ScheduledMachine controller previously only watched its own CR — all downstream state was polled during reconciles, which CLAUDE.md explicitly forbids ("ALWAYS use event-driven programming… as opposed to polling"). With these two watches, CAPI setting status.nodeRef or a Node being drained now enqueues an immediate reconcile via kube-rs's watch stream (<1s debounce) instead of waiting for the next periodic requeue. The label-based reverse map on Machines uses the label we already stamp on every child — zero new ownership semantics, no controller: true owner references introduced, no contention with CAPI's own controllers.

Impact

  • Breaking change
  • Requires cluster rollout — controller now opens additional watches (CAPI Machine with label selector; Node cluster-wide)
  • Config change only
  • Documentation only

Operational note

  • RBAC: the controller already has get/list/watch on CAPI Machine (used for drain lookup) and needs list/watch on core Node. Verify get,list,watch on nodes is present in the ClusterRole before rollout; if not, add it in the same rollout.
  • Observability: the existing reconcile_queue_depth metric will show enqueues driven by Machine and Node events in addition to SM events.

[2026-04-19 10:00] - Phase 2: Close test-coverage gap for async CAPI helpers

Author: Erick Bourgeois

Changed

  • src/reconcilers/helpers_tests.rs: Added 14 tower-test-backed unit tests covering positive, negative, and exception paths for the three async helpers introduced in the previous Phase 2 entry:
    • fetch_capi_machine — 200 returns Some, 404 returns Ok(None), 500 and 403 map to CapiError.
    • patch_machine_refs_status — both-fields patch asserts full body shape, single-field patches assert the other key is omitted (not null) so merge-patch never clears existing values, both-None case asserts zero HTTP traffic, 500 and 404 map to KubeError.
    • get_node_from_machine — success returns the node name, machine-404 and nodeRef-missing both return Ok(None), 500 propagates as CapiError.

Why

Per durable user guidance: every function (public and private) must have unit tests covering the happy path, negative cases, and exception/error paths. The original Phase 2 change landed with tests only for the pure extract_machine_refs function — the three async helpers, which actually touch the Kubernetes API surface, were undertested. This entry closes that gap so the coverage floor now matches the project rule (not just CLAUDE.md's "public function" minimum).

Impact

  • Breaking change
  • Requires cluster rollout
  • Config change only
  • Documentation only (test-only change — no runtime behaviour modified)

[2026-04-18 12:00] - Phase 2: Populate providerID and nodeRef from CAPI Machine

Author: Erick Bourgeois

Changed

  • src/reconcilers/helpers.rs: Added three new helpers — extract_machine_refs (pure function pulling providerID + full NodeRef out of a CAPI Machine DynamicObject), fetch_capi_machine (typed 404 → Ok(None) wrapper around the dynamic GET), and patch_machine_refs_status (merge-patches both fields onto ScheduledMachine.status). Refactored get_node_from_machine to delegate to the first two helpers for the drain path.
  • src/reconcilers/scheduled_machine.rs: handle_active_phase happy path now fetches the CAPI Machine, extracts refs, and patches them onto the SM status. Failures are logged and ignored — status enrichment must never block reconciliation.
  • src/reconcilers/helpers_tests.rs: Added 6 TDD cases for extract_machine_refs covering fully populated, empty, providerID-only, nodeRef-without-uid, incomplete-nodeRef-returns-None, and malformed-providerID-ignored.

Why

Phase 2 of the event-driven watches + status enrichment roadmap. The schema landed in Phase 1; this change actually populates the new fields every time the reconciler visits an active machine, so kubectl get sm -o jsonpath='{.status.providerID}{"\t"}{.status.nodeRef.name}' returns meaningful values.

Impact

  • Breaking change
  • Requires cluster rollout — controller image carries the new reconcile logic
  • Config change only
  • Documentation only

[2026-04-16 17:00] - Phase 1: Enrich ScheduledMachine status with providerID and full nodeRef

Author: Erick Bourgeois

Changed

  • src/crd.rs: Added provider_id: Option<String> (serialized as providerID) to ScheduledMachineStatus. Replaced the thin LocalObjectReference { name } node reference with a new NodeRef { apiVersion, kind, name, uid } struct mirroring CAPI's Machine.status.nodeRef. Removed the now-unused LocalObjectReference type.
  • src/crd_tests.rs: Added 6 TDD cases covering providerID round-trip, full nodeRef deserialization, optional uid, serialization omission, old-shape rejection, and NodeRef round-trip.
  • src/bin/crddoc.rs: Documented new providerID and nodeRef status fields.
  • deploy/crds/scheduledmachine.yaml: Regenerated from the updated Rust types.
  • docs/reference/api.md: Regenerated to reflect new status schema.

Why

Phase 1 of the event-driven watches + status enrichment roadmap. Surfacing providerID and a full Node reference (with UID) on ScheduledMachine.status lets operators correlate a scheduled machine to a specific VM and Node from kubectl get sm -o jsonpath=..., without manual lookups across CAPI Machines and the Node API. This is the schema foundation that Phase 2 (reconciler populates the fields) and Phases 3–4 (event-driven watches on CAPI Machine and Node) build upon.

Impact

  • Breaking change — status.nodeRef shape changed from { name } to { apiVersion, kind, name, uid }. Existing CRs with the old shape must clear status.nodeRef before rollout, or the controller will report deserialization errors on that field.
  • Requires cluster rollout — CRD must be re-applied alongside the new controller image.
  • Config change only
  • Documentation only

[2026-04-17] - Bump base image to cc-debian13 and fix GLIBC_2.39 crash (issue #17)

Author: Daniel Guns

Changed

  • Dockerfile: Base image bumped from gcr.io/distroless/cc-debian12:nonroot (glibc 2.36) to gcr.io/distroless/cc-debian13:nonroot (glibc 2.41)
  • .github/workflows/build.yaml: Pinned Linux x86_64 runner from ubuntu-latest to ubuntu-24.04 for CI stability
  • Cargo.lock: Updated transitive dependency rustls-webpki from 0.103.11 to 0.103.12

Why

ubuntu-latest now resolves to Ubuntu 24.04 (glibc 2.39), producing binaries that require GLIBC_2.39 at runtime. The previous cc-debian12 base only provides glibc 2.36, causing a hard crash at container startup. Bumping to cc-debian13 (glibc 2.41) resolves the mismatch. Runners are explicitly pinned to ubuntu-24.04 so CI doesn't break silently when ubuntu-latest moves to 26.04. Additionally, two CVEs in rustls-webpki 0.103.11 (RUSTSEC-2026-0098, RUSTSEC-2026-0099) were patched by bumping to 0.103.12. Fixes issue #17.

Impact

  • Breaking change
  • Requires cluster rollout — new base image
  • Config change only
  • Documentation only

[2026-04-10 08:50] - Extend Cosign image signing to main-branch pushes

Author: Erick Bourgeois

Changed

  • .github/workflows/build.yaml: Changed Cosign signing step condition from github.event_name == 'release' to github.event_name != 'pull_request'

Why

Main-branch images are tagged latest and main-YYYY-MM-DD and may be deployed to staging. Signing them allows cosign verify to work on staging images, not just production releases. PR images remain unsigned — they are ephemeral, tagged pr-{number}, and not deployed anywhere.

Impact

  • Breaking change
  • Requires cluster rollout
  • Config change only
  • Documentation only

[2026-04-10 08:45] - Fix attest job: add GHCR login before push-to-registry

Author: Erick Bourgeois

Changed

  • .github/workflows/build.yaml: Added docker/login-action@v3 step to the attest job before actions/attest-build-provenance@v2

Why

push-to-registry: true in actions/attest-build-provenance pushes the attestation bundle as an OCI artifact to GHCR, which requires registry credentials. Each job runs in a fresh environment — the Docker login performed by firestoned/github-actions/docker/setup-docker in the docker job does not carry over to the attest job.

Impact

  • Breaking change
  • Requires cluster rollout
  • Config change only
  • Documentation only

[2026-04-10 08:30] - Consolidate pr.yaml, main.yaml, release.yaml into single build.yaml

Author: Erick Bourgeois

Changed

  • .github/workflows/build.yaml: New consolidated workflow replacing the three separate files; triggers on pull_request, push to main, and release: published; uses if: at job and step level to gate event-specific behaviour
  • .github/workflows/pr.yaml: Deleted
  • .github/workflows/main.yaml: Deleted
  • .github/workflows/release.yaml: Deleted

Why

Three workflows shared the same build matrix, env vars, and most job logic, requiring the same fix to be applied in three places (e.g. the linker override, the attest job). A single file is easier to maintain and gives a complete picture of CI behaviour in one place.

Impact

  • Breaking change
  • Requires cluster rollout
  • Config change only
  • Documentation only

Key design decisions:

  • extract-version serves as the quality gate: it runs after all checks that apply to the current event (verify-commits, license-check, format) and all downstream jobs depend on it.
  • Docker metadata uses three separate docker/metadata-action steps gated by if:; docker/build-push-action concatenates all outputs and filters empty lines.
  • Cosign signing, Docker SBOM generation, sign-artifacts, SLSA provenance, and upload-release-assets are guarded by if: github.event_name == 'release'.
  • test and format/clippy are guarded by if: github.event_name == 'pull_request'.
  • trivy is guarded by if: github.event_name != 'pull_request'.
  • Artifact retention: PR/push = 1 day (two upload steps); release = default.

[2026-04-10 07:45] - Add GitHub artifact attestation job to all three CI/CD workflows

Author: Erick Bourgeois

Changed

  • .github/workflows/pr.yaml: Added id: docker_build to docker build step; added outputs: block to docker job (per-variant digests); added export-digest step; added attest job depending on docker and extract-version
  • .github/workflows/main.yaml: Same additions to docker job and new attest job
  • .github/workflows/release.yaml: Same additions to docker-release job and new attest job (depends on docker-release)

Why

GitHub's actions/attest-build-provenance generates a signed SLSA provenance attestation stored natively in GitHub Artifact Attestations and optionally pushed to the OCI registry alongside the image. This is queryable with gh attestation verify and complements the existing Cosign signatures in the release workflow. Requires the matrix digest-export pattern to pass per-image digests from a matrix job to a downstream job.

Impact

  • Breaking change
  • Requires cluster rollout
  • Config change only
  • Documentation only

[2026-04-08 17:30] - Add documentation build and GitHub Pages deployment workflow

Author: Erick Bourgeois

Changed

  • .github/workflows/docs.yaml: New workflow — builds MkDocs documentation (including make docs which runs cargo run --bin crddoc) and deploys to GitHub Pages on push to main; runs link checks on PRs

Why

The project has a full MkDocs documentation site under docs/ but no automated build or publishing pipeline. This workflow closes that gap by building on every relevant change, checking for broken links on PRs, and publishing to GitHub Pages on every merge to main.

Impact

  • Breaking change
  • Requires cluster rollout
  • Config change only
  • Documentation only

Note: GitHub Pages must be enabled in the repository settings (Settings → Pages → Source: GitHub Actions) for the deploy job to succeed. The poetry.lock file should be committed after the first poetry install run to improve cache efficiency and build reproducibility.

[2026-04-09 11:00] - Fix CI linker error caused by .cargo/config.toml on Linux runners

Author: Erick Bourgeois

Changed

  • .github/workflows/pr.yaml: Added CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER: cc and CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER: cc to the top-level env: block
  • .github/workflows/main.yaml: Same
  • .github/workflows/release.yaml: Same

Why

.cargo/config.toml specifies linker = "x86_64-unknown-linux-gnu-gcc" and linker = "aarch64-unknown-linux-gnu-gcc" for the respective targets — Homebrew cross-compilers needed when a macOS developer uses cargo build --target <linux-triple> locally (the Makefile fallback path). On Linux CI runners, cargo build --release with no explicit target resolves to the native triple (e.g., x86_64-unknown-linux-gnu on ubuntu-latest), which picks up the same override and fails because the cross-compiler is not installed on GitHub Actions runners. CARGO_TARGET_*_LINKER environment variables take precedence over config.toml, restoring cc (the system linker) in CI without modifying the config file.

Impact

  • Breaking change
  • Requires cluster rollout
  • Config change only
  • Documentation only

[2026-04-09 10:00] - Replace cross-compilation with native cargo builds in CI

Author: Erick Bourgeois

Changed

  • .github/workflows/pr.yaml: Replaced firestoned/github-actions/rust/setup-rust-build@v1.3.6 + build-binary@v1.3.6 + generate-sbom@v1.3.6 with dtolnay/rust-toolchain@stable + cargo build --release + cargo-cyclonedx; switched ARM64 build to ubuntu-24.04-arm native runner; updated artifact paths from target/$target/release/ to target/release/; fixed license-id: "MIT" → "Apache-2.0"
  • .github/workflows/main.yaml: Same build and license-check changes
  • .github/workflows/release.yaml: Same build and license-check changes; replaced setup-rust-build in package-deploy-manifests job with dtolnay/rust-toolchain@stable

Why

The firestoned/github-actions/rust/build-binary@v1.3.6 action internally sets -C linker=x86_64-unknown-linux-gnu-gcc, which is not installed on GitHub Actions ubuntu-latest runners, causing all build jobs to fail. Native cargo build --release on arch-appropriate runners eliminates cross-compilation entirely. License-id was stale after the FINOS Apache-2.0 migration.

Impact

  • Breaking change
  • Requires cluster rollout
  • Config change only
  • Documentation only

[2026-04-09 03:00] - Phase 2 (P2-4): leader election via kube-lease-manager

Author: Erick Bourgeois

Changed

  • Cargo.toml: Added kube-lease-manager = "0.11" dependency
  • src/constants.rs: Added DEFAULT_LEASE_NAME, DEFAULT_LEASE_DURATION_SECS, DEFAULT_LEASE_RENEW_DEADLINE_SECS, DEFAULT_LEASE_RETRY_PERIOD_SECS, DEFAULT_LEASE_GRACE_SECS, DEFAULT_LEASE_NAMESPACE constants
  • src/reconcilers/scheduled_machine.rs: Added is_leader: Arc<AtomicBool> to Context (defaults to true for backward-compatible single-instance mode); added leader guard in reconcile_guarded — non-leaders return Action::await_change() immediately
  • src/main.rs: Added enable_leader_election, lease_name, lease_namespace, lease_duration_secs, lease_renew_deadline_secs CLI args; when enabled, sets is_leader = false at startup and spawns a background kube-lease-manager task that flips is_leader on acquisition/loss
  • deploy/deployment/deployment.yaml: Fixed POD_NAME → CONTROLLER_POD_NAME env var (aligns with Context::new and leader election holder identity)
  • src/reconcilers/scheduled_machine_tests.rs: Added 2 TDD tests — test_context_new_defaults_is_leader_to_true and test_reconcile_guarded_awaits_change_when_not_leader
  • docs/src/operations/configuration.md: Added all leader election env vars, CLI args, Leader Election section, Lease RBAC rules

Why

Basel III HA (P2-4): a single-replica controller is a single point of failure. With ENABLE_LEADER_ELECTION=true and replicas: 2, only the lease holder reconciles resources. Standby replicas react within one LEASE_DURATION_SECONDS window on leader failure. Context::is_leader defaults to true so existing single-replica deployments continue without any config change.

Impact

  • Breaking change
  • Requires cluster rollout — set ENABLE_LEADER_ELECTION=true and replicas: 2; RBAC for leases already in clusterrole.yaml
  • Config change only
  • Documentation only

[2026-04-09 02:00] - Add SPDX license headers to all .github YAML files

Author: Erick Bourgeois

Changed

  • .github/ISSUE_TEMPLATE/bug_report.yml: Added # Copyright (c) 2025 Erick Bourgeois, finos + # SPDX-License-Identifier: Apache-2.0 header
  • .github/ISSUE_TEMPLATE/feature_request.yml: Same
  • .github/ISSUE_TEMPLATE/meeting_minutes.yml: Same
  • .github/ISSUE_TEMPLATE/support_question.yml: Same

Why

Supply-chain provenance and automated license scanning (NIST SA-4) require SPDX headers on all project-owned files. The three workflow files and both composite actions already had headers from P2-10; these four issue templates were the remaining .github/ YAML files without them. dco.yml was intentionally left untouched — it is managed by FINOS and carries an explicit "Do not edit" notice.

Impact

  • Breaking change
  • Requires cluster rollout
  • Config change only
  • Documentation only

[2026-04-09 01:00] - Phase 2 (P2-5): exponential back-off in error policy

Author: Erick Bourgeois

Changed

  • src/constants.rs: Added MAX_RECONCILE_RETRIES: u32 = 10 constant
  • src/reconcilers/scheduled_machine.rs: Added retry_counts: Arc<Mutex<HashMap<String, u32>>> to Context; updated Context::new to initialise it; updated reconcile_guarded to clear the retry count on successful reconciliation
  • src/reconcilers/helpers.rs: Added compute_backoff_secs(retry_count: u32) -> u64 (pure, capped exponential); replaced fixed-delay error_policy with retry-count-aware implementation that increments the per-resource counter and computes ERROR_REQUEUE_SECS * 2^n capped at MAX_BACKOFF_SECS
  • src/reconcilers/helpers_tests.rs: Added 5 TDD tests for compute_backoff_secs (base, doubling, cap at retry 4, cap at MAX_RECONCILE_RETRIES, large count)

Why

Basel III HA resilience (P2-5): a fixed 30 s retry interval can cause thundering-herd pressure when many resources fail simultaneously. Bounded exponential back-off distributes retry load while ensuring eventual recovery. Retry counts are cleared on success so transient failures do not permanently elevate delay. Aligns with NIST SI-2 flaw remediation by limiting retry storms.

Impact

  • Breaking change
  • Requires cluster rollout
  • Config change only
  • Documentation only

[2026-04-09 00:00] - Phase 2 (P2-3/P2-7): reconciliation correlation IDs and Condition status enum

Author: Erick Bourgeois

Changed

  • src/reconcilers/scheduled_machine.rs: Added generate_reconcile_id() — derives a short correlation ID from the resource's UID last segment + nanosecond hex timestamp; refactored reconcile_scheduled_machine into reconcile_guarded wrapped in a tracing::info_span! carrying reconcile_id, resource, and namespace — every log line in a reconciliation now carries these fields in JSON output (NIST AU-3 / SOX §404 P2-3)
  • src/reconcilers/scheduled_machine_tests.rs: Added 5 TDD tests for generate_reconcile_id() covering non-empty output, UID-last-segment prefix, hex timestamp suffix, unknown-fallback when no UID, and uniqueness across calls
  • src/crd.rs: Added condition_status_schema() and wired it to Condition.status via #[schemars(schema_with = "...")] — constrains the CRD field to enum: [True, False, Unknown] (NIST CM-5 / P2-7)
  • src/crd_tests.rs: Added 5 TDD tests for Condition.status schema enum: constraint exists, all three values present, and runtime Condition::new() still accepts string status unchanged
  • deploy/crds/scheduledmachine.yaml: Regenerated — Condition.status now has enum: [True, False, Unknown] in the CRD OpenAPI schema
  • docs/reference/api.md: Regenerated to reflect schema change

Why

  • P2-3: Every reconciliation now emits a unique reconcile_id on all log lines via a tracing span, enabling full end-to-end correlation in a SIEM or log aggregation platform. Closes the NIST AU-3 / SOX §404 correlation ID gap.
  • P2-7: The Condition.status field previously accepted any string; the CRD schema now enforces the Kubernetes-standard True/False/Unknown enum as required by NIST CM-5 configuration change control. Runtime behaviour is unchanged — the constraint is schema-only.

Impact

  • Breaking change
  • Requires cluster rollout — CRD must be reapplied (kubectl apply -f deploy/crds/scheduledmachine.yaml); existing CRs with valid status values are unaffected
  • Config change only
  • Documentation only

[2026-04-08 15:00] - Add Security section to MkDocs with Admission Validation guide

Author: Erick Bourgeois

Added

  • docs/src/security/index.md: New security section landing page — security posture at a glance table, compliance mapping summary, and links to sub-pages
  • docs/src/security/admission-validation.md: Comprehensive user-facing guide for the ValidatingAdmissionPolicy covering: VAP vs. webhook comparison table, Mermaid admission flow sequence diagram, full 13-rule reference table with per-rule detail and examples, deployment instructions, rollout strategy (Audit → Deny → AuditAndDeny), four concrete kubectl test examples, namespace scoping guidance, and Kubernetes version compatibility table
  • docs/mkdocs.yml: Added Security top-level nav section (between Advanced Topics and Developer Guide) containing Overview, Admission Validation, and Threat Model pages

Why

The ValidatingAdmissionPolicy deployed in the previous entry had no user-facing documentation. Operators need to know what is validated, how to deploy it, how to do a safe rollout, and how to test it. The new Security section also surfaces the threat model in the main navigation — previously it existed only in the repo but was not reachable from the docs site.

Impact

  • Breaking change
  • Requires cluster rollout
  • Config change only
  • Documentation only

[2026-04-08 14:00] - Phase 2 (P2-6/P2-8/P2-9/P2-10): eviction correctness, JSON logging, supply-chain provenance

Author: Erick Bourgeois

Changed

  • src/reconcilers/helpers.rs: Fixed P2-6 — evict_pod 429 PDB-blocked arm now returns Err(ReconcilerError::CapiError(...)) instead of silently returning Ok(()); log level raised from info to warn; doc comment updated to remove the incorrect "429 is not an error" statement
  • src/reconcilers/helpers_tests.rs: Added 5 TDD mock API tests for evict_pod covering: success (200), already-deleted (404 → Ok), PDB-blocked (429 → CapiError), server error (500 → CapiError), and forbidden (403 → CapiError)
  • src/main.rs: Wired P2-8 — added --log-format CLI arg mapped to RUST_LOG_FORMAT env var (default "json"); tracing subscriber now uses .json() layer for json and plain text layer for text/anything else
  • deploy/deployment/deployment.yaml: Changed RUST_LOG_FORMAT default from "text" to "json" so production pods emit structured JSON for SIEM ingestion
  • src/**/*.rs (all 18 files): Added P2-10 SPDX supply-chain provenance headers to every Rust source file: 
    // Copyright (c) 2025 Erick Bourgeois, RBC Capital Markets
// SPDX-License-Identifier: Apache-2.0

Why

  • P2-6: A PDB-blocked eviction (HTTP 429) was silently treated as success, causing the drain loop to believe the pod was evicted when it wasn't — a data-integrity bug that could leave a node non-empty. Now propagated as CapiError so the caller can decide to retry or abort.
  • P2-8: Structured JSON logging is required for SIEM ingestion and NIST AU-3 compliance; text format was only appropriate for local development.
  • P2-9: cargo-audit 0.22.0 was already running via firestoned/github-actions/rust/security-scan@v1.3.6 on all PRs and main — no code change required, marked ✅ in roadmap.
  • P2-10: SPDX headers enable automated license scanning and supply-chain provenance tracking per NIST SA-4.

Impact

  • Breaking change
  • Requires cluster rollout — RUST_LOG_FORMAT=json default; existing log parsers expecting plain text must be updated
  • Config change only
  • Documentation only

[2026-04-08 13:00] - Add ValidatingAdmissionPolicy for ScheduledMachine (NIST CM-5)

Author: Erick Bourgeois

Added

  • deploy/admission/validatingadmissionpolicy.yaml: ValidatingAdmissionPolicy with 13 CEL validation rules covering: clusterName non-empty; gracefulShutdownTimeout/nodeDrainTimeout duration format (^\d+[smh]$); cron XOR daysOfWeek/hoursOfDay mutual exclusivity; daysOfWeek day-name/range item format; hoursOfDay hour/range item format; bootstrapSpec/infrastructureSpec apiVersion namespaced-group requirement; bootstrap/infrastructure provider API group allowlist (mirrors ALLOWED_BOOTSTRAP_API_GROUPS / ALLOWED_INFRASTRUCTURE_API_GROUPS in src/constants.rs); bootstrapSpec.kind/infrastructureSpec.kind non-empty
  • deploy/admission/validatingadmissionpolicybinding.yaml: ValidatingAdmissionPolicyBinding with validationActions: [Deny] applied cluster-wide

Changed

  • docs/roadmaps/compliance-sox-basel3-nist.md: Marked P3-4, CM-5, and all CRD schema validation gaps as resolved; updated gap table and compliance control mapping
  • docs/roadmaps/project-roadmap-2026.md: Updated Phase 3.1 Admission Webhooks → Admission Validation; checked off all implemented rules; noted future mutating webhook and reference-existence check as separate items
  • docs/src/security/threat-model.md: Updated Deployment-Layer Controls table to reflect VAP deployed (was a recommendation)

Why

ValidatingAdmissionPolicy (Kubernetes ≥ 1.26) enforces spec constraints at API-server admission time without requiring a separate webhook server, TLS certificate, or additional binary. Closes the NIST CM-5 gap: invalid specs that previously reached the reconciler are now rejected before being persisted to etcd.

Impact

  • Breaking change
  • Requires cluster rollout — apply deploy/admission/ manifests; requires Kubernetes ≥ 1.26 (alpha), ≥ 1.28 (beta), ≥ 1.30 (GA)
  • Config change only
  • Documentation only

[2026-04-08 12:00] - Complete rustdoc coverage across all Rust source files

Author: Erick Bourgeois

Changed

  • src/reconcilers/helpers.rs: Expanded thin one-liner docs on all remaining functions — add_finalizer, handle_deletion, handle_kill_switch, check_grace_period_elapsed, update_phase_with_last_schedule, update_phase_with_grace_period, bootstrap_resource_name, infrastructure_resource_name, machine_resource_name, create_dynamic_resource, parse_api_version, remove_machine_from_cluster, should_evict_pod, evict_pod, and error_policy — with full /// docs covering purpose, behaviour details, and # Errors sections
  • src/bin/crdgen.rs: Replaced // comment header with //! module doc explaining purpose, usage, and regeneration requirement; added /// on main() with # Panics note
  • src/bin/crddoc.rs: Added //! module doc explaining purpose, usage, and implementation note about static-println generation vs schema-driven approach; added /// on main()

Why

All public items and binary entry points now have complete rustdoc coverage to satisfy the project's documentation standard (CLAUDE.md §Code Comments) and to provide clear in-IDE guidance for future contributors.

Impact

  • Breaking change
  • Requires cluster rollout
  • Config change only
  • Documentation only

[2026-04-08 00:03] - Phase 2 (P2-1/P2-2): Kubernetes Event audit trail and before/after phase logging

Author: Erick Bourgeois

Changed

  • src/reconcilers/scheduled_machine.rs: Added Recorder field to Context struct (created from Reporter with controller name and pod name); removed separate client/recorder args from all update_phase* call sites — now pass &ctx directly
  • src/reconcilers/helpers.rs: Added build_phase_transition_event() pure function that constructs a KubeEvent from phase transition parameters (Warning for Error/Terminated, Normal otherwise); updated update_phase(), update_phase_with_last_schedule(), and update_phase_with_grace_period() to accept &Context (replacing separate &Client + &Recorder params), log from → to phase transition at INFO level, and publish an immutable Kubernetes Event via the recorder (best-effort — failures emit WARN but do not abort the transition)
  • deploy/deployment/rbac/clusterrole.yaml: Added events.k8s.io / events create+patch rule alongside the existing core "" events rule (kube-rs Recorder uses the events.k8s.io/v1 API)
  • src/reconcilers/helpers_tests.rs: Added 7 unit tests for build_phase_transition_event() covering Normal/Warning event types, note format, unknown from-phase fallback, action field, and reason field

Why

P2-1 and P2-2 from the SOX/Basel III/NIST compliance roadmap. Every machine phase transition now writes an immutable Kubernetes Event visible via kubectl describe scheduledmachine <name>, providing an auditable record of state changes required by SOX §404 (immutable audit trail) and NIST AU-2/AU-3 (event recording and audit record content). Before/after logging closes the gap against AU-3 by making the previous phase explicit in each log line.

Impact

  • Breaking change
  • Requires cluster rollout
  • Config change only
  • Documentation only

[2026-04-08 00:02] - Phase 1 Compliance Remediation (SOX/Basel III/NIST SP 800-53)

Author: Erick Bourgeois

Changed

  • src/reconcilers/scheduled_machine.rs: Replaced 7 unwrap() calls with ok_or_else error propagation in all phase handlers (reconcile_inner, handle_pending_phase, handle_active_phase, handle_shutting_down_phase, handle_inactive_phase, handle_disabled_phase, handle_error_phase) — NIST SI-3, P1-1
  • src/reconcilers/helpers.rs: Replaced 3 unwrap() calls with ok_or_else error propagation in add_finalizer, handle_deletion, and handle_kill_switch — NIST SI-3, P1-1
  • src/metrics.rs: Replaced 11 .expect() panics with graceful fallback pattern using private helper functions (fallback_counter_vec, fallback_gauge, fallback_gauge_vec, fallback_histogram_vec); metrics initialization failures now log a warning and continue rather than crash — NIST SI-3, P1-2
  • deploy/deployment/networkpolicy.yaml: Created Kubernetes NetworkPolicy implementing NIST SC-7 boundary protection — ingress restricted to Prometheus scrape (port 8080, monitoring namespace only) and kubelet probes (port 8081); egress restricted to DNS (port 53) and Kubernetes API server (port 6443) — NIST SC-7, P1-3
  • src/main.rs: Explicitly applied K8S_API_TIMEOUT_SECS constant to kube::Config read_timeout and write_timeout fields to enforce connection timeouts against Kubernetes API server — Basel III operational resilience, P1-4

Why

Phase 1 of the SOX/Basel III/NIST SP 800-53 compliance remediation roadmap (docs/roadmaps/compliance-sox-basel3-nist.md). All unwrap()/expect() calls in production code paths represent potential uncontrolled panics that violate NIST SI-3 (Malicious Code Protection) and operational resilience requirements. The NetworkPolicy enforces least-privilege network access per NIST SC-7. Explicit API timeouts align with Basel III operational resilience requirements for bounded failure modes.

Impact

  • Breaking change
  • Requires cluster rollout
  • Config change only
  • Documentation only

[2026-04-08 00:01c] - Convert all remaining ASCII diagrams to Mermaid

Author: Erick Bourgeois

Changed

  • docs/src/concepts/schedules.md: Converted cron field reference ASCII art to flowchart LR Mermaid diagram (5 labelled field nodes)

Why

Project standard requires all diagrams to use Mermaid. This was the last remaining ASCII diagram across all docs.

Impact

  • Breaking change
  • Requires cluster rollout
  • Config change only
  • Documentation only

[2026-04-08 00:01b] - Convert threat model ASCII diagrams to Mermaid

Author: Erick Bourgeois

Changed

  • docs/src/security/threat-model.md: Converted Section 2 system overview ASCII art to flowchart TB Mermaid diagram; converted Section 4 trust boundaries text block to flowchart LR Mermaid diagram

Why

Project standard is Mermaid for all diagrams (consistent with architecture.md and other docs/src files).

Impact

  • Breaking change
  • Requires cluster rollout
  • Config change only
  • Documentation only

[2026-04-08 00:01] - Add threat model document

Author: Erick Bourgeois

Changed

  • docs/src/security/threat-model.md: New STRIDE threat model covering all controller components, trust boundaries, threat actors, 30+ threats with likelihood/impact ratings, full mitigations matrix, and 6 residual risk items with remediation guidance

Why

Regulatory requirement in a banking environment: all security-significant components must have a documented threat model traceable to identified controls. This also captures the rationale behind the security hardening changes made in the same session.

Impact

  • Breaking change
  • Requires cluster rollout
  • Config change only
  • Documentation only

[2026-04-08 00:00] - Add GitHub Actions CI/CD Workflows

Author: Erick Bourgeois

Changed

  • .github/workflows/pr.yaml: Pull Request CI — lint, test, Linux binary builds, Docker build/push, security scan
  • .github/workflows/main.yaml: Main branch CI/CD — builds, Docker push (latest + date tags), security scan, Trivy container scan
  • .github/workflows/release.yaml: Release workflow — versioned Docker images with Cosign signing, SLSA provenance, binary signing, deploy manifest packaging, release asset upload

Why

Establish baseline CI/CD pipeline for the 5-spot operator using the same firestoned GitHub Actions patterns as bindy. Linux-only builds (x86_64 + ARM64) since this is a Kubernetes operator.

Impact

  • Breaking change
  • Requires cluster rollout
  • Config change only
  • Documentation only

[2026-04-08 00:00] - Security hardening: namespace isolation, input validation, RBAC narrowing

Author: Erick Bourgeois

Changed

  • src/crd.rs: Removed namespace field from EmbeddedResource — bootstrap and infrastructure resources are now always created in the ScheduledMachine's own namespace, preventing cross-namespace attacks
  • src/crd.rs: Added timezone_schema() with maxLength: 64 and character-class pattern constraint to block log injection via the timezone field
  • src/reconcilers/helpers.rs: Fixed integer overflow in parse_duration() — now uses checked_mul and rejects durations exceeding 24 hours (MAX_DURATION_SECS)
  • src/reconcilers/helpers.rs: Added validate_labels() — rejects label/annotation keys using reserved prefixes (kubernetes.io/, k8s.io/, cluster.x-k8s.io/, 5spot.finos.org/) before merging into CAPI Machine resources
  • src/reconcilers/helpers.rs: Added validate_api_group() — enforces an allowlist of permitted API groups for bootstrap and infrastructure embedded resources; blocks core Kubernetes APIs (v1, rbac.authorization.k8s.io/v1, etc.)
  • src/reconcilers/helpers.rs: Wrapped remove_machine_from_cluster in tokio::time::timeout inside handle_deletion — finalizer cleanup now has a hard 10-minute deadline, preventing indefinite namespace deletion blocks
  • src/reconcilers/scheduled_machine.rs: Added ValidationError and TimeoutError variants to ReconcilerError
  • src/constants.rs: Added MAX_DURATION_SECS, MAX_TIMEZONE_LEN, FINALIZER_CLEANUP_TIMEOUT_SECS, RESERVED_LABEL_PREFIXES, ALLOWED_BOOTSTRAP_API_GROUPS, ALLOWED_INFRASTRUCTURE_API_GROUPS
  • deploy/deployment/rbac/clusterrole.yaml: Narrowed k0smotron.io resources from wildcard to explicit list (k0sworkerconfigs, remotemachines and their /status subresources)
  • src/reconcilers/helpers_tests.rs: New test file — 25 security-focused tests covering overflow protection, reserved label rejection, and API group allowlist enforcement
  • src/crd_tests.rs, src/reconcilers/scheduled_machine_tests.rs: Removed namespace: None from EmbeddedResource test fixtures (field removed)

Why

Comprehensive security audit identified: cross-namespace resource creation via user-controlled namespace overrides, integer overflow in duration parsing, label injection into CAPI resources, unbounded apiVersion/kind inputs, and missing finalizer cleanup timeouts. These are now all addressed to meet zero-trust security requirements for a regulated banking environment.

Impact

  • Breaking change — EmbeddedResource.namespace field removed from CRD schema (existing CRs with this field: Kubernetes ignores unknown fields, no action required)
  • Requires cluster rollout — CRDs must be regenerated (regen-crds skill) before deploying
  • Config change only
  • Documentation only

[2026-03-21 12:00] - Adopt .claude Skills Structure

Author: Erick Bourgeois

Changed

  • Created .claude/ directory with SKILL.md and CHANGELOG.md
  • Adopted skills-based workflow from bindy project
  • Updated documentation structure for better organization

Why

Standardize project instructions and skills across projects, improving consistency and making procedures reusable and discoverable.

Impact

  • Breaking change
  • Requires cluster rollout
  • Config change only
  • Documentation only

[2026-01-18 12:00] - Add VMware cloud-init preparation script

Author: Unknown

Added

  • scripts/install-cloud-init.sh: Linux-only script to convert VMDK→raw, mount LVM with conflict-safe handling, chroot to install cloud-init and open-vm-tools, optional initramfs rebuild, raw→streamOptimized VMDK, and import as vSphere template via govc.

Why

Enable automated preparation and deployment of a cloud-init-enabled RHEL image on a VMware VM. Credentials and vSphere target configuration are provided via environment variables to avoid storing secrets in code.

Impact

  • Breaking change
  • Requires cluster rollout
  • Config change only
  • Documentation only

[2026-01-18 17:45] - Harden govc VM existence check in upload script

Author: Unknown

Changed

  • scripts/install-cloud-init.sh: Replaced fragile govc vm.info-based existence check with robust govc find -type m -name <name> logic; iterates over matched inventory paths, converts templates to VMs when needed, and destroys them before import.

Why

govc vm.info can return exit code 0 with no output, leading to false positives. Using govc find and inspecting inventory paths provides reliable detection of existing VMs/templates with the target name and avoids confusing "not found" errors.

Impact

  • Breaking change
  • Requires cluster rollout
  • Config change only
  • Documentation only

[2026-01-18 18:30] - Simplify LVM VG handling with isolated system directory

Author: Unknown

Changed

  • scripts/install-cloud-init.sh: Use LVM_SYSTEM_DIR to isolate loop device LVM metadata to a separate directory (/tmp/lvm-loop-$$); use temporary VG name (vg00_loop) if host has same VG name to avoid device-mapper conflicts in /dev/mapper/.

Why

Device-mapper device names in /dev/mapper/ are global at the kernel level, even with isolated LVM metadata via LVM_SYSTEM_DIR. If both host and loop device have vg00 with LVs named root, var, etc., device-mapper refuses to create duplicate devices ("Device or resource busy"). By using vgimportclone -n vg00_loop when a conflict exists, we give the loop device VG a unique name for device-mapper while keeping metadata isolated. No rename needed after deactivation since the isolated metadata directory is simply deleted.

Impact

  • Breaking change
  • Requires cluster rollout
  • Config change only
  • Documentation only