1. Child-cluster Node API support (the bulk of the branch)

Routes Node/Pod operations to a CAPI/k0smotron workload (child) cluster while the ScheduledMachine lives on the management cluster.

- New CRD field spec.kubeconfigSecretRef (src/crd.rs)
— optional reference to a same-namespace <clusterName>-kubeconfig Secret; cross-namespace refs forbidden by design.
- src/reconcilers/child_client.rs (+tests, ~1,400 lines)
— resolves the right kube::Client (management vs child), caches by Secret with LRU + resourceVersion-based rotation.
- src/reconcilers/child_watch.rs (+tests, untracked, ~520 lines)
— one Node watcher per child cluster; maps child Node events back to ScheduledMachine reconciles via Controller::reconcile_on.
- tests/integration_child_kubeconfig.rs (untracked, 416 lines)
— end-to-end kubeconfig resolution tests.
- Wiring/support: src/main.rs (controller plumbing), src/metrics.rs (Phase-2 child-cluster metrics), new error variants in the ScheduledMachine lives on the management cluster.

- New CRD field spec.kubeconfigSecretRef (src/crd.rs)
— optional reference to a same-namespace <clusterName>-kubeconfig Secret; cross-namespace refs resourceVersion-based rotation.
- src/reconcilers/child_watch.rs (+tests, untracked, ~520 lines)
— one Node watcher per child cluster; maps child Node events back to ScheduledMachine reconciles via Controller::reconcile_on.
- tests/integration_child_kubeconfig.rs (untracked, 416 lines)
— end-to-end kubeconfig resolution tests.
- Wiring/support: src/main.rs (controller plumbing), src/metrics.rs (Phase-2 child-cluster metrics), new error variants in scheduled_machine.rs, Cargo.toml deps (kube unstable-runtime, tokio-stream, base64), and docs docs/src/concepts/child-cluster-kubeconfig.md + examples/scheduledmachine-child-cluster.yaml.

2. Security hardening

- RBAC validation of bootstrapSpec/infrastructureSpec
— user anti-escalation via VAP authorizer rules (13a/13b) + controller-SA SelfSubjectAccessReview pre-flight (ensure_can_create()); new PermissionDenied error + metric.
- Embedded metadata enforcement
— loudly reject metadata.name/metadata.namespace (VAP rules 13c–13f + runtime validate_embedded_metadata()),
  while newly supporting metadata.labels/annotations (reserved-prefix-checked, controller labels win).
  Required making the embedded metadata preserve-unknown so the field isn't pruned before rejection.
- Updated deploy/admission/validatingadmissionpolicy.yaml, regenerated CRD + api.md, threat-model (T1 hardened), admission-validation doc, and SCHEDULED_MACHINE_LABEL constant.

Signed-off-by: Erick Bourgeois <erick@jeb.ca>

Author: ebourgeois

.claude/CHANGELOG.md

@@ -6,7 +6,8 @@ name: Documentation
 on:
   # Build-only triggers — exercise the docs pipeline so breakage is
   # caught at PR / merge time, but do NOT publish to GitHub Pages.
-  # Deployment is gated on the `release` event below.
+  # Deployment is gated on the Build workflow completing successfully
+  # for a release (see workflow_run trigger and deploy job below).
   push:
     branches:
       - main
@@ -19,18 +20,25 @@ on:
       - 'docs/**'
       - 'src/**/*.rs'
       - '.github/workflows/docs.yaml'
-  # Deploy trigger — fires when an operator publishes a GitHub Release.
-  # The docs site published to Pages is what end users see, so we tie
-  # publish cadence to the release cadence: every release ships matching
-  # docs, and intermediate main commits don't churn the live site.
-  release:
-    types: [published]
+  # Deploy trigger — chained to the Build workflow's completion rather
+  # than the raw `release: published` event so docs only ship when the
+  # release's binaries / container images / signatures actually built
+  # cleanly. The deploy job below filters strictly on
+  # workflow_run.event == 'release' and conclusion == 'success';
+  # PR / push-to-main Build completions still fire this workflow as a
+  # redundant docs-pipeline sanity check, but never deploy.
+  workflow_run:
+    workflows: ["Build"]
+    types: [completed]
   workflow_dispatch:
 
+# Least-privilege default for all jobs. `id-token: write` (OIDC, needed only by
+# actions/deploy-pages) is NOT granted here — it is scoped to the deploy job
+# alone, so the build job that checks out code never holds it. `pages: write`
+# is required by the release-only Setup Pages step in the build job.
 permissions:
   contents: read
   pages: write
-  id-token: write
 
 # Build runs on every PR / push and can race with itself — cancel
 # in-progress to keep CI from queuing redundant builds. Release deploys
@@ -39,8 +47,8 @@ permissions:
 # the `cancel-in-progress` expression below disables cancellation only
 # for the release event.
 concurrency:
-  group: pages-${{ github.event_name == 'release' && github.event.release.tag_name || github.ref }}
-  cancel-in-progress: ${{ github.event_name != 'release' }}
+  group: pages-${{ github.event_name == 'workflow_run' && github.event.workflow_run.head_branch || github.ref }}
+  cancel-in-progress: ${{ !(github.event_name == 'workflow_run' && github.event.workflow_run.event == 'release') }}
 
 env:
   CARGO_TERM_COLOR: always
@@ -87,11 +95,36 @@ jobs:
     name: 📚 Build Documentation
     needs: [calm-validate, calm-diagrams]
     runs-on: ubuntu-latest
+    # SECURITY — pwn-request guard. `workflow_run` is a privileged trigger: it
+    # runs in the default-branch context with the repo's secrets and this
+    # workflow's write permissions, even when the triggering Build came from a
+    # fork PR. Checking out `workflow_run.head_sha` and running build steps
+    # (make docs / poetry install / npm install) for a PR Build would therefore
+    # execute untrusted code in a privileged context (CodeQL "Checkout of
+    # untrusted code in a privileged context").
+    #
+    # Gate the job so the workflow_run path runs ONLY for a SUCCESSFUL RELEASE
+    # Build — where head_sha is a trusted release-tag commit on the protected
+    # branch. PR / push / workflow_dispatch builds still run (in their own
+    # non-privileged contexts) and already validate the docs pipeline, so no
+    # coverage is lost; the previous "redundant sanity check on PR Build
+    # completions" was exactly the untrusted path and is now removed.
+    if: >-
+      github.event_name != 'workflow_run' ||
+      (github.event.workflow_run.event == 'release' &&
+       github.event.workflow_run.conclusion == 'success')
     steps:
       - name: Checkout repository
         uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
         with:
           fetch-depth: 0  # Full history for git-revision-date-localized plugin
+          # workflow_run executes in the default-branch context (main),
+          # so without an explicit ref the docs would be built from main
+          # even on a release-tagged Build. Pin to the SHA the Build ran
+          # against so released docs reflect released code. For other
+          # events (push / pull_request / workflow_dispatch) fall back to
+          # the workflow's triggering ref.
+          ref: ${{ github.event_name == 'workflow_run' && github.event.workflow_run.head_sha || github.ref }}
 
       - name: Download rendered CALM diagrams
         uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
@@ -132,8 +165,14 @@ jobs:
           cd docs
           poetry config virtualenvs.in-project true
 
-      - name: Cache Poetry dependencies
-        uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
+      # SECURITY — cache poisoning. RESTORE the cache in every context but only
+      # SAVE it from a trusted push to the default branch (below). This stops an
+      # untrusted PR (or any non-default-branch run) from writing files into a
+      # cache that the default branch later restores (CodeQL "Cache Poisoning
+      # via caching of untrusted files").
+      - name: Restore Poetry cache
+        id: poetry-cache
+        uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
         with:
           path: docs/.venv
           key: ${{ runner.os }}-poetry-${{ hashFiles('docs/pyproject.toml') }}
@@ -145,8 +184,18 @@ jobs:
           cd docs
           poetry install --no-interaction --no-ansi
 
-      - name: Cache cargo build
-        uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
+      - name: Save Poetry cache
+        if: github.event_name == 'push' && github.ref == 'refs/heads/main'
+        uses: actions/cache/save@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
+        with:
+          path: docs/.venv
+          key: ${{ steps.poetry-cache.outputs.cache-primary-key }}
+
+      # SECURITY — cache poisoning (see Poetry cache note above). Restore
+      # everywhere; save only from a trusted default-branch push.
+      - name: Restore cargo build cache
+        id: cargo-cache
+        uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
         with:
           path: |
             ~/.cargo/registry
@@ -163,6 +212,16 @@ jobs:
           SKIP_CALM_DIAGRAMS: "1"
         run: make docs
 
+      - name: Save cargo build cache
+        if: github.event_name == 'push' && github.ref == 'refs/heads/main'
+        uses: actions/cache/save@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
+        with:
+          path: |
+            ~/.cargo/registry
+            ~/.cargo/git
+            target
+          key: ${{ steps.cargo-cache.outputs.cache-primary-key }}
+
       - name: Check for broken links
         if: github.event_name == 'pull_request'
         continue-on-error: true
@@ -173,28 +232,38 @@ jobs:
           npm install -g "linkinator@${LINKINATOR_VERSION}"
           linkinator docs/site/ --recurse --skip "rustdoc/.*" --verbosity error
 
-      # Pages setup + artifact upload only happen when the trigger is a
-      # release publication. PR / push runs build the site (proving it
-      # compiles cleanly) but never produce a deploy artifact.
+      # Pages setup + artifact upload only happen when this workflow was
+      # triggered by a successful Build run for a release. PR / push runs
+      # (and Build completions for non-release events) build the site to
+      # prove it compiles cleanly, but never produce a deploy artifact.
       - name: Setup Pages
-        if: github.event_name == 'release'
+        if: github.event_name == 'workflow_run' && github.event.workflow_run.event == 'release' && github.event.workflow_run.conclusion == 'success'
         uses: actions/configure-pages@45bfe0192ca1faeb007ade9deae92b16b8254a0d # v6.0.0
 
       - name: Upload Pages artifact
-        if: github.event_name == 'release'
+        if: github.event_name == 'workflow_run' && github.event.workflow_run.event == 'release' && github.event.workflow_run.conclusion == 'success'
         uses: actions/upload-pages-artifact@fc324d3547104276b827a68afc52ff2a11cc49c9 # v5.0.0
         with:
           path: docs/site
 
   deploy:
     name: 🚀 Deploy to GitHub Pages
-    # Gated on the `release: published` event — a main push or PR build
-    # validates the docs but does NOT publish them. The published site
-    # tracks release cadence so end users see a stable, versioned doc
-    # set rather than every intermediate main commit.
-    if: github.event_name == 'release'
+    # Gated on the Build workflow finishing successfully for a release.
+    # This is stricter than the prior `release: published` trigger:
+    # if Build fails (e.g., a binary won't compile, a Cosign signature
+    # step errors, an SBOM upload breaks), docs no longer publish for
+    # that tag. PR / push-to-main runs and Build completions whose
+    # triggering event was not a release validate the docs pipeline but
+    # do NOT publish.
+    if: github.event_name == 'workflow_run' && github.event.workflow_run.event == 'release' && github.event.workflow_run.conclusion == 'success'
     needs: build
     runs-on: ubuntu-latest
+    # OIDC token for actions/deploy-pages is scoped here, not workflow-wide,
+    # so only this deploy job (which runs no untrusted code) can mint it.
+    permissions:
+      contents: read
+      pages: write
+      id-token: write
     environment:
       name: github-pages
       url: ${{ steps.deployment.outputs.page_url }}

.github/workflows/docs.yaml

@@ -11,3 +11,5 @@ __pycache__/
 # Generated CALM diagrams (rendered by `make calm-diagrams` / CI).
 # Source: docs/architecture/calm/templates/mermaid/*.md.hbs
 /docs/src/architecture/
+
+.DS_Store

.gitignore

@@ -29,10 +29,19 @@ name = "auto-vex-reachability"
 path = "src/bin/auto_vex_reachability.rs"
 
 [dependencies]
-kube = { version = "3.1", features = ["runtime", "derive", "client", "ws"] }
+# `unstable-runtime` is opt-in for `Controller::reconcile_on`, which we
+# use in main.rs to feed per-child-cluster Node-watch events into the
+# Controller. The feature is "unstable" in the SemVer sense (the API
+# may change between minor kube-runtime versions) but is the canonical
+# way to push external triggers into the controller loop.
+kube = { version = "3.1", features = ["runtime", "derive", "client", "ws", "unstable-runtime"] }
 kube-lease-manager = "0.11"
 k8s-openapi = { version = "0.27", features = ["latest", "schemars"] }
 tokio = { version = "1", features = ["full"] }
+# `ReceiverStream` adapter so an `mpsc::Receiver<ObjectRef<…>>` can be
+# fed into `Controller::reconcile_on`. Used by main.rs to plumb
+# child-cluster Node-watch events back into the controller.
+tokio-stream = "0.1"
 serde = { version = "1", features = ["derive"] }
 serde_json = "1"
 serde_yaml = "0.9"
@@ -79,6 +88,10 @@ test-log = "0.2"
 tempfile = "3"
 tower-test = "0.4"
 http = "1"
+# Used by src/reconcilers/child_client_tests.rs to build base64-encoded
+# Secret payloads for the tower-test mock server. base64 0.22 is already
+# in the dependency graph transitively (kube / k8s-openapi pull it).
+base64 = "0.22"
 
 [profile.release]
 opt-level = 3

Cargo.lock

@@ -32,6 +32,21 @@ spec:
         resources: ["scheduledmachines"]
         operations: ["CREATE", "UPDATE"]
 
+  # Derived values reused by the RBAC authorizer checks below. The group is the
+  # part of apiVersion before '/'; the resource is the naive lowercase plural of
+  # kind (lowerAscii + 's'). This mirrors resource_plural() in
+  # src/reconcilers/helpers.rs so the permission checked here is exactly the one
+  # the controller will exercise when it creates the resource.
+  variables:
+    - name: bootstrapGroup
+      expression: "object.spec.bootstrapSpec.apiVersion.split('/')[0]"
+    - name: bootstrapResource
+      expression: "object.spec.bootstrapSpec.kind.lowerAscii() + 's'"
+    - name: infraGroup
+      expression: "object.spec.infrastructureSpec.apiVersion.split('/')[0]"
+    - name: infraResource
+      expression: "object.spec.infrastructureSpec.kind.lowerAscii() + 's'"
+
   validations:
 
     # ── 1. clusterName: non-empty ────────────────────────────────────────────
@@ -196,6 +211,82 @@ spec:
       message: "spec.infrastructureSpec.kind must not be empty"
       reason: Invalid
 
+    # ── 13c–13f. Embedded metadata: name/namespace are controller-owned ───────
+    # The controller owns the created resource's identity: it always names the
+    # bootstrap/infrastructure resource after the ScheduledMachine and creates
+    # it in the SM's own namespace (cross-namespace creation is forbidden —
+    # threat T1). A user-supplied metadata.name/metadata.namespace is therefore
+    # rejected, not silently ignored.
+    #
+    # NOTE: this only works because EmbeddedResource's `metadata` is marked
+    # `x-kubernetes-preserve-unknown-fields: true` in src/crd.rs. Without it the
+    # API server PRUNES unknown fields before admission policies run, so the
+    # field would be gone before this rule could see it. The runtime
+    # `validate_embedded_metadata()` in src/reconcilers/helpers.rs enforces the
+    # same rule as defence-in-depth. Only metadata.labels / metadata.annotations
+    # are user-settable (and are reserved-prefix-checked at reconcile time).
+    - expression: "!has(object.spec.bootstrapSpec.metadata) || !has(object.spec.bootstrapSpec.metadata.namespace)"
+      message: >-
+        spec.bootstrapSpec.metadata.namespace is not permitted — the 5Spot
+        controller always creates the bootstrap resource in the
+        ScheduledMachine's own namespace
+      reason: Forbidden
+    - expression: "!has(object.spec.bootstrapSpec.metadata) || !has(object.spec.bootstrapSpec.metadata.name)"
+      message: >-
+        spec.bootstrapSpec.metadata.name is not permitted — the 5Spot controller
+        names the bootstrap resource after the ScheduledMachine
+      reason: Forbidden
+    - expression: "!has(object.spec.infrastructureSpec.metadata) || !has(object.spec.infrastructureSpec.metadata.namespace)"
+      message: >-
+        spec.infrastructureSpec.metadata.namespace is not permitted — the 5Spot
+        controller always creates the infrastructure resource in the
+        ScheduledMachine's own namespace
+      reason: Forbidden
+    - expression: "!has(object.spec.infrastructureSpec.metadata) || !has(object.spec.infrastructureSpec.metadata.name)"
+      message: >-
+        spec.infrastructureSpec.metadata.name is not permitted — the 5Spot
+        controller names the infrastructure resource after the ScheduledMachine
+      reason: Forbidden
+
+    # ── 13a. RBAC: requesting user may create the bootstrapSpec resource ──────
+    # Privilege-escalation guard. The 5Spot controller runs with broad RBAC so
+    # it can create bootstrap/infrastructure/Machine objects on the user's
+    # behalf. Without this check, a user able to create a ScheduledMachine but
+    # NOT a K0sWorkerConfig could have the controller create one for them —
+    # escalating through the controller. We require the *requesting user* to
+    # independently hold `create` on the embedded bootstrap GVK in the target
+    # namespace, mirroring how CAPI's own webhooks gate templated resources.
+    # The controller-side equivalent (its own service account) is enforced at
+    # reconcile time by ensure_can_create() in src/reconcilers/helpers.rs.
+    - expression: >-
+        authorizer.group(variables.bootstrapGroup)
+          .resource(variables.bootstrapResource)
+          .namespace(object.metadata.namespace)
+          .check('create')
+          .allowed()
+      message: >-
+        user is not permitted to create the spec.bootstrapSpec resource type —
+        creating a ScheduledMachine requires 'create' permission on the
+        embedded bootstrap resource (RBAC) to prevent privilege escalation
+        through the 5Spot controller
+      reason: Forbidden
+
+    # ── 13b. RBAC: requesting user may create the infrastructureSpec resource ──
+    # Same privilege-escalation guard as 13a, applied to the embedded
+    # infrastructure GVK (e.g. RemoteMachine).
+    - expression: >-
+        authorizer.group(variables.infraGroup)
+          .resource(variables.infraResource)
+          .namespace(object.metadata.namespace)
+          .check('create')
+          .allowed()
+      message: >-
+        user is not permitted to create the spec.infrastructureSpec resource
+        type — creating a ScheduledMachine requires 'create' permission on the
+        embedded infrastructure resource (RBAC) to prevent privilege escalation
+        through the 5Spot controller
+      reason: Forbidden
+
     # ── 14. nodeTaints: key format (RFC-1123 qualified name) ──────────────────
     # Mirrors validate_taint_key() in src/crd.rs. The optional "<prefix>/" part
     # is a DNS subdomain; the name part is a qualified name. Both capped at 63