1. Child-cluster Node API support (the bulk of the branch) (#71)

Routes Node/Pod operations to a CAPI/k0smotron workload (child) cluster while the ScheduledMachine lives on the management cluster.

- New CRD field spec.kubeconfigSecretRef (src/crd.rs)
— optional reference to a same-namespace <clusterName>-kubeconfig Secret; cross-namespace refs forbidden by design.
- src/reconcilers/child_client.rs (+tests, ~1,400 lines)
— resolves the right kube::Client (management vs child), caches by Secret with LRU + resourceVersion-based rotation.
- src/reconcilers/child_watch.rs (+tests, untracked, ~520 lines)
— one Node watcher per child cluster; maps child Node events back to ScheduledMachine reconciles via Controller::reconcile_on.
- tests/integration_child_kubeconfig.rs (untracked, 416 lines)
— end-to-end kubeconfig resolution tests.
- Wiring/support: src/main.rs (controller plumbing), src/metrics.rs (Phase-2 child-cluster metrics), new error variants in the ScheduledMachine lives on the management cluster.

- New CRD field spec.kubeconfigSecretRef (src/crd.rs)
— optional reference to a same-namespace <clusterName>-kubeconfig Secret; cross-namespace refs resourceVersion-based rotation.
- src/reconcilers/child_watch.rs (+tests, untracked, ~520 lines)
— one Node watcher per child cluster; maps child Node events back to ScheduledMachine reconciles via Controller::reconcile_on.
- tests/integration_child_kubeconfig.rs (untracked, 416 lines)
— end-to-end kubeconfig resolution tests.
- Wiring/support: src/main.rs (controller plumbing), src/metrics.rs (Phase-2 child-cluster metrics), new error variants in scheduled_machine.rs, Cargo.toml deps (kube unstable-runtime, tokio-stream, base64), and docs docs/src/concepts/child-cluster-kubeconfig.md + examples/scheduledmachine-child-cluster.yaml.

2. Security hardening

- RBAC validation of bootstrapSpec/infrastructureSpec
— user anti-escalation via VAP authorizer rules (13a/13b) + controller-SA SelfSubjectAccessReview pre-flight (ensure_can_create()); new PermissionDenied error + metric.
- Embedded metadata enforcement
— loudly reject metadata.name/metadata.namespace (VAP rules 13c–13f + runtime validate_embedded_metadata()),
  while newly supporting metadata.labels/annotations (reserved-prefix-checked, controller labels win).
  Required making the embedded metadata preserve-unknown so the field isn't pruned before rejection.
- Updated deploy/admission/validatingadmissionpolicy.yaml, regenerated CRD + api.md, threat-model (T1 hardened), admission-validation doc, and SCHEDULED_MACHINE_LABEL constant.

Signed-off-by: Erick Bourgeois <erick@jeb.ca>

Author: ebourgeois

.claude/CHANGELOG.md

@@ -11,3 +11,5 @@ __pycache__/
 # Generated CALM diagrams (rendered by `make calm-diagrams` / CI).
 # Source: docs/architecture/calm/templates/mermaid/*.md.hbs
 /docs/src/architecture/
+
+.DS_Store

.gitignore

@@ -29,10 +29,19 @@ name = "auto-vex-reachability"
 path = "src/bin/auto_vex_reachability.rs"
 
 [dependencies]
-kube = { version = "3.1", features = ["runtime", "derive", "client", "ws"] }
+# `unstable-runtime` is opt-in for `Controller::reconcile_on`, which we
+# use in main.rs to feed per-child-cluster Node-watch events into the
+# Controller. The feature is "unstable" in the SemVer sense (the API
+# may change between minor kube-runtime versions) but is the canonical
+# way to push external triggers into the controller loop.
+kube = { version = "3.1", features = ["runtime", "derive", "client", "ws", "unstable-runtime"] }
 kube-lease-manager = "0.11"
 k8s-openapi = { version = "0.27", features = ["latest", "schemars"] }
 tokio = { version = "1", features = ["full"] }
+# `ReceiverStream` adapter so an `mpsc::Receiver<ObjectRef<…>>` can be
+# fed into `Controller::reconcile_on`. Used by main.rs to plumb
+# child-cluster Node-watch events back into the controller.
+tokio-stream = "0.1"
 serde = { version = "1", features = ["derive"] }
 serde_json = "1"
 serde_yaml = "0.9"
@@ -79,6 +88,10 @@ test-log = "0.2"
 tempfile = "3"
 tower-test = "0.4"
 http = "1"
+# Used by src/reconcilers/child_client_tests.rs to build base64-encoded
+# Secret payloads for the tower-test mock server. base64 0.22 is already
+# in the dependency graph transitively (kube / k8s-openapi pull it).
+base64 = "0.22"
 
 [profile.release]
 opt-level = 3

Cargo.lock

@@ -32,6 +32,21 @@ spec:
         resources: ["scheduledmachines"]
         operations: ["CREATE", "UPDATE"]
 
+  # Derived values reused by the RBAC authorizer checks below. The group is the
+  # part of apiVersion before '/'; the resource is the naive lowercase plural of
+  # kind (lowerAscii + 's'). This mirrors resource_plural() in
+  # src/reconcilers/helpers.rs so the permission checked here is exactly the one
+  # the controller will exercise when it creates the resource.
+  variables:
+    - name: bootstrapGroup
+      expression: "object.spec.bootstrapSpec.apiVersion.split('/')[0]"
+    - name: bootstrapResource
+      expression: "object.spec.bootstrapSpec.kind.lowerAscii() + 's'"
+    - name: infraGroup
+      expression: "object.spec.infrastructureSpec.apiVersion.split('/')[0]"
+    - name: infraResource
+      expression: "object.spec.infrastructureSpec.kind.lowerAscii() + 's'"
+
   validations:
 
     # ── 1. clusterName: non-empty ────────────────────────────────────────────
@@ -196,6 +211,82 @@ spec:
       message: "spec.infrastructureSpec.kind must not be empty"
       reason: Invalid
 
+    # ── 13c–13f. Embedded metadata: name/namespace are controller-owned ───────
+    # The controller owns the created resource's identity: it always names the
+    # bootstrap/infrastructure resource after the ScheduledMachine and creates
+    # it in the SM's own namespace (cross-namespace creation is forbidden —
+    # threat T1). A user-supplied metadata.name/metadata.namespace is therefore
+    # rejected, not silently ignored.
+    #
+    # NOTE: this only works because EmbeddedResource's `metadata` is marked
+    # `x-kubernetes-preserve-unknown-fields: true` in src/crd.rs. Without it the
+    # API server PRUNES unknown fields before admission policies run, so the
+    # field would be gone before this rule could see it. The runtime
+    # `validate_embedded_metadata()` in src/reconcilers/helpers.rs enforces the
+    # same rule as defence-in-depth. Only metadata.labels / metadata.annotations
+    # are user-settable (and are reserved-prefix-checked at reconcile time).
+    - expression: "!has(object.spec.bootstrapSpec.metadata) || !has(object.spec.bootstrapSpec.metadata.namespace)"
+      message: >-
+        spec.bootstrapSpec.metadata.namespace is not permitted — the 5Spot
+        controller always creates the bootstrap resource in the
+        ScheduledMachine's own namespace
+      reason: Forbidden
+    - expression: "!has(object.spec.bootstrapSpec.metadata) || !has(object.spec.bootstrapSpec.metadata.name)"
+      message: >-
+        spec.bootstrapSpec.metadata.name is not permitted — the 5Spot controller
+        names the bootstrap resource after the ScheduledMachine
+      reason: Forbidden
+    - expression: "!has(object.spec.infrastructureSpec.metadata) || !has(object.spec.infrastructureSpec.metadata.namespace)"
+      message: >-
+        spec.infrastructureSpec.metadata.namespace is not permitted — the 5Spot
+        controller always creates the infrastructure resource in the
+        ScheduledMachine's own namespace
+      reason: Forbidden
+    - expression: "!has(object.spec.infrastructureSpec.metadata) || !has(object.spec.infrastructureSpec.metadata.name)"
+      message: >-
+        spec.infrastructureSpec.metadata.name is not permitted — the 5Spot
+        controller names the infrastructure resource after the ScheduledMachine
+      reason: Forbidden
+
+    # ── 13a. RBAC: requesting user may create the bootstrapSpec resource ──────
+    # Privilege-escalation guard. The 5Spot controller runs with broad RBAC so
+    # it can create bootstrap/infrastructure/Machine objects on the user's
+    # behalf. Without this check, a user able to create a ScheduledMachine but
+    # NOT a K0sWorkerConfig could have the controller create one for them —
+    # escalating through the controller. We require the *requesting user* to
+    # independently hold `create` on the embedded bootstrap GVK in the target
+    # namespace, mirroring how CAPI's own webhooks gate templated resources.
+    # The controller-side equivalent (its own service account) is enforced at
+    # reconcile time by ensure_can_create() in src/reconcilers/helpers.rs.
+    - expression: >-
+        authorizer.group(variables.bootstrapGroup)
+          .resource(variables.bootstrapResource)
+          .namespace(object.metadata.namespace)
+          .check('create')
+          .allowed()
+      message: >-
+        user is not permitted to create the spec.bootstrapSpec resource type —
+        creating a ScheduledMachine requires 'create' permission on the
+        embedded bootstrap resource (RBAC) to prevent privilege escalation
+        through the 5Spot controller
+      reason: Forbidden
+
+    # ── 13b. RBAC: requesting user may create the infrastructureSpec resource ──
+    # Same privilege-escalation guard as 13a, applied to the embedded
+    # infrastructure GVK (e.g. RemoteMachine).
+    - expression: >-
+        authorizer.group(variables.infraGroup)
+          .resource(variables.infraResource)
+          .namespace(object.metadata.namespace)
+          .check('create')
+          .allowed()
+      message: >-
+        user is not permitted to create the spec.infrastructureSpec resource
+        type — creating a ScheduledMachine requires 'create' permission on the
+        embedded infrastructure resource (RBAC) to prevent privilege escalation
+        through the 5Spot controller
+      reason: Forbidden
+
     # ── 14. nodeTaints: key format (RFC-1123 qualified name) ──────────────────
     # Mirrors validate_taint_key() in src/crd.rs. The optional "<prefix>/" part
     # is a DNS subdomain; the name part is a qualified name. Both capped at 63

Cargo.toml

@@ -1,5 +1,3 @@
-    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.12s
-     Running `target/debug/crdgen`
 apiVersion: apiextensions.k8s.io/v1
 kind: CustomResourceDefinition
 metadata:
@@ -58,6 +56,21 @@ spec:
                   kind:
                     description: Kind of the resource (e.g., 'K0sWorkerConfig', 'RemoteMachine')
                     type: string
+                  metadata:
+                    description: Optional labels/annotations to stamp on the created resource. Only 'labels' and 'annotations' are honoured; 'name' and 'namespace' are controller-owned and rejected at admission.
+                    properties:
+                      annotations:
+                        additionalProperties:
+                          type: string
+                        description: Annotations merged onto the created resource (reserved prefixes are rejected)
+                        type: object
+                      labels:
+                        additionalProperties:
+                          type: string
+                        description: Labels merged onto the created resource (reserved prefixes are rejected)
+                        type: object
+                    type: object
+                    x-kubernetes-preserve-unknown-fields: true
                   spec:
                     description: Provider-specific configuration
                     type: object
@@ -97,6 +110,21 @@ spec:
                   kind:
                     description: Kind of the resource (e.g., 'K0sWorkerConfig', 'RemoteMachine')
                     type: string
+                  metadata:
+                    description: Optional labels/annotations to stamp on the created resource. Only 'labels' and 'annotations' are honoured; 'name' and 'namespace' are controller-owned and rejected at admission.
+                    properties:
+                      annotations:
+                        additionalProperties:
+                          type: string
+                        description: Annotations merged onto the created resource (reserved prefixes are rejected)
+                        type: object
+                      labels:
+                        additionalProperties:
+                          type: string
+                        description: Labels merged onto the created resource (reserved prefixes are rejected)
+                        type: object
+                    type: object
+                    x-kubernetes-preserve-unknown-fields: true
                   spec:
                     description: Provider-specific configuration
                     type: object
@@ -137,6 +165,62 @@ spec:
                 default: false
                 description: When true, immediately removes the machine from cluster
                 type: boolean
+              kubeconfigSecretRef:
+                description: |-
+                  Reference to a Secret in this ScheduledMachine's namespace containing
+                  a kubeconfig for the workload (child) cluster whose Node(s) this
+                  resource manages.
+
+                  When set, every Node and Pod API call the controller makes on behalf
+                  of this resource — cordon, taint, drain (pod list + delete),
+                  reclaim-agent annotations / labels / ConfigMaps, status enrichment —
+                  is routed through that kubeconfig. CAPI / bootstrap / infrastructure
+                  / Machine objects continue to use the management cluster's in-cluster
+                  client.
+
+                  When unset (default), the controller first tries to auto-discover a
+                  Secret named `<spec.clusterName>-kubeconfig` in this same namespace
+                  (CAPI convention). If that Secret does not exist, the management
+                  client is used for Node/Pod operations as well — the degenerate
+                  single-cluster dev/test posture where management ≡ workload cluster.
+
+                  Cross-namespace Secret references are NOT supported: the Secret MUST
+                  live in this resource's own namespace. This is a security boundary
+                  (cross-namespace would let a tenant in one namespace read a kubeconfig
+                  in another).
+
+                  The supplied kubeconfig MUST grant: `nodes` get/list/watch/patch and
+                  `pods` get/list/delete in all namespaces of the child cluster, plus
+                  `configmaps` get/create/patch/delete in the reclaim-agent namespace
+                  if `killIfCommands` is also set. See
+                  `docs/src/concepts/child-cluster-kubeconfig.md` for the full RBAC
+                  requirements and threat model.
+                nullable: true
+                properties:
+                  key:
+                    default: value
+                    description: |-
+                      Key within the Secret's `data` map whose value is the kubeconfig YAML
+                      document. Defaults to `value` — CAPI's convention for
+                      `<clusterName>-kubeconfig` Secrets generated by the control-plane
+                      provider. Common overrides: `kubeconfig` (some k0smotron flows),
+                      `admin.conf` (kubeadm-style).
+                    maxLength: 253
+                    minLength: 1
+                    pattern: ^[A-Za-z0-9._-]+$
+                    type: string
+                  name:
+                    description: |-
+                      Name of the Secret. RFC-1123 DNS subdomain (max 253 chars).
+                      The Secret MUST live in the same namespace as the ScheduledMachine —
+                      there is no `namespace` field by design.
+                    maxLength: 253
+                    minLength: 1
+                    pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$
+                    type: string
+                required:
+                - name
+                type: object
               machineTemplate:
                 description: |-
                   Optional configuration for the created CAPI Machine