Skip to content

Child-cluster Node API support #71

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ebourgeois merged 1 commit into from
May 30, 2026
Merged

Child-cluster Node API support #71

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
422 changes: 422 additions & 0 deletions .claude/CHANGELOG.md
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 2 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,5 @@ __pycache__/
# Generated CALM diagrams (rendered by `make calm-diagrams` / CI).
# Source: docs/architecture/calm/templates/mermaid/*.md.hbs
/docs/src/architecture/

.DS_Store
2 changes: 2 additions & 0 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

15 changes: 14 additions & 1 deletion Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,10 +29,19 @@ name = "auto-vex-reachability"
path = "src/bin/auto_vex_reachability.rs"

[dependencies]
kube = { version = "3.1", features = ["runtime", "derive", "client", "ws"] }
# `unstable-runtime` is opt-in for `Controller::reconcile_on`, which we
# use in main.rs to feed per-child-cluster Node-watch events into the
# Controller. The feature is "unstable" in the SemVer sense (the API
# may change between minor kube-runtime versions) but is the canonical
# way to push external triggers into the controller loop.
kube = { version = "3.1", features = ["runtime", "derive", "client", "ws", "unstable-runtime"] }
kube-lease-manager = "0.11"
k8s-openapi = { version = "0.27", features = ["latest", "schemars"] }
tokio = { version = "1", features = ["full"] }
# `ReceiverStream` adapter so an `mpsc::Receiver<ObjectRef<…>>` can be
# fed into `Controller::reconcile_on`. Used by main.rs to plumb
# child-cluster Node-watch events back into the controller.
tokio-stream = "0.1"
serde = { version = "1", features = ["derive"] }
serde_json = "1"
serde_yaml = "0.9"
Expand Down Expand Up @@ -79,6 +88,10 @@ test-log = "0.2"
tempfile = "3"
tower-test = "0.4"
http = "1"
# Used by src/reconcilers/child_client_tests.rs to build base64-encoded
# Secret payloads for the tower-test mock server. base64 0.22 is already
# in the dependency graph transitively (kube / k8s-openapi pull it).
base64 = "0.22"

[profile.release]
opt-level = 3
Expand Down
91 changes: 91 additions & 0 deletions deploy/admission/validatingadmissionpolicy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,21 @@ spec:
resources: ["scheduledmachines"]
operations: ["CREATE", "UPDATE"]

# Derived values reused by the RBAC authorizer checks below. The group is the
# part of apiVersion before '/'; the resource is the naive lowercase plural of
# kind (lowerAscii + 's'). This mirrors resource_plural() in
# src/reconcilers/helpers.rs so the permission checked here is exactly the one
# the controller will exercise when it creates the resource.
variables:
- name: bootstrapGroup
expression: "object.spec.bootstrapSpec.apiVersion.split('/')[0]"
- name: bootstrapResource
expression: "object.spec.bootstrapSpec.kind.lowerAscii() + 's'"
- name: infraGroup
expression: "object.spec.infrastructureSpec.apiVersion.split('/')[0]"
- name: infraResource
expression: "object.spec.infrastructureSpec.kind.lowerAscii() + 's'"

validations:

# ── 1. clusterName: non-empty ────────────────────────────────────────────
Expand Down Expand Up @@ -196,6 +211,82 @@ spec:
message: "spec.infrastructureSpec.kind must not be empty"
reason: Invalid

# ── 13c–13f. Embedded metadata: name/namespace are controller-owned ───────
# The controller owns the created resource's identity: it always names the
# bootstrap/infrastructure resource after the ScheduledMachine and creates
# it in the SM's own namespace (cross-namespace creation is forbidden —
# threat T1). A user-supplied metadata.name/metadata.namespace is therefore
# rejected, not silently ignored.
#
# NOTE: this only works because EmbeddedResource's `metadata` is marked
# `x-kubernetes-preserve-unknown-fields: true` in src/crd.rs. Without it the
# API server PRUNES unknown fields before admission policies run, so the
# field would be gone before this rule could see it. The runtime
# `validate_embedded_metadata()` in src/reconcilers/helpers.rs enforces the
# same rule as defence-in-depth. Only metadata.labels / metadata.annotations
# are user-settable (and are reserved-prefix-checked at reconcile time).
- expression: "!has(object.spec.bootstrapSpec.metadata) || !has(object.spec.bootstrapSpec.metadata.namespace)"
message: >-
spec.bootstrapSpec.metadata.namespace is not permitted — the 5Spot
controller always creates the bootstrap resource in the
ScheduledMachine's own namespace
reason: Forbidden
- expression: "!has(object.spec.bootstrapSpec.metadata) || !has(object.spec.bootstrapSpec.metadata.name)"
message: >-
spec.bootstrapSpec.metadata.name is not permitted — the 5Spot controller
names the bootstrap resource after the ScheduledMachine
reason: Forbidden
- expression: "!has(object.spec.infrastructureSpec.metadata) || !has(object.spec.infrastructureSpec.metadata.namespace)"
message: >-
spec.infrastructureSpec.metadata.namespace is not permitted — the 5Spot
controller always creates the infrastructure resource in the
ScheduledMachine's own namespace
reason: Forbidden
- expression: "!has(object.spec.infrastructureSpec.metadata) || !has(object.spec.infrastructureSpec.metadata.name)"
message: >-
spec.infrastructureSpec.metadata.name is not permitted — the 5Spot
controller names the infrastructure resource after the ScheduledMachine
reason: Forbidden

# ── 13a. RBAC: requesting user may create the bootstrapSpec resource ──────
# Privilege-escalation guard. The 5Spot controller runs with broad RBAC so
# it can create bootstrap/infrastructure/Machine objects on the user's
# behalf. Without this check, a user able to create a ScheduledMachine but
# NOT a K0sWorkerConfig could have the controller create one for them —
# escalating through the controller. We require the *requesting user* to
# independently hold `create` on the embedded bootstrap GVK in the target
# namespace, mirroring how CAPI's own webhooks gate templated resources.
# The controller-side equivalent (its own service account) is enforced at
# reconcile time by ensure_can_create() in src/reconcilers/helpers.rs.
- expression: >-
authorizer.group(variables.bootstrapGroup)
.resource(variables.bootstrapResource)
.namespace(object.metadata.namespace)
.check('create')
.allowed()
message: >-
user is not permitted to create the spec.bootstrapSpec resource type —
creating a ScheduledMachine requires 'create' permission on the
embedded bootstrap resource (RBAC) to prevent privilege escalation
through the 5Spot controller
reason: Forbidden

# ── 13b. RBAC: requesting user may create the infrastructureSpec resource ──
# Same privilege-escalation guard as 13a, applied to the embedded
# infrastructure GVK (e.g. RemoteMachine).
- expression: >-
authorizer.group(variables.infraGroup)
.resource(variables.infraResource)
.namespace(object.metadata.namespace)
.check('create')
.allowed()
message: >-
user is not permitted to create the spec.infrastructureSpec resource
type — creating a ScheduledMachine requires 'create' permission on the
embedded infrastructure resource (RBAC) to prevent privilege escalation
through the 5Spot controller
reason: Forbidden

# ── 14. nodeTaints: key format (RFC-1123 qualified name) ──────────────────
# Mirrors validate_taint_key() in src/crd.rs. The optional "<prefix>/" part
# is a DNS subdomain; the name part is a qualified name. Both capped at 63
Expand Down
88 changes: 86 additions & 2 deletions deploy/crds/scheduledmachine.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.12s
Running `target/debug/crdgen`
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
Expand Down Expand Up @@ -58,6 +56,21 @@ spec:
kind:
description: Kind of the resource (e.g., 'K0sWorkerConfig', 'RemoteMachine')
type: string
metadata:
description: Optional labels/annotations to stamp on the created resource. Only 'labels' and 'annotations' are honoured; 'name' and 'namespace' are controller-owned and rejected at admission.
properties:
annotations:
additionalProperties:
type: string
description: Annotations merged onto the created resource (reserved prefixes are rejected)
type: object
labels:
additionalProperties:
type: string
description: Labels merged onto the created resource (reserved prefixes are rejected)
type: object
type: object
x-kubernetes-preserve-unknown-fields: true
spec:
description: Provider-specific configuration
type: object
Expand Down Expand Up @@ -97,6 +110,21 @@ spec:
kind:
description: Kind of the resource (e.g., 'K0sWorkerConfig', 'RemoteMachine')
type: string
metadata:
description: Optional labels/annotations to stamp on the created resource. Only 'labels' and 'annotations' are honoured; 'name' and 'namespace' are controller-owned and rejected at admission.
properties:
annotations:
additionalProperties:
type: string
description: Annotations merged onto the created resource (reserved prefixes are rejected)
type: object
labels:
additionalProperties:
type: string
description: Labels merged onto the created resource (reserved prefixes are rejected)
type: object
type: object
x-kubernetes-preserve-unknown-fields: true
spec:
description: Provider-specific configuration
type: object
Expand Down Expand Up @@ -137,6 +165,62 @@ spec:
default: false
description: When true, immediately removes the machine from cluster
type: boolean
kubeconfigSecretRef:
description: |-
Reference to a Secret in this ScheduledMachine's namespace containing
a kubeconfig for the workload (child) cluster whose Node(s) this
resource manages.

When set, every Node and Pod API call the controller makes on behalf
of this resource — cordon, taint, drain (pod list + delete),
reclaim-agent annotations / labels / ConfigMaps, status enrichment —
is routed through that kubeconfig. CAPI / bootstrap / infrastructure
/ Machine objects continue to use the management cluster's in-cluster
client.

When unset (default), the controller first tries to auto-discover a
Secret named `<spec.clusterName>-kubeconfig` in this same namespace
(CAPI convention). If that Secret does not exist, the management
client is used for Node/Pod operations as well — the degenerate
single-cluster dev/test posture where management ≡ workload cluster.

Cross-namespace Secret references are NOT supported: the Secret MUST
live in this resource's own namespace. This is a security boundary
(cross-namespace would let a tenant in one namespace read a kubeconfig
in another).

The supplied kubeconfig MUST grant: `nodes` get/list/watch/patch and
`pods` get/list/delete in all namespaces of the child cluster, plus
`configmaps` get/create/patch/delete in the reclaim-agent namespace
if `killIfCommands` is also set. See
`docs/src/concepts/child-cluster-kubeconfig.md` for the full RBAC
requirements and threat model.
nullable: true
properties:
key:
default: value
description: |-
Key within the Secret's `data` map whose value is the kubeconfig YAML
document. Defaults to `value` — CAPI's convention for
`<clusterName>-kubeconfig` Secrets generated by the control-plane
provider. Common overrides: `kubeconfig` (some k0smotron flows),
`admin.conf` (kubeadm-style).
maxLength: 253
minLength: 1
pattern: ^[A-Za-z0-9._-]+$
type: string
name:
description: |-
Name of the Secret. RFC-1123 DNS subdomain (max 253 chars).
The Secret MUST live in the same namespace as the ScheduledMachine —
there is no `namespace` field by design.
maxLength: 253
minLength: 1
pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$
type: string
required:
- name
type: object
machineTemplate:
description: |-
Optional configuration for the created CAPI Machine
Expand Down
Loading
Loading