Add support for different versions of CAPI bootstrap and infrastructure specs, passthrough

Signed-off-by: Erick Bourgeois <erick@jeb.ca>

Author: ebourgeois

.claude/CHANGELOG.md

@@ -9,6 +9,62 @@ The format is based on the regulated environment requirements:
 
 ---
 
+## [2026-06-22 12:00] - CAPI Machine API version: passthrough from bootstrapSpec + mandatory discovery (no hardcoded version)
+
+**Author:** Erick Bourgeois
+
+### Changed
+- `src/constants.rs`: removed the hardcoded `CAPI_MACHINE_API_VERSION` /
+  `CAPI_MACHINE_API_VERSION_FULL` consts. The version is now a runtime value
+  (`OnceLock`) set once at startup via `set_capi_machine_api_version`; the accessor
+  `capi_machine_api_version()` panics if read before resolution (an invariant —
+  `main` resolves before any reconcile) and returns `v1beta1` only under
+  `#[cfg(test)]` for fixtures. Group/kind (`cluster.x-k8s.io`/`Machine`) stay fixed
+  by the CAPI contract; only the version is data-driven.
+- `src/main.rs`: `resolve_capi_machine_version` now returns `Result` and discovery
+  is **mandatory** — precedence is `CAPI_MACHINE_API_VERSION` env override →
+  `kube::discovery` of the served `machines.cluster.x-k8s.io` version → error (6×5s
+  retry, then the controller exits and is restarted by Kubernetes). No silent
+  default. The Machine watch uses the resolved version.
+- `src/reconcilers/helpers.rs`: new `machine_api_version_for` helper derives the
+  Machine version from the version component of the user's `bootstrapSpec.apiVersion`
+  (passthrough), falling back to the discovered served version. Machine **create**
+  and **delete** both use it; **watch**/**fetch** use the discovered version (CAPI
+  conversion webhooks bridge any mismatch). `extract_machine_refs` relaxed so a
+  v1beta2 `nodeRef` (only `name` guaranteed) still resolves (defaults apiVersion→`v1`,
+  kind→`Node`).
+- `src/reconcilers/helpers_tests.rs`: +5 unit tests — 4 for `machine_api_version_for`
+  (v1beta2/v1beta1 passthrough, `None` fallback, malformed fallback) and a mock-client
+  test asserting `remove_machine_from_cluster` issues the Machine DELETE at the
+  bootstrapSpec version.
+- `docs/src/operations/configuration.md`: documented the new `CAPI_MACHINE_API_VERSION`
+  env override and added a "CAPI Machine API version" section explaining the
+  passthrough + mandatory-discovery model.
+- `docs/src/advanced/capi-integration.md`: added a "CAPI API version (passthrough)"
+  note to the Machine Creation Flow — the Machine version follows `bootstrapSpec`,
+  the `v1beta1` examples are illustrative not fixed.
+- `docs/src/operations/troubleshooting.md`: new entry for the controller exiting at
+  startup when Cluster API isn't discoverable ("could not resolve the CAPI Machine
+  API version").
+
+### Why
+5-Spot v0.2.2 hardcoded the CAPI Machine version to `v1beta1`, which crash-loops
+modern k0smotron / CAPI ≥ v1.11 (they serve/watch `v1beta2`). Making the version
+data-driven (user's bootstrapSpec) with mandatory served-version discovery lets one
+controller image work against both v1beta1 (CAPD) and v1beta2 (modern k0smotron)
+clusters, with no hardcoded opinion. Enables the workshop's k0smotron tier.
+
+### Impact
+- [ ] Breaking change
+- [x] Requires cluster rollout
+- [ ] Config change only
+- [ ] Documentation only (code + docs)
+
+Note: the controller now **requires** CAPI to be discoverable at startup (it retries
+~30s then exits→restarts) instead of silently defaulting to `v1beta1`. Security model
+unchanged — the bootstrap/infra **group** allowlist (`validate_api_group`) is the
+control; the Machine group is fixed and the version is not a security boundary.
+
 ## [2026-06-17 09:00] - Pin deploy manifest image tags to the release version; crdgen back to stdout
 
 **Author:** Erick Bourgeois

docs/src/advanced/capi-integration.md

@@ -107,6 +107,18 @@ sequenceDiagram
     5S->>SM: Update status: Active
 ```
 
+> **CAPI API version (passthrough).** The `Machine` 5-Spot creates is always in the
+> fixed `cluster.x-k8s.io` group, but its **version** is not hardcoded — it follows
+> the version component of your `bootstrapSpec.apiVersion`. Declare
+> `bootstrap.cluster.x-k8s.io/v1beta1` and 5-Spot creates (and deletes) a `v1beta1`
+> Machine; declare `…/v1beta2` and it uses `v1beta2`, so the Machine shares one CAPI
+> contract version with your bootstrap and infrastructure objects. For watching and
+> reading Machines the controller uses the version the cluster *serves* (discovered
+> at startup) and **requires** Cluster API to be installed — see
+> [Configuration → CAPI Machine API version](../operations/configuration.md#capi-machine-api-version).
+> The `cluster.x-k8s.io/v1beta1` shown in the examples above is therefore one valid
+> choice, not a fixed requirement.
+
 ## Machine Deletion Flow
 
 ```mermaid

docs/src/operations/configuration.md

@@ -19,6 +19,7 @@
 | `LEASE_DURATION_SECONDS` | `15` | How long the Lease is considered valid; a new leader is elected if not renewed in time |
 | `LEASE_RENEW_DEADLINE_SECONDS` | `10` | The leader must renew the Lease within this many seconds; grace = duration − deadline |
 | `LEASE_RETRY_PERIOD_SECONDS` | `2` | Documented for ops parity; not a direct LeaseManager parameter |
+| `CAPI_MACHINE_API_VERSION` | _(discovered)_ | Pins the Cluster API `Machine` version (`v1beta1` / `v1beta2`). Normally **left unset** — the controller discovers the version the cluster serves. Set only to override discovery (e.g. an air-gapped cluster where API discovery is restricted). See [CAPI Machine API version](#capi-machine-api-version). |
 
 ## Command-Line Arguments
 
@@ -69,6 +70,28 @@ Non-leader replicas watch for leadership changes and take over automatically wit
 
 > **Note:** Leader election and multi-instance sharding (`OPERATOR_INSTANCE_COUNT > 1`) are alternative HA strategies. Use leader election for active/standby HA; use instance sharding to distribute load across all replicas.
 
+### CAPI Machine API version
+
+5-Spot creates and watches Cluster API `Machine` objects. The `Machine` *group* and
+*kind* (`cluster.x-k8s.io` / `Machine`) are fixed by the CAPI contract, but the
+served **version** varies by cluster — `v1beta1` on CAPI ≤ v1.10, `v1beta2` on
+v1.11+ and modern k0smotron. The controller resolves it at startup with **no
+hardcoded default**:
+
+1. **`CAPI_MACHINE_API_VERSION`** — if set, this exact version is used (operator override).
+2. **Discovery** — otherwise the controller discovers the version the API server
+   serves for `machines.cluster.x-k8s.io`.
+
+Discovery is **mandatory**: if neither source yields a version (for example Cluster
+API is not installed yet), the controller retries briefly, then exits and is
+restarted by Kubernetes — it never guesses a version.
+
+Per `ScheduledMachine`, the **created** `Machine` follows the version component of
+your `bootstrapSpec.apiVersion` (passthrough), so the Machine, bootstrap, and
+infrastructure objects all share one CAPI contract version. One controller image
+therefore works against both `v1beta1` (e.g. CAPD) and `v1beta2` (e.g. modern
+k0smotron) clusters with no configuration.
+
 ## Reclaim agent (DaemonSet)
 
 The node-side `5spot-reclaim-agent` is a separate binary deployed

docs/src/operations/troubleshooting.md

@@ -181,6 +181,26 @@ kubectl describe machine <name>
 - Check CAPI provider health
 - Review CAPI controller logs
 
+### Controller exits at startup: "could not resolve the CAPI Machine API version"
+
+**Symptoms:**
+- The controller pod crash-loops shortly after starting (`CrashLoopBackOff`)
+- Logs end with `could not resolve the CAPI Machine API version after 6 attempts … is Cluster API installed?`
+- Preceding lines show `CAPI Machine API version not resolved yet; retrying`
+
+**Cause:**
+The controller resolves which `cluster.x-k8s.io` Machine version the cluster serves
+at startup (see [Configuration → CAPI Machine API version](configuration.md#capi-machine-api-version)).
+Discovery is **mandatory** — it never guesses a version. If Cluster API is not
+installed, or its CRDs / aggregated discovery aren't ready yet, the controller
+retries (~30s) and then exits so Kubernetes restarts it.
+
+**Solution:**
+- Install Cluster API before or alongside 5-Spot — `kubectl get crd machines.cluster.x-k8s.io` must succeed.
+- If discovery is restricted (air-gapped / locked-down clusters), set
+  `CAPI_MACHINE_API_VERSION` (`v1beta1` or `v1beta2`) to skip discovery.
+- Once CAPI is present the controller resolves on its next restart — no manual action needed.
+
 ### Reconciliation Retrying with Increasing Delay
 
 **Symptoms:**

src/constants.rs

@@ -17,6 +17,8 @@
 //! - Operator instance environment variables
 //! - Security constraints (`MAX_DURATION_SECS`, `RESERVED_LABEL_PREFIXES`, …)
 
+use std::sync::OnceLock;
+
 // ============================================================================
 // Kubernetes API Constants
 // ============================================================================
@@ -623,11 +625,52 @@ pub const CAPI_MACHINE_API_GROUP: &str = "cluster.x-k8s.io";
 /// CAPI API group
 pub const CAPI_GROUP: &str = "cluster.x-k8s.io";
 
-/// CAPI Machine API version
-pub const CAPI_MACHINE_API_VERSION: &str = "v1beta1";
-
-/// Full CAPI Machine API version string
-pub const CAPI_MACHINE_API_VERSION_FULL: &str = "cluster.x-k8s.io/v1beta1";
+/// Runtime-resolved CAPI Machine API version (`"v1beta1"` / `"v1beta2"`), recorded
+/// once at startup by `main` via **mandatory discovery** (or the
+/// `CAPI_MACHINE_API_VERSION` env override). There is deliberately NO compile-time
+/// default version: the controller refuses to start if it can't determine which
+/// version the cluster serves, rather than silently guessing one. The Machine
+/// *group* and *kind* (`cluster.x-k8s.io` / `Machine`) are fixed by the CAPI
+/// contract; only the version varies, and it is never hardcoded for production.
+static CAPI_MACHINE_API_VERSION_RUNTIME: OnceLock<String> = OnceLock::new();
+
+/// Record the CAPI Machine API version resolved at startup (discovery or the
+/// `CAPI_MACHINE_API_VERSION` env override). Idempotent — only the first call
+/// takes effect; later calls are ignored (the served version is fixed for the
+/// controller's lifetime).
+pub fn set_capi_machine_api_version(version: String) {
+    let _ = CAPI_MACHINE_API_VERSION_RUNTIME.set(version);
+}
+
+/// The CAPI Machine API version the controller should create/watch Machines at.
+///
+/// Panics if called before [`set_capi_machine_api_version`] has run — that is an
+/// invariant violation: `main` resolves the version (mandatory discovery) before
+/// any reconcile runs, so the value is always present in production. Unit tests,
+/// which never perform real discovery, get the `v1beta1` their fixtures assume.
+#[must_use]
+pub fn capi_machine_api_version() -> &'static str {
+    match CAPI_MACHINE_API_VERSION_RUNTIME.get() {
+        Some(version) => version.as_str(),
+        None => {
+            #[cfg(test)]
+            {
+                "v1beta1"
+            }
+            #[cfg(not(test))]
+            panic!(
+                "CAPI Machine API version not resolved — set_capi_machine_api_version() \
+                 must run (after mandatory discovery) before any reconcile"
+            )
+        }
+    }
+}
+
+/// Full `cluster.x-k8s.io/<version>` string for the active CAPI Machine version.
+#[must_use]
+pub fn capi_machine_api_version_full() -> String {
+    format!("{CAPI_GROUP}/{}", capi_machine_api_version())
+}
 
 /// CAPI cluster name label
 pub const CAPI_CLUSTER_NAME_LABEL: &str = "cluster.x-k8s.io/cluster-name";

src/main.rs

@@ -18,10 +18,11 @@ use std::sync::Arc;
 use anyhow::Result;
 use clap::Parser;
 use five_spot::constants::{
-    CAPI_GROUP, CAPI_MACHINE_API_VERSION, CAPI_RESOURCE_MACHINES, CHILD_NODE_EVENT_CHANNEL_CAP,
-    DEFAULT_LEASE_DURATION_SECS, DEFAULT_LEASE_GRACE_SECS, DEFAULT_LEASE_NAME,
-    DEFAULT_LEASE_NAMESPACE, DEFAULT_LEASE_RENEW_DEADLINE_SECS, DEFAULT_LEASE_RETRY_PERIOD_SECS,
-    HEALTH_PORT, K8S_API_TIMEOUT_SECS, METRICS_PORT, SPOT_SCHEDULE_EVENT_CHANNEL_CAP,
+    capi_machine_api_version, set_capi_machine_api_version, CAPI_GROUP, CAPI_RESOURCE_MACHINES,
+    CHILD_NODE_EVENT_CHANNEL_CAP, DEFAULT_LEASE_DURATION_SECS, DEFAULT_LEASE_GRACE_SECS,
+    DEFAULT_LEASE_NAME, DEFAULT_LEASE_NAMESPACE, DEFAULT_LEASE_RENEW_DEADLINE_SECS,
+    DEFAULT_LEASE_RETRY_PERIOD_SECS, HEALTH_PORT, K8S_API_TIMEOUT_SECS, METRICS_PORT,
+    SPOT_SCHEDULE_EVENT_CHANNEL_CAP,
 };
 use five_spot::crd::ScheduledMachine;
 use five_spot::health::{start_health_server, HealthState};
@@ -179,6 +180,15 @@ async fn main() -> Result<()> {
     // Mark Kubernetes as connected
     health_state.set_k8s_connected(true);
 
+    // Resolve the CAPI Machine API version the cluster actually serves (v1beta1 on
+    // CAPI ≤ v1.10, v1beta2 on v1.11+ / modern k0smotron), so we create and watch
+    // Machines at the version downstream controllers expect. Env override wins;
+    // discovery otherwise. MANDATORY — if it can't be resolved the controller errors
+    // out (and is restarted by Kubernetes) rather than guessing a version.
+    let capi_machine_version = resolve_capi_machine_version(&client).await?;
+    info!(version = %capi_machine_version, "Using CAPI Machine API version");
+    set_capi_machine_api_version(capi_machine_version);
+
     // Create shared context
     let context = Arc::new(
         Context::new(client.clone(), cli.instance_id, cli.instance_count)
@@ -296,7 +306,7 @@ async fn main() -> Result<()> {
     //    `SM.status.nodeRef.name` closes the watch-amplification vector
     //    documented in Phase 3 of the 2026-04-25 security audit roadmap.
     let machine_ar = ApiResource::from_gvk_with_plural(
-        &GroupVersionKind::gvk(CAPI_GROUP, CAPI_MACHINE_API_VERSION, "Machine"),
+        &GroupVersionKind::gvk(CAPI_GROUP, capi_machine_api_version(), "Machine"),
         CAPI_RESOURCE_MACHINES,
     );
     let machines_api: Api<DynamicObject> = Api::all_with(client.clone(), &machine_ar);
@@ -422,6 +432,57 @@ async fn main() -> Result<()> {
     Ok(())
 }
 
+/// Resolve which `cluster.x-k8s.io` Machine API version this cluster serves.
+///
+/// Precedence: the `CAPI_MACHINE_API_VERSION` env var (explicit operator pin) →
+/// API discovery of the group's recommended/served version. Discovery is
+/// **mandatory** — there is no compile-time default. If the version can't be
+/// determined (Cluster API not installed / not yet ready) this returns an error and
+/// the controller exits, to be restarted by Kubernetes — we never guess a version.
+/// One binary works against CAPI lines serving `v1beta1` (≤ v1.10) or `v1beta2`
+/// (v1.11+ / modern k0smotron) with no config.
+async fn resolve_capi_machine_version(client: &Client) -> Result<String> {
+    if let Ok(pinned) = std::env::var("CAPI_MACHINE_API_VERSION") {
+        let pinned = pinned.trim();
+        if !pinned.is_empty() {
+            info!(version = %pinned, "CAPI Machine API version pinned via env");
+            return Ok(pinned.to_string());
+        }
+    }
+    // Retry briefly so a controller that starts moments before CAPI finishes
+    // installing converges instead of crash-looping on the first probe.
+    const ATTEMPTS: u32 = 6;
+    let mut last_reason = String::from("discovery not attempted");
+    for attempt in 1..=ATTEMPTS {
+        match kube::discovery::group(client, CAPI_GROUP).await {
+            Ok(apigroup) => match apigroup.recommended_kind("Machine") {
+                Some((ar, _caps)) => {
+                    info!(version = %ar.version, "Discovered CAPI Machine API version");
+                    return Ok(ar.version);
+                }
+                None => {
+                    last_reason =
+                        format!("group '{CAPI_GROUP}' is served but exposes no Machine kind");
+                }
+            },
+            Err(error) => {
+                last_reason = format!("discovery of '{CAPI_GROUP}' failed: {error}");
+            }
+        }
+        warn!(
+            attempt,
+            max = ATTEMPTS,
+            reason = %last_reason,
+            "CAPI Machine API version not resolved yet; retrying"
+        );
+        tokio::time::sleep(std::time::Duration::from_secs(5)).await;
+    }
+    Err(anyhow::anyhow!(
+        "could not resolve the CAPI Machine API version after {ATTEMPTS} attempts ({last_reason}); \
+         is Cluster API installed? set the CAPI_MACHINE_API_VERSION env var to override discovery"
+    ))
+}
+
 /// Check if the `ScheduledMachine` CRD is installed in the cluster
 async fn check_crd_installed(api: &Api<ScheduledMachine>) -> Result<()> {
     // Try to list resources with limit 0 - this will fail if CRD doesn't exist