Remove the mTLS check and resumption change, as this was a red herring

.claude/CHANGELOG.md

@@ -9,29 +9,32 @@ The format is based on the regulated environment requirements:
 
 ---
 
-## [2026-06-26 10:00] - Switch rustls crypto provider ring -> aws-lc-rs (fixes mTLS to k0smotron-hosted control planes)
+## [2026-06-27 14:00] - THE ACTUAL k0smotron taint/drain fix: NetworkPolicy egress + server-side-apply type meta
 
 **Author:** Erick Bourgeois
 
 ### Changed
-- `Cargo.toml`: `kube` now uses `default-features = false` + `["…, "rustls-tls", "aws-lc-rs"]`
-  instead of the default that pulls the `ring` crypto provider. `ring` leaves the lock; the
-  rustls TLS stack now runs on aws-lc-rs (BoringSSL-derived). 682 lib tests pass, fmt/clippy clean.
-
-### Why
-5-Spot's child client could not complete the mTLS TLS-1.3 handshake against a k0smotron-hosted
-control plane — it stalled after the client-certificate step and timed out
-(`client error (Connect) -> deadline has elapsed`), so node taints/drains never happened on the
-hosted-CP tier. Proven on a live cluster that `curl --cert/--key` (OpenSSL) and `client-go`
-(kubelet/kubectl) complete the *identical* handshake to the *same* endpoint in ~13ms, while
-kube-rs/rustls-with-`ring` hangs (captured via `rustls=trace`). aws-lc-rs shares OpenSSL/BoringSSL's
-crypto lineage, so it behaves like the clients that work.
-
-### Build note
-CI builds **natively per arch** (`ubuntu-24.04` + `ubuntu-24.04-arm`, plain `cargo build --release`),
-so aws-lc-rs compiles natively (cmake + cc, present on GitHub runners) — no cross-toolchain and no
-OpenSSL runtime lib in the distroless image, i.e. it avoids the cross-compile problem that motivated
-`ring` in the first place.
+- `src/reconcilers/helpers.rs` (`apply_node_taints`): the server-side-apply patch was missing
+  `apiVersion: v1` + `kind: Node`. SSA REQUIRES the object's type meta in the body; without it the
+  API server rejects with `invalid object type: /, Kind=: BadRequest (400)` and the taint never lands.
+- `src/reconcilers/helpers_tests.rs`: the apply-taints test now asserts `apiVersion`+`kind` are in the
+  patch body (would have caught this).
+- `deploy/deployment/networkpolicy.yaml`: egress allowed only `53` + `6443`, which **silently dropped
+  every child-cluster connection at the network layer** — k0smotron's hosted CP is on the NodePort
+  `apiPort` (default **30443**), not 6443. Added 30443 to the egress (with guidance for other ports).
+- `docs/src/security/index.md`, `docs/src/security/threat-model.md`: the egress-posture rows now
+  state the policy permits the mgmt apiserver (6443) + child/k0smotron-hosted CPs (30443) + DNS,
+  matching the updated NetworkPolicy.
+
+### Why — and a correction
+On the k0smotron tier the child client could never taint/drain. Live diagnosis on the playground proved
+**two** independent root causes, both above were confirmed by experiment (delete the NetworkPolicy →
+connection succeeds; a properly-typed `kubectl apply --server-side` → taint lands). The earlier
+TLS-centric changes — `insecure-skip-tls-verify`, the `ring`→`aws-lc-rs` provider swap, and disabling
+rustls session resumption — were **misdiagnoses**: the `rustls=trace` "resumption stall" was the
+*management* client's normal TLS, while the child client was firewalled before it ever reached TLS. Those
+TLS experiments have been **reverted** (back to the default `ring` provider + `Client::try_from`);
+this NetworkPolicy + server-side-apply change is the real fix.
 
 ### Impact
 - [ ] Breaking change

Cargo.toml

@@ -45,13 +45,7 @@ path = "src/bin/spot_schedule_capital_markets.rs"
 # Controller. The feature is "unstable" in the SemVer sense (the API
 # may change between minor kube-runtime versions) but is the canonical
 # way to push external triggers into the controller loop.
-# TLS: use rustls with the aws-lc-rs crypto provider (BoringSSL-derived) instead of the
-# `ring` provider kube pulls by default. `ring` cannot complete the mTLS TLS-1.3 handshake
-# against a k0smotron-hosted control plane (it stalls after the client cert step), whereas
-# aws-lc-rs — same crypto lineage as OpenSSL/BoringSSL, which client-go and curl use
-# successfully against that exact endpoint — does. default-features=false drops the default
-# `ring`; we re-add `client` + `rustls-tls` explicitly alongside `aws-lc-rs`.
-kube = { version = "3.1", default-features = false, features = ["runtime", "derive", "client", "ws", "unstable-runtime", "rustls-tls", "aws-lc-rs"] }
+kube = { version = "3.1", features = ["runtime", "derive", "client", "ws", "unstable-runtime"] }
 kube-lease-manager = "0.11"
 k8s-openapi = { version = "0.27", features = ["latest", "schemars"] }
 tokio = { version = "1", features = ["full"] }

deploy/deployment/networkpolicy.yaml

@@ -5,8 +5,10 @@
 #
 # Ingress: Only Prometheus (monitoring namespace) may scrape metrics on 8080.
 #          Kubelet health probes are allowed on 8081 from any source (k8s internal).
-# Egress:  Only Kubernetes API server (6443) and DNS (53) are permitted.
-#          All other egress is denied, preventing data exfiltration.
+# Egress:  DNS (53) and Kubernetes API servers (6443 management + standard child
+#          clusters, 30443 k0smotron-hosted control planes) are permitted. All other
+#          egress is denied, preventing data exfiltration. Child clusters on other
+#          API ports must add that port to the egress rule below.
 #
 # NIST SP 800-53 SC-7: Boundary Protection
 apiVersion: networking.k8s.io/v1
@@ -45,7 +47,19 @@ spec:
         - protocol: TCP
           port: 53
 
-    # Kubernetes API server
+    # Kubernetes API servers — BOTH the management cluster AND any child/workload
+    # cluster the controller drives (taint/drain in CAPI / k0smotron topologies).
+    #   * 6443  — the management apiserver (the kube API ClusterIP DNATs to it) and
+    #             standard child clusters.
+    #   * 30443 — a k0smotron-hosted control plane's default NodePort apiPort; the
+    #             child kubeconfig's server is `…:30443`, and the connection is
+    #             evaluated on that pod port, so it MUST be allowed here.
+    #
+    # ⚠ A child cluster on any OTHER API port must add it here, or the child-cluster
+    # client is silently dropped at the network layer ("client error (Connect) ->
+    # deadline has elapsed") and taints/drains never run.
     - ports:
         - protocol: TCP
           port: 6443
+        - protocol: TCP
+          port: 30443

docs/src/security/index.md

@@ -34,7 +34,7 @@ Per-field validation status and downstream sinks for every attacker-controllable
 |---|---|---|
 | Non-root container, read-only rootfs, all caps dropped | ✅ | `deploy/deployment/deployment.yaml` |
 | Least-privilege RBAC — explicit resources, no wildcards | ✅ | `deploy/deployment/rbac/clusterrole.yaml` |
-| NetworkPolicy — egress to API server only | ✅ | `deploy/deployment/networkpolicy.yaml` |
+| NetworkPolicy — egress to API servers only (mgmt 6443 + child/k0smotron 30443) and DNS | ✅ | `deploy/deployment/networkpolicy.yaml` |
 | Admission validation — 13 CEL rules, `failurePolicy: Fail` | ✅ | `deploy/admission/` |
 | Finalizer cleanup timeout (10 min) — prevents deletion hangs | ✅ | `src/constants.rs` |
 | Cross-namespace resource creation prevented | ✅ | `namespace` field removed from `EmbeddedResource` |

docs/src/security/threat-model.md

@@ -325,7 +325,7 @@ in `5spot-system`.
 |---|---|
 | Kubernetes ResourceQuota | Limit `ScheduledMachine` count per namespace (e.g., max 50) |
 | `ValidatingAdmissionPolicy` ✅ deployed 2026-04-08 | Validates `bootstrapSpec.apiVersion`, `infrastructureSpec.apiVersion`, `kind` fields, duration format, and day/hour item format at admission time — see `deploy/admission/` |
-| NetworkPolicy | Restrict controller pod egress to Kubernetes API server only |
+| NetworkPolicy | Restrict controller pod egress to the management API server (6443), child/k0smotron-hosted control planes (NodePort apiPort 30443), and DNS — all other egress denied |
 | Audit logging | Enable API server audit log at `RequestResponse` level for `scheduledmachines` resources |
 | RBAC for SM creation | Only grant `create` on `scheduledmachines` to trusted identities; do not grant to end users directly |
 | Secrets for bootstrap data | Move sensitive bootstrap config out of CR spec into Secrets; reference from spec |

src/reconcilers/helpers.rs

@@ -3485,7 +3485,12 @@ pub async fn apply_node_taints(
         .collect();
     let annotation_value = serde_json::to_string(&owned).unwrap_or_else(|_| "[]".to_string());
 
+    // Server-side apply REQUIRES the object's type meta in the body; without
+    // `apiVersion` + `kind` the API server rejects the patch with
+    // `invalid object type: /, Kind=: BadRequest (400)` and the taint never lands.
     let patch = json!({
+        "apiVersion": "v1",
+        "kind": "Node",
         "metadata": {
             "annotations": {
                 APPLIED_TAINTS_ANNOTATION: annotation_value,