Disable rustls TLS-1.3 session resumption for the child client

.claude/CHANGELOG.md

@@ -9,7 +9,36 @@ The format is based on the regulated environment requirements:
 
 ---
 
-## [2026-06-26 10:00] - Switch rustls crypto provider ring -> aws-lc-rs (fixes mTLS to k0smotron-hosted control planes)
+## [2026-06-27 10:00] - Disable rustls TLS-1.3 session resumption for child clients (the real mTLS-to-hosted-CP fix)
+
+**Author:** Erick Bourgeois
+
+### Changed
+- `src/reconcilers/child_client.rs`: build the child-cluster `kube::Client` by hand (instead of
+  `Client::try_from`) so we can set `rustls::client::Resumption::disabled()` on its TLS config.
+  Otherwise mirrors kube's own connector stack (TimeoutConnector with the wire timeouts, base-URI
+  + auth layers, `BoxError` mapping).
+- `Cargo.toml`: add direct deps `hyper-rustls`, `rustls`, `hyper-timeout` for the custom connector.
+- 682 lib tests pass (incl. the 17 child-client tests through the custom connector), fmt/clippy clean.
+
+### Why
+A k0smotron-hosted control plane requires fresh **client-certificate (mTLS)** auth on every
+connection and — like Go's TLS server — will not complete a **resumed** TLS-1.3 session that skips
+client auth. kube-rs/rustls, however, offers a PSK (session ticket) on reconnect; the `rustls=trace`
+shows `Resuming using PSK` on every connection, the resumed handshake **stalls right after the
+server's EncryptedExtensions**, and times out (`client error (Connect) -> deadline has elapsed`) — so
+node taints/drains never happen on the hosted-CP tier. client-go and `curl` always do a *full* mTLS
+handshake (proven on the playground: both connect in ~10ms). Disabling resumption makes 5-Spot's
+child client do the same. NOTE: this is provider-independent — both `ring` and `aws-lc-rs` exhibit
+the stall, so the earlier provider swap was **not** the fix.
+
+### Impact
+- [ ] Breaking change
+- [x] Requires cluster rollout
+- [ ] Config change only
+- [ ] Documentation only
+
+## [2026-06-26 10:00] - Switch rustls crypto provider ring -> aws-lc-rs (NOT the fix — superseded by disabling resumption)
 
 **Author:** Erick Bourgeois
 

Cargo.toml

@@ -80,6 +80,14 @@ http-body-util = "0.1"
 # API error statuses we branch on, instead of hardcoding 404/409/429).
 http = "1"
 tower = "0.5"
+# Child-cluster client: kube-rs/rustls keeps RESUMING the TLS-1.3 session (PSK) against
+# a k0smotron-hosted control plane that requires fresh client-cert (mTLS) auth, and the
+# resumed handshake stalls (client error (Connect) -> deadline elapsed). We build the
+# child client's connector by hand with rustls session resumption DISABLED so every
+# connection performs a full mTLS handshake (what client-go/curl do, which works).
+hyper-rustls = { version = "0.27", default-features = false, features = ["http1", "http2", "aws-lc-rs"] }
+rustls = { version = "0.23", default-features = false, features = ["aws_lc_rs"] }
+hyper-timeout = "0.5"
 regex = "1"
 lazy_static = "1.5"
 # warp 0.4 made every previously-default feature opt-in. We use

src/reconcilers/child_client.rs

@@ -39,7 +39,7 @@ use std::sync::{Arc, RwLock};
 use std::time::Instant;
 
 use k8s_openapi::api::core::v1::Secret;
-use kube::{config::Kubeconfig, Api, Client, ResourceExt};
+use kube::{client::ConfigExt, config::Kubeconfig, Api, Client, ResourceExt};
 use tracing::{debug, info, warn};
 
 use crate::constants::{CHILD_CLIENT_CACHE_CAP, HTTP_NOT_FOUND, K8S_API_TIMEOUT_SECS};
@@ -453,12 +453,53 @@ impl ChildClientCache {
         // would stall reconciliation indefinitely.
         config.read_timeout = Some(std::time::Duration::from_secs(K8S_API_TIMEOUT_SECS));
         config.write_timeout = Some(std::time::Duration::from_secs(K8S_API_TIMEOUT_SECS));
-
-        let child = Client::try_from(config).map_err(|e| ReconcilerError::KubeconfigInvalid {
+        config.connect_timeout = Some(std::time::Duration::from_secs(K8S_API_TIMEOUT_SECS));
+
+        // Build the child client BY HAND with rustls TLS-1.3 session resumption DISABLED.
+        //
+        // kube-rs's default `Client::try_from` resumes the TLS session (PSK) on reconnect.
+        // A k0smotron-hosted control plane requires fresh client-certificate (mTLS) auth on
+        // every connection and will not complete a resumed handshake that skips it — so the
+        // connection stalls until the wire timeout ("client error (Connect) -> deadline has
+        // elapsed") and the Node is never tainted/drained. Disabling resumption forces a full
+        // mTLS handshake every time, which is exactly what client-go and curl do (both connect
+        // in ~10ms). This otherwise mirrors kube's own connector stack (timeouts, base-URI and
+        // auth layers).
+        let make_err = |reason: String| ReconcilerError::KubeconfigInvalid {
             namespace: namespace.to_string(),
             name: secret_name.to_string(),
-            reason: format!("could not build kube::Client: {e}"),
-        })?;
+            reason,
+        };
+        let mut tls = config
+            .rustls_client_config()
+            .map_err(|e| make_err(format!("could not build rustls config: {e}")))?;
+        tls.resumption = rustls::client::Resumption::disabled();
+        let mut http = hyper_util::client::legacy::connect::HttpConnector::new();
+        http.enforce_http(false);
+        let https = hyper_rustls::HttpsConnectorBuilder::new()
+            .with_tls_config(tls)
+            .https_or_http()
+            .enable_http1()
+            .enable_http2()
+            .wrap_connector(http);
+        let mut connector = hyper_timeout::TimeoutConnector::new(https);
+        connector.set_connect_timeout(config.connect_timeout);
+        connector.set_read_timeout(config.read_timeout);
+        connector.set_write_timeout(config.write_timeout);
+        let hyper_client: hyper_util::client::legacy::Client<_, kube::client::Body> =
+            hyper_util::client::legacy::Client::builder(hyper_util::rt::TokioExecutor::new())
+                .build(connector);
+        let auth = config
+            .auth_layer()
+            .map_err(|e| make_err(format!("could not build auth layer: {e}")))?;
+        let service = tower::ServiceBuilder::new()
+            .layer(config.base_uri_layer())
+            .option_layer(auth)
+            // Box the hyper client's error so the auth / base-URI layers see a uniform
+            // `BoxError` (mirrors kube's own `make_generic_builder`).
+            .map_err(tower::BoxError::from)
+            .service(hyper_client);
+        let child = Client::new(service, config.default_namespace.clone());
 
         // Insert (with LRU eviction if at cap) and return. Track whether
         // we're replacing an existing entry for the same key (RV change