New error_chain() helper flattens the error source() chain. Idempotency sweep across the code base to make sure 409 tend to OK.

Signed-off-by: Erick Bourgeois <erick@jeb.ca>

Author: ebourgeois

.claude/CHANGELOG.md

@@ -9,6 +9,103 @@ The format is based on the regulated environment requirements:
 
 ---
 
+## [2026-06-24 10:30] - Idempotency sweep: 409 AlreadyExists / duplicate finalizers no longer Error the ScheduledMachine
+
+**Author:** Erick Bourgeois
+
+### Changed
+- `src/reconcilers/helpers.rs` (`create_dynamic_resource`): treat a `409 AlreadyExists`
+  from `api.create(...)` as success instead of propagating it. The bootstrap config,
+  infrastructure object, and CAPI Machine are all created through this helper; a prior
+  reconcile having created them IS the desired state.
+- `src/reconcilers/helpers.rs` (`add_finalizer`): only append our finalizer if it isn't
+  already present — a double-call can no longer write a duplicate (which would wedge GC).
+- `src/reconcilers/helpers_tests.rs`: +3 unit tests (create 409 → Ok; create 500 still
+  propagates; add_finalizer writes exactly one copy).
+- Audited the rest of the codebase for idempotency and confirmed it already holds: the
+  only other `.create()` is a `SelfSubjectAccessReview` (a virtual auth check, never
+  409s); all `.delete()` sites swallow 404; the kata ConfigMap uses `Patch::Apply` (SSA);
+  Events go through the kube `recorder`; status/spec writes use `Patch::Merge`/`Apply`.
+
+### Why
+On every reconcile after the first, `add_machine_to_cluster` re-issues the create and
+got `409 AlreadyExists` (e.g. `k0sworkerconfigs "…" already exists`). That bubbled up as
+`MachineCreationFailed`, flipping the ScheduledMachine into the **Error** phase — which
+skips ALL Active-phase work, including node-taint reconciliation. On the k0smotron
+workshop tier this presented as **Flag 1 (worker joins) passing but Flag 2 (the Node
+carries the spot taint) never happening**: the SM was wedged in Error, so it never ran
+`handle_active_phase`. Idempotent create is standard controller behavior and fixes it.
+
+### Impact
+- [ ] Breaking change
+- [x] Requires cluster rollout
+- [ ] Config change only
+- [ ] Documentation only
+
+## [2026-06-24 03:10] - Fix Mermaid flow diagrams broken by special characters
+
+**Author:** Erick Bourgeois
+
+### Changed
+- `docs/architecture/calm/architecture.json`: escaped Mermaid-hostile characters in
+  three flow transition descriptions using Mermaid entity codes (which render as the
+  literal characters):
+  - `flow-emergency-reclaim` t1: `reclaim-requested="true"` → `reclaim-requested=#quot;true#quot;`.
+    The embedded double quotes terminated the Mermaid label string early, causing a parse
+    error that left the **Emergency Reclaim** diagram blank on the docs site.
+  - `flow-kata-config-delivery` t2: `/etc/5spot/kata-config/<key>` → `…/#lt;key#gt;`.
+  - `flow-kata-config-delivery` t5: `restart <restartService>` → `restart #lt;restartService#gt;`.
+    Raw `<…>` placeholders were parsed as HTML tags and silently dropped from the rendered label.
+- `docs/src/architecture/flows.md`: regenerated via `make calm-diagrams` (auto-generated;
+  not hand-edited).
+- `docs/src/concepts/kata-config-delivery.md`: hand-written `sequenceDiagram` — replaced a
+  `;` with `,` inside a `Note over Agent,Host: …` label. Mermaid treats `;` as a statement
+  separator even inside note text, so it split the note mid-string and failed the whole
+  diagram to parse.
+
+### Why
+The `flows.md.hbs` template emits transition descriptions verbatim into `flowchart` node
+labels (`t1["…"]`). Mermaid cannot contain an unescaped `"` inside a quoted label, and
+treats `<…>` as HTML — so these characters either broke parsing (Emergency Reclaim) or
+ate label text (Kata Config Delivery). Fixing at the source-of-truth `architecture.json`
+keeps the diagrams correct after every regeneration. The `phase -> …` arrows in other
+flows contain only `>` and render fine, so they were left unchanged.
+
+### Impact
+- [ ] Breaking change
+- [ ] Requires cluster rollout
+- [ ] Config change only
+- [x] Documentation only
+
+---
+
+## [2026-06-24 02:30] - Surface the real cause of child-cluster connection failures (error_chain)
+
+**Author:** Erick Bourgeois
+
+### Changed
+- `src/reconcilers/helpers.rs`: new `error_chain(&dyn Error)` helper that flattens an
+  error's `source()` chain into one string, and wired it into all six child-cluster
+  failure logs (taint GET/PATCH/reconcile, kata clear/stamp). These previously logged
+  only the terse top-level `Display` ("client error (Connect)"), hiding the real cause.
+- `src/reconcilers/helpers_tests.rs`: +2 unit tests for `error_chain` (multi-level
+  source chain; single error with no source).
+
+### Why
+A k0smotron-hosted control plane (workshop k0smotron tier) exposes a self-signed API
+server cert that **rustls** (5-Spot's TLS backend, kube 3.1 default) rejects where
+OpenSSL/curl `-k` accept it — so 5-Spot's child client fails to taint/drain the worker
+Node with `client error (Connect)`, with no visible cause. Logging the full chain makes
+the exact rustls reason (unknown issuer / SAN mismatch / unsupported signature) visible
+so the fix can target it (a proper cert, `tls_server_name`, or `insecure-skip-tls-verify`
+on the child kubeconfig — kube already honors the latter via `Config::accept_invalid_certs`).
+
+### Impact
+- [ ] Breaking change
+- [ ] Requires cluster rollout
+- [ ] Config change only
+- [x] Diagnostics only (no behavior change)
+
 ## [2026-06-23 11:45] - Ship the spot-schedule provider binaries in the image (ADR 0009 providers were unrunnable)
 
 **Author:** Erick Bourgeois

Cargo.lock

@@ -832,7 +832,7 @@
         {
           "sequence-number": 1,
           "relationship-unique-id": "rel-reclaim-agent-workload-kube-api",
-          "description": "Agent detects a process matching spec.killIfCommands via /proc scan and PATCHes the reclaim annotation triple (5spot.finos.org/reclaim-requested=\"true\", /reclaim-reason, /reclaim-requested-at) onto its own Node via the kubelet node-scoped token. Field manager: 5spot-reclaim-agent."
+          "description": "Agent detects a process matching spec.killIfCommands via /proc scan and PATCHes the reclaim annotation triple (5spot.finos.org/reclaim-requested=#quot;true#quot;, /reclaim-reason, /reclaim-requested-at) onto its own Node via the kubelet node-scoped token. Field manager: 5spot-reclaim-agent."
         },
         {
           "sequence-number": 2,
@@ -895,7 +895,7 @@
         {
           "sequence-number": 2,
           "relationship-unique-id": "rel-kata-agent-workload-kube-api",
-          "description": "The kata config agent DaemonSet (scheduled onto the node by the opt-in label) watches its per-node ConfigMap and reads the projected drop-in content at /etc/5spot/kata-config/<key>.",
+          "description": "The kata config agent DaemonSet (scheduled onto the node by the opt-in label) watches its per-node ConfigMap and reads the projected drop-in content at /etc/5spot/kata-config/#lt;key#gt;.",
           "direction": "destination-to-source"
         },
         {
@@ -911,7 +911,7 @@
         {
           "sequence-number": 5,
           "relationship-unique-id": "rel-kata-agent-writes-host",
-          "description": "Agent runs `nsenter -t 1 -m -u -i -n -p -- systemctl restart <restartService>` to restart the host k0s service; containerd reloads the new drop-in. The restart SIGKILLs the agent pod; systemd still completes the D-Bus job. Kubelet restarts the agent; on the next loop hashes match the applied annotation, so it is a no-op. Single-cycle convergence."
+          "description": "Agent runs `nsenter -t 1 -m -u -i -n -p -- systemctl restart #lt;restartService#gt;` to restart the host k0s service; containerd reloads the new drop-in. The restart SIGKILLs the agent pod; systemd still completes the D-Bus job. Kubelet restarts the agent; on the next loop hashes match the applied annotation, so it is a no-op. Single-cycle convergence."
         }
       ],
       "controls": {

docs/architecture/calm/architecture.json

@@ -86,7 +86,7 @@ sequenceDiagram
             Note over Agent: Applied record written BEFORE<br/>the restart — the restart-loop guard
             Agent->>Host: nsenter -t 1 -m -u -i -n -p --<br/>systemctl restart k0sworker.service
             Host-->>Agent: containerd bounces → pod SIGKILLed
-            Note over Agent,Host: kubelet restarts the agent;<br/>next tick: hash matches → no-op.<br/>Single-cycle convergence.
+            Note over Agent,Host: kubelet restarts the agent,<br/>next tick: hash matches → no-op.<br/>Single-cycle convergence.
         else hashes match
             Agent->>Agent: no-op (sleep 30s)
         end

docs/src/concepts/kata-config-delivery.md

@@ -180,7 +180,12 @@ pub async fn add_finalizer(
     let api: Api<ScheduledMachine> = Api::namespaced(ctx.client.clone(), &namespace);
 
     let mut finalizers = resource.meta().finalizers.clone().unwrap_or_default();
-    finalizers.push(FINALIZER_SCHEDULED_MACHINE.to_string());
+    // Idempotent: only add ours if it is not already present, so a double-call (or a
+    // caller that forgets the has_finalizer() guard) can never write a duplicate
+    // finalizer — which would otherwise wedge deletion (the GC waits for every copy).
+    if !finalizers.iter().any(|f| f == FINALIZER_SCHEDULED_MACHINE) {
+        finalizers.push(FINALIZER_SCHEDULED_MACHINE.to_string());
+    }
 
     let patch = json!({
         "metadata": {
@@ -1557,9 +1562,24 @@ async fn create_dynamic_resource(
     let dyn_obj: kube::core::DynamicObject =
         serde_json::from_value(obj).map_err(kube::Error::SerdeError)?;
 
-    api.create(&kube::api::PostParams::default(), &dyn_obj)
-        .await?;
-    Ok(())
+    match api
+        .create(&kube::api::PostParams::default(), &dyn_obj)
+        .await
+    {
+        Ok(_) => Ok(()),
+        // Idempotent reconciliation: a prior reconcile already created this object
+        // (bootstrap / infrastructure / Machine), which IS the desired state. Treat
+        // 409 AlreadyExists as success — otherwise every re-reconcile after the first
+        // create fails the ScheduledMachine into the Error phase, which then skips all
+        // Active-phase work (node-taint reconciliation, status projection, …). This
+        // surfaced on the k0smotron tier as Flag 1 (worker joins) passing but Flag 2
+        // (node carries the spot taint) never happening.
+        Err(kube::Error::Api(ae)) if ae.code == 409 => {
+            debug!(kind = %kind, "resource already exists; treating create as idempotent success");
+            Ok(())
+        }
+        Err(e) => Err(e),
+    }
 }
 
 /// Split a Kubernetes `apiVersion` string into `(group, version)`.
@@ -3139,7 +3159,7 @@ pub async fn reconcile_kata_config_delivery(
             )
             .await
             .map_err(|e| {
-                error!(node = %node_name, error = %e, "Failed to clear kata-config reference annotation");
+                error!(node = %node_name, error = %e, cause = %error_chain(&e), "Failed to clear kata-config reference annotation");
                 ReconcilerError::KubeError(e)
             })?;
         return Ok(KataDeliveryOutcome::Disabled);
@@ -3169,7 +3189,7 @@ pub async fn reconcile_kata_config_delivery(
             .patch(node_name, &PatchParams::default(), &Patch::Merge(&patch))
             .await
             .map_err(|e| {
-                error!(node = %node_name, error = %e, "Failed to stamp kata-config Node opt-in");
+                error!(node = %node_name, error = %e, cause = %error_chain(&e), "Failed to stamp kata-config Node opt-in");
                 ReconcilerError::KubeError(e)
             })?;
         debug!(
@@ -3199,7 +3219,7 @@ pub async fn reconcile_kata_config_delivery(
         )
         .await
         .map_err(|e| {
-            error!(node = %node_name, error = %e, "Failed to clear kata-config reference annotation");
+            error!(node = %node_name, error = %e, cause = %error_chain(&e), "Failed to clear kata-config reference annotation");
             ReconcilerError::KubeError(e)
         })?;
 
@@ -3354,6 +3374,26 @@ pub fn diff_node_taints(
     plan
 }
 
+/// Flatten an error's [`source`](std::error::Error::source) chain into one string.
+///
+/// kube/hyper surface child-cluster connection failures with a terse top-level
+/// `Display` ("client error (Connect)") that hides the real cause — most often a
+/// rustls certificate rejection (unknown issuer, name/SAN mismatch, unsupported
+/// signature scheme) when talking to a hosted control plane whose self-signed cert
+/// OpenSSL/curl would accept but rustls (stricter) will not. Logging this chain at
+/// the failure site makes that cause visible instead of guessing.
+#[must_use]
+pub fn error_chain(err: &dyn std::error::Error) -> String {
+    let mut out = err.to_string();
+    let mut source = err.source();
+    while let Some(cause) = source {
+        out.push_str(" -> ");
+        out.push_str(&cause.to_string());
+        source = cause.source();
+    }
+    out
+}
+
 /// Apply a [`NodeTaintPlan`] to the named Node via a single server-side apply
 /// PATCH, and write the `5spot.finos.org/applied-taints` ownership annotation.
 ///
@@ -3383,7 +3423,7 @@ pub async fn apply_node_taints(
 
     let nodes: Api<Node> = Api::all(client.clone());
     let current_node = nodes.get(node_name).await.map_err(|e| {
-        error!(node = %node_name, error = %e, "Failed to GET Node for taint apply");
+        error!(node = %node_name, error = %e, cause = %error_chain(&e), "Failed to GET Node for taint apply");
         ReconcilerError::KubeError(e)
     })?;
 
@@ -3460,7 +3500,7 @@ pub async fn apply_node_taints(
         .patch(node_name, &params, &Patch::Apply(&patch))
         .await
         .map_err(|e| {
-            error!(node = %node_name, error = %e, "Failed to PATCH Node taints");
+            error!(node = %node_name, error = %e, cause = %error_chain(&e), "Failed to PATCH Node taints");
             ReconcilerError::KubeError(e)
         })?;
 
@@ -3539,7 +3579,7 @@ pub async fn reconcile_node_taints(
             return Ok(NodeTaintReconcileOutcome::NoNodeYet);
         }
         Err(e) => {
-            error!(node = %input.node_name, error = %e, "Failed to GET Node for taint reconcile");
+            error!(node = %input.node_name, error = %e, cause = %error_chain(&e), "Failed to GET Node for taint reconcile");
             return Err(ReconcilerError::KubeError(e));
         }
     };