linode: attribute NodeBalancers by lke<id> label; tag volumes before every cluster delete (#1296)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>

Author: travagliad

.claude/integrations/slack/relay/relay.js

@@ -597,6 +597,11 @@ async function reapLke() {
         for (const t of tags) { const mm = /^expires-(\d+)$/.exec(t); if (mm) { expiry = Number(mm[1]); break; } }
         if (expiry == null) { const created = Date.parse(c.created || "") / 1000; expiry = Number.isFinite(created) ? created + hardMax : 0; }
         if (nowS > expiry) {
+          // Tag the cluster's volumes before deleting it: cluster-delete does not
+          // cascade to them, and only an attached volume can be attributed. Best-effort.
+          try {
+            await execFileP("bash", [`${HA_DIR}/tag-lke-resources.sh`, String(c.id)], { env: process.env, timeout: 120000, maxBuffer: 4 * 1024 * 1024 });
+          } catch (e) { console.error(`lke-reaper: pre-delete volume tag skipped (non-fatal) for ${c.id}: ${e.message}`); }
           const del = await linodeApi(`/lke/clusters/${c.id}`, { method: "DELETE" });
           console.log(`lke-reaper: deleted cluster ${c.id} "${c.label}" (expiry ${expiry} < now ${nowS}) -> ${del.status}`);
           if (del.ok) {

.claude/skills/linode-ha-provisioning/SKILL.md

@@ -200,4 +200,4 @@ curl -sS -m 240 --fail-with-body -X POST "$RELAY/linode/destroy-lke" \
   -d "$(jq -n --arg id "$RUN_ID" '{run_id:$id}')"
 ```
 
-Delete the cluster whether the run passed, failed, or was blocked — this is the last step, always. Unlike the single-VM path there is **no on-box self-destruct timer**; the guarantee is the **relay's TTL reaper**, which deletes any `pmm-qa-ephemeral` cluster past its `expires-<epoch>` tag (default 24h) even if this call never runs — the LKE equivalent of the VM's on-box timer. The SessionEnd hook fires the same `/linode/destroy-lke` for any run dir carrying an `lke` marker, so a normal session cleans up on its own; still call it explicitly at end of run — the reaper is the backstop, not the primary path. If you also created a box with `linode-docker-provisioning`, tear that VM down too — destroying the cluster does not touch it. Both teardown and the reaper now delete the cluster's unique account-level tags (`expires-<epoch>`, `pmm-qa-run:<id>`); Linode leaves those behind otherwise, so they pile up. Sweep leftovers on the relay with `LINODE_TOKEN=… terraform/linode-runner/prune-tags.sh --dry-run` (then without `--dry-run`). The **relay reaper** also deletes the orphaned Block Storage volumes and NodeBalancer that `cluster-delete` leaves behind — the biggest HA cost leak — via `.claude/skills/linode-ha-provisioning/scripts/prune-lke-orphans.sh` (`--dry-run` to preview). It deletes by **positive attribution**: `create-lke` stamps each cluster's volumes **and its NodeBalancer** with its `pmm-qa-run:<id>` tag (on every exit path, so failed bring-ups are covered), and the sweep removes one only when that run has **no live cluster** — so a live cluster's resources (volumes unattached during provisioning/failover) and other owners' volumes/NodeBalancers are never touched. `destroy-lke` does **not** sweep (cluster-delete is async — it would only see other runs' resources); the reaper owns it.
+Delete the cluster whether the run passed, failed, or was blocked — this is the last step, always. Unlike the single-VM path there is **no on-box self-destruct timer**; the guarantee is the **relay's TTL reaper**, which deletes any `pmm-qa-ephemeral` cluster past its `expires-<epoch>` tag (default 24h) even if this call never runs — the LKE equivalent of the VM's on-box timer. The SessionEnd hook fires the same `/linode/destroy-lke` for any run dir carrying an `lke` marker, so a normal session cleans up on its own; still call it explicitly at end of run — the reaper is the backstop, not the primary path. If you also created a box with `linode-docker-provisioning`, tear that VM down too — destroying the cluster does not touch it. Both teardown and the reaper now delete the cluster's unique account-level tags (`expires-<epoch>`, `pmm-qa-run:<id>`); Linode leaves those behind otherwise, so they pile up. Sweep leftovers on the relay with `LINODE_TOKEN=… terraform/linode-runner/prune-tags.sh --dry-run` (then without `--dry-run`). The **relay reaper** also deletes the orphaned Block Storage volumes and NodeBalancer that `cluster-delete` leaves behind — the biggest HA cost leak — via `.claude/skills/linode-ha-provisioning/scripts/prune-lke-orphans.sh` (`--dry-run` to preview). It deletes by **positive attribution**. **Volumes**: tagged with the cluster's `pmm-qa-run:<id>` at provision *and* again right before every cluster delete (`destroy-lke` and the reaper both call `tag-lke-resources.sh` while the nodes still exist), so volumes created after provisioning are covered too; the sweep removes a volume only when its run has **no live cluster**. **NodeBalancers**: attributed by their immutable `lke<clusterid>-` label — the Linode CCM reconciles a NodeBalancer's tags back to its defaults, so a `pmm-qa-run` tag does **not** survive on it and cannot be used; the sweep removes one only when its `lke<id>` cluster no longer exists. A live cluster's resources (volumes unattached during provisioning/failover) and other owners' `pvc-*` volumes are never touched. `destroy-lke` does **not** sweep (cluster-delete is async — it would only see other runs' resources); the reaper owns it.

.claude/skills/linode-ha-provisioning/scripts/create-lke-pmm-ha.sh

@@ -112,24 +112,14 @@ _diag() {
     kubectl get events -n "$NAMESPACE" --sort-by=.metadata.creationTimestamp >"$RUN_DIR/events.txt" 2>&1 || true
     kubectl describe pods -n "$NAMESPACE" >"$RUN_DIR/describe.txt" 2>&1 || true
 }
-# Stamp this cluster's volumes + NodeBalancer with pmm-qa-run:<id> so teardown
+# Stamp this cluster's volumes with pmm-qa-run:<id> so teardown
 # (prune-lke-orphans.sh) can attribute them. Runs on every exit path via the trap
-# so a failed bring-up is tagged too. Best-effort.
+# so a failed bring-up is tagged too. Best-effort. destroy-lke / the reaper tag
+# again right before deleting the cluster, catching volumes created after this.
 _tag_for_teardown() {
     [ -n "${CLUSTER_ID:-}" ] || return 0
-    local ids vols lid vid nbid
-    ids="$(linode-cli lke pools-list "$CLUSTER_ID" --json 2>/dev/null | jq -r '.[].nodes[].instance_id' 2>/dev/null)"
-    vols="$(linode-cli volumes list --page-size 500 --json 2>/dev/null)"
-    for lid in $ids; do
-        [ -n "$lid" ] && [ "$lid" != "null" ] || continue
-        for vid in $(printf '%s' "$vols" | jq -r --argjson l "$lid" '.[] | select(.linode_id==$l) | .id' 2>/dev/null); do
-            linode-cli volumes update "$vid" --tags pmm-qa-ephemeral --tags "pmm-qa-run:$RUN_ID" >/dev/null 2>&1 || true
-        done
-    done
-    for nbid in $(linode-cli nodebalancers list --page-size 500 --json 2>/dev/null \
-                    | jq -r --arg c "lke$CLUSTER_ID-" '.[] | select(.label|startswith($c)) | .id' 2>/dev/null); do
-        linode-cli nodebalancers update "$nbid" --tags pmm-qa-ephemeral --tags "pmm-qa-run:$RUN_ID" >/dev/null 2>&1 || true
-    done
+    local SD; SD="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+    LINODE_TOKEN="$LINODE_TOKEN" bash "$SD/tag-lke-resources.sh" "$CLUSTER_ID" "$RUN_ID" || true
 }
 trap '_diag; _tag_for_teardown' EXIT
 # Linode reports the pool "ready" before the nodes register with the k8s API

.claude/skills/linode-ha-provisioning/scripts/destroy-lke.sh

@@ -14,12 +14,17 @@ if [ -z "$CLUSTER_ID" ] && [ -n "${RUN_ID:-}" ]; then
 fi
 [ -n "$CLUSTER_ID" ] || { echo "usage: destroy-lke.sh <cluster_id>  (or set RUN_ID)" >&2; exit 2; }
 
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+# Tag the cluster's volumes before deleting it: cluster-delete does not cascade to
+# them, and only an attached volume can be attributed. Best-effort.
+LINODE_TOKEN="$LINODE_TOKEN" bash "$SCRIPT_DIR/tag-lke-resources.sh" "$CLUSTER_ID" \
+  || echo "[pmm-ha] volume tagging skipped (non-fatal)" >&2
+
 echo "[pmm-ha] Deleting LKE cluster $CLUSTER_ID"
 linode-cli lke cluster-delete "$CLUSTER_ID"
 echo "[pmm-ha] Deleted."
 
-SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-
 # Sweep the cluster's orphan tags (best-effort).
 PRUNE="$SCRIPT_DIR/../../../../terraform/linode-runner/prune-tags.sh"
 if [ -x "$PRUNE" ]; then

.claude/skills/linode-ha-provisioning/scripts/prune-lke-orphans.sh

@@ -15,12 +15,10 @@
 #     pmm-qa-run:<id> tag whose run has NO live cluster -- so a live cluster's
 #     volumes (its run is still listed) and untagged/other-owner volumes are
 #     never touched.
-#   * NodeBalancers: create-lke stamps the cluster's NodeBalancer with the same
-#     pmm-qa-run:<id> tag. Delete only one carrying that tag whose run has no live
-#     cluster -- so another owner's NB, or a deliberately-kept migrate target, is
-#     never touched.
-# If a volume was never tagged (tagging is best-effort at provision), it is left
-# alone -- the failure mode is a leak (recoverable by hand), never a wrong delete.
+#   * NodeBalancers: attributed by their immutable lke<clusterid>- label, deleted
+#     only when the cluster is gone AND no backend node is up (see the loop).
+# If a volume was never tagged (tagging is best-effort), it is left alone -- the
+# failure mode is a leak (recoverable by hand), never a wrong delete.
 #
 # Requires the token to carry Volumes and NodeBalancers Read/Write (plus LKE Read).
 # API-only (curl+jq); no cluster access needed.
@@ -48,6 +46,7 @@ pages() {  # $1 = path (may contain ?query); prints each .data[] as compact JSON
 
 clusters="$(pages /lke/clusters)"
 live_runs="$(printf '%s' "$clusters" | jq -r '.tags[]? | select(startswith("pmm-qa-run:"))' | sort -u)"
+live_cluster_ids="$(printf '%s' "$clusters" | jq -r '.id' | sort -u)"
 
 vol=0 nb=0 fail=0
 # Volumes: ours (pmm-qa-run:<id> tag) AND run has no live cluster AND unattached.
@@ -63,20 +62,28 @@ done < <(pages /volumes | jq -r '
   . as $v | ($v.tags[]? | select(startswith("pmm-qa-run:"))) as $t
   | [$v.id, $v.label, $t, ($v.linode_id|tostring)] | @tsv')
 
-# NodeBalancers: ours (pmm-qa-run:<id> tag, stamped at provision) AND run has no
-# live cluster. Same positive attribution as volumes -- a NodeBalancer without
-# our run tag (another owner's, or a deliberately-kept migrate target) is never
-# touched, even if its lke<id> cluster is gone.
-while IFS=$'\t' read -r id label run; do
+# NodeBalancers: the CCM strips custom tags, so attribute by the immutable
+# lke<clusterid>- label instead. Delete one only when its cluster is no longer
+# live AND no backend node is up -- the idle check guards against an NB adopted
+# and reused by another live cluster, which keeps the original cluster's id in
+# its label.
+while IFS=$'\t' read -r id label; do
   [ -n "$id" ] || continue
-  grep -qx "$run" <<<"$live_runs" && continue
-  if [ "$DRY" -eq 1 ]; then echo "would delete nodebalancer $id ($label) [$run]"; nb=$((nb + 1)); continue; fi
+  cid="$(sed -n 's/^lke\([0-9]\{1,\}\)-.*/\1/p' <<<"$label")"
+  [ -n "$cid" ] || continue                       # not an LKE NodeBalancer -> never ours
+  grep -qx "$cid" <<<"$live_cluster_ids" && continue   # its cluster is still live -> keep
+  # An orphan has zero backends up once its cluster is gone; a reused NB (label
+  # keeps the old cluster id) still serves traffic. If configs are unreadable, keep
+  # it -- fail safe.
+  cfg="$("${CURL[@]}" "$BASE/nodebalancers/$id/configs?page_size=100" 2>/dev/null)" \
+    || { echo "keeping nodebalancer $id ($label): configs unreadable" >&2; continue; }
+  up="$(printf '%s' "$cfg" | jq '[.data[]?.nodes_status.up // 0] | add // 0')"
+  [ "${up:-0}" -eq 0 ] || { echo "keeping nodebalancer $id ($label): $up backend(s) up (in use)" >&2; continue; }
+  if [ "$DRY" -eq 1 ]; then echo "would delete nodebalancer $id ($label) [cluster $cid gone, idle]"; nb=$((nb + 1)); continue; fi
   if "${CURL[@]}" -o /dev/null -X DELETE "$BASE/nodebalancers/$id"; then
-    echo "deleted nodebalancer $id ($label) [$run]"; nb=$((nb + 1))
+    echo "deleted nodebalancer $id ($label) [cluster $cid gone, idle]"; nb=$((nb + 1))
   else echo "FAILED nodebalancer $id ($label)" >&2; fail=$((fail + 1)); fi
-done < <(pages /nodebalancers | jq -r '
-  . as $n | ($n.tags[]? | select(startswith("pmm-qa-run:"))) as $t
-  | [$n.id, $n.label, $t] | @tsv')
+done < <(pages /nodebalancers | jq -r '[.id, .label] | @tsv')
 
 echo "prune-lke-orphans: volumes=$vol nodebalancers=$nb failed=$fail$([ "$DRY" -eq 1 ] && echo ' (dry-run)')"
 [ "$fail" -eq 0 ]

.claude/skills/linode-ha-provisioning/scripts/tag-lke-resources.sh

@@ -0,0 +1,53 @@
+#!/usr/bin/env bash
+# Tag an LKE cluster's attached Block Storage volumes with its pmm-qa-run:<id> tag
+# so prune-lke-orphans.sh can attribute and delete them once the cluster is gone.
+# Must run while the cluster's nodes still exist -- cluster-delete does not cascade
+# to the volumes, and only an attached volume can be linked back to the cluster
+# here (a volume unattached at call time is not tagged). NodeBalancers are not
+# tagged: the CCM reconciles their tags away; the sweep attributes them by their
+# lke<id> label instead. Best-effort and API-only; never fails its caller.
+#   LINODE_TOKEN=... tag-lke-resources.sh <cluster_id> [run_id]
+set -euo pipefail
+: "${LINODE_TOKEN:?LINODE_TOKEN must be set}"
+CID="${1:?usage: tag-lke-resources.sh <cluster_id> [run_id]}"
+RUN="${2:-}"
+
+BASE="https://api.linode.com/v4"
+CURL=(curl -fsS --connect-timeout 10 --max-time 30 -H "Authorization: Bearer $LINODE_TOKEN")
+
+# Resolve the run id from the cluster's own pmm-qa-run tag (or its label) when the
+# caller did not pass one -- destroy-lke / the reaper only know the cluster id.
+if [ -z "$RUN" ]; then
+  cluster="$("${CURL[@]}" "$BASE/lke/clusters/$CID" 2>/dev/null || true)"
+  RUN="$(printf '%s' "$cluster" | jq -r '.tags[]? | select(startswith("pmm-qa-run:")) | sub("^pmm-qa-run:";"")' 2>/dev/null | head -1)"
+  [ -n "$RUN" ] || RUN="$(printf '%s' "$cluster" | jq -r '.label // empty' 2>/dev/null | sed 's/^pmm-ha-//')"
+fi
+[ -n "$RUN" ] || { echo "tag-lke-resources: could not resolve a run id for cluster $CID (skipping)" >&2; exit 0; }
+
+ids="$("${CURL[@]}" "$BASE/lke/clusters/$CID/pools?page_size=500" 2>/dev/null \
+        | jq -r '.data[].nodes[].instance_id' 2>/dev/null | grep -vx null || true)"
+
+# All volumes as JSONL, across every page -- the account can hold more than one.
+vols=""
+page=1 total=1
+while [ "$page" -le "$total" ]; do
+  resp="$("${CURL[@]}" "$BASE/volumes?page=$page&page_size=500" 2>/dev/null || true)"
+  [ -n "$resp" ] || break
+  vols+="$(printf '%s' "$resp" | jq -c '.data[]')"$'\n'
+  total="$(printf '%s' "$resp" | jq -r '.pages // 1')"
+  page=$((page + 1))
+done
+
+n=0
+for lid in $ids; do
+  while IFS= read -r vjson; do
+    [ -n "$vjson" ] || continue
+    vid="$(printf '%s' "$vjson" | jq -r '.id')"
+    # Merge our tags with the volume's existing ones (PUT replaces the whole list),
+    # so unrelated tags are preserved.
+    tags="$(printf '%s' "$vjson" | jq -c --arg r "pmm-qa-run:$RUN" '((.tags // []) + ["pmm-qa-ephemeral", $r]) | unique')"
+    "${CURL[@]}" -o /dev/null -X PUT -H "Content-Type: application/json" -d "{\"tags\":$tags}" \
+      "$BASE/volumes/$vid" && n=$((n + 1)) || echo "tag-lke-resources: tag failed for volume $vid (non-fatal)" >&2
+  done < <(printf '%s\n' "$vols" | jq -c --argjson l "$lid" 'select(.linode_id==$l)')
+done
+echo "tag-lke-resources: cluster $CID run=$RUN tagged $n volume(s)"