linode: tag every LKE volume at birth via StorageClass volumeTags (#1311)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>

Author: 3 people

.claude/skills/linode-ha-provisioning/SKILL.md

@@ -289,4 +289,4 @@ curl -sS -m 240 --fail-with-body -X POST "$RELAY/linode/destroy-lke" \
   -d "$(jq -n --arg id "$RUN_ID" '{run_id:$id}')"
 ```
 
-Delete the cluster whether the run passed, failed, or was blocked — this is the last step, always. Unlike the single-VM path there is **no on-box self-destruct timer**; the guarantee is the **relay's TTL reaper**, which deletes any `pmm-qa-ephemeral` cluster past its `expires-<epoch>` tag (default 24h) even if this call never runs — the LKE equivalent of the VM's on-box timer. The SessionEnd hook fires the same `/linode/destroy-lke` for any run dir carrying an `lke` marker, so a normal session cleans up on its own; still call it explicitly at end of run — the reaper is the backstop, not the primary path. If you also created a box with `linode-docker-provisioning`, tear that VM down too — destroying the cluster does not touch it. Both teardown and the reaper now delete the cluster's unique account-level tags (`expires-<epoch>`, `pmm-qa-run:<id>`); Linode leaves those behind otherwise, so they pile up. Sweep leftovers on the relay with `LINODE_TOKEN=… terraform/linode-runner/prune-tags.sh --dry-run` (then without `--dry-run`). The **relay reaper** also deletes the orphaned Block Storage volumes and NodeBalancer that `cluster-delete` leaves behind — the biggest HA cost leak — via `.claude/skills/linode-ha-provisioning/scripts/prune-lke-orphans.sh` (`--dry-run` to preview). It deletes by **positive attribution**. **Volumes**: tagged with the cluster's `pmm-qa-run:<id>` at provision *and* again right before every cluster delete (`destroy-lke` and the reaper both call `tag-lke-resources.sh` while the nodes still exist), so volumes created after provisioning are covered too; the sweep removes a volume only when its run has **no live cluster**. **NodeBalancers**: attributed by their immutable `lke<clusterid>-` label — the Linode CCM reconciles a NodeBalancer's tags back to its defaults, so a `pmm-qa-run` tag does **not** survive on it and cannot be used; the sweep removes one only when its `lke<id>` cluster no longer exists. A live cluster's resources (volumes unattached during provisioning/failover) and other owners' `pvc-*` volumes are never touched. `destroy-lke` does **not** sweep (cluster-delete is async — it would only see other runs' resources); the reaper owns it.
+Delete the cluster whether the run passed, failed, or was blocked — this is the last step, always. Unlike the single-VM path there is **no on-box self-destruct timer**; the guarantee is the **relay's TTL reaper**, which deletes any `pmm-qa-ephemeral` cluster past its `expires-<epoch>` tag (default 24h) even if this call never runs — the LKE equivalent of the VM's on-box timer. The SessionEnd hook fires the same `/linode/destroy-lke` for any run dir carrying an `lke` marker, so a normal session cleans up on its own; still call it explicitly at end of run — the reaper is the backstop, not the primary path. If you also created a box with `linode-docker-provisioning`, tear that VM down too — destroying the cluster does not touch it. Both teardown and the reaper now delete the cluster's unique account-level tags (`expires-<epoch>`, `pmm-qa-run:<id>`); Linode leaves those behind otherwise, so they pile up. Sweep leftovers on the relay with `LINODE_TOKEN=… terraform/linode-runner/prune-tags.sh --dry-run` (then without `--dry-run`). The **relay reaper** also deletes the orphaned Block Storage volumes and NodeBalancer that `cluster-delete` leaves behind — the biggest HA cost leak — via `.claude/skills/linode-ha-provisioning/scripts/prune-lke-orphans.sh` (`--dry-run` to preview). It deletes by **positive attribution**. **Volumes**: `create-lke` recreates the cluster's default StorageClass (`linode-block-storage-retain`) with `linodebs.csi.linode.com/volumeTags: pmm-qa-ephemeral,pmm-qa-run:<id>`, so the CSI driver stamps the run tag on **every** volume at creation — attached or not, at provision or later. (The provision EXIT-trap and the pre-delete `tag-lke-resources.sh` in `destroy-lke`/reaper stay as a backstop for the window before the SC applies.) The sweep removes a volume only when its run has **no live cluster**. **NodeBalancers**: attributed by their immutable `lke<clusterid>-` label — the Linode CCM reconciles a NodeBalancer's tags back to its defaults, so a `pmm-qa-run` tag does **not** survive on it and cannot be used; the sweep removes one only when its `lke<id>` cluster no longer exists. A live cluster's resources (volumes unattached during provisioning/failover) and other owners' `pvc-*` volumes are never touched. `destroy-lke` does **not** sweep (cluster-delete is async — it would only see other runs' resources); the reaper owns it.

.claude/skills/linode-ha-provisioning/scripts/create-lke-pmm-ha.sh

@@ -131,6 +131,25 @@ until [ "$(kubectl get nodes --no-headers 2>/dev/null | grep -c .)" -ge "$NODE_C
 kubectl wait --for=condition=Ready nodes --all --timeout=300s
 kubectl get nodes
 
+# --- storage class: tag every CSI volume at birth ----------------------------
+# LKE's default SC has no volumeTags and a StorageClass's parameters are immutable,
+# so its volumes are born untagged and prune-lke-orphans.sh can never attribute them.
+kubectl delete storageclass linode-block-storage-retain --ignore-not-found
+kubectl apply -f - <<EOF
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+  name: linode-block-storage-retain
+  annotations:
+    storageclass.kubernetes.io/is-default-class: "true"
+provisioner: linodebs.csi.linode.com
+parameters:
+  linodebs.csi.linode.com/volumeTags: "pmm-qa-ephemeral,pmm-qa-run:$RUN_ID"
+reclaimPolicy: Retain
+volumeBindingMode: Immediate
+allowVolumeExpansion: true
+EOF
+
 # --- dependencies (operators) ------------------------------------------------
 kubectl create namespace "$NAMESPACE" --dry-run=client -o yaml | kubectl apply -f -