up

Author: mitchross

.wolf/anatomy.md

@@ -1,14 +1,14 @@
 # anatomy.md
 
-> Auto-maintained by OpenWolf. Last scanned: 2026-04-09T01:45:00.204Z
+> Auto-maintained by OpenWolf. Last scanned: 2026-04-09T01:45:45.973Z
 > Files: 518 tracked | Anatomy hits: 0 | Misses: 0
 
 ## ./
 
 - `.gitattributes` — Git attributes (~6 tok)
 - `.gitignore` — Git ignore rules (~611 tok)
 - `astro.config.mjs` — Astro configuration (~274 tok)
-- `CLAUDE.md` — OpenWolf (~3047 tok)
+- `CLAUDE.md` — OpenWolf (~3081 tok)
 - `firewalla-dns-config.txt` — Firewalla Local DNS Configuration for vanillax.me (~424 tok)
 - `MIGRATION_EXTERNAL_DNS.md` — Migration to ExternalDNS-Based Split DNS Architecture (~1870 tok)
 - `mkdocs.yml` (~269 tok)
@@ -150,7 +150,7 @@
 
 ## infrastructure/controllers/kyverno/
 
-- `CLAUDE.md` — Kyverno Backup & Restore System (~2614 tok)
+- `CLAUDE.md` — Kyverno Backup & Restore System (~3270 tok)
 - `kustomization.yaml` — K8s Kustomization: kyverno (~338 tok)
 - `namespace.yaml` — K8s Namespace: kyverno (~17 tok)
 - `rbac-patch.yaml` — K8s ClusterRole: kyverno:background-controller:volsync (~759 tok)

.wolf/hooks/_session.json

@@ -23,8 +23,8 @@
       "first_read": "2026-04-08T23:10:14.425Z"
     },
     "/home/vanillax/programming/talos-argocd-proxmox/infrastructure/controllers/kyverno/CLAUDE.md": {
-      "count": 2,
-      "tokens": 2907,
+      "count": 3,
+      "tokens": 3047,
       "first_read": "2026-04-08T23:10:21.123Z"
     },
     "/home/vanillax/programming/talos-argocd-proxmox/docs/pvc-plumber-full-flow.md": {
@@ -88,8 +88,8 @@
       "first_read": "2026-04-09T00:37:54.042Z"
     },
     "/home/vanillax/programming/talos-argocd-proxmox/CLAUDE.md": {
-      "count": 2,
-      "tokens": 2907,
+      "count": 4,
+      "tokens": 3047,
       "first_read": "2026-04-09T01:44:34.360Z"
     }
   },
@@ -189,6 +189,18 @@
       "action": "edit",
       "tokens": 211,
       "at": "2026-04-09T01:45:00.210Z"
+    },
+    {
+      "file": "/home/vanillax/programming/talos-argocd-proxmox/infrastructure/controllers/kyverno/CLAUDE.md",
+      "action": "edit",
+      "tokens": 758,
+      "at": "2026-04-09T01:45:26.520Z"
+    },
+    {
+      "file": "/home/vanillax/programming/talos-argocd-proxmox/CLAUDE.md",
+      "action": "edit",
+      "tokens": 83,
+      "at": "2026-04-09T01:45:45.979Z"
     }
   ],
   "edit_counts": {
@@ -200,11 +212,12 @@
     "infrastructure/controllers/kyverno/policies/volsync-pvc-validate.yaml": 2,
     "infrastructure/controllers/kyverno/policies/volsync-pvc-mutate.yaml": 1,
     "infrastructure/controllers/kyverno/policies/volsync-pvc-generate.yaml": 2,
-    "CLAUDE.md": 1
+    "CLAUDE.md": 2,
+    "infrastructure/controllers/kyverno/CLAUDE.md": 1
   },
   "anatomy_hits": 15,
   "anatomy_misses": 3,
-  "repeated_reads_warned": 17,
+  "repeated_reads_warned": 20,
   "cerebrum_warnings": 0,
   "stop_count": 43
 }

.wolf/memory.md

@@ -298,3 +298,5 @@
 | 21:44 | Edited my-apps/home/project-zomboid/deployment.yaml | 3→6 lines | ~46 |
 | 21:44 | Session end: 15 writes across 8 files (pvc.yaml, deployment.yaml, 2026-04-09-kyverno-cel-migration.md, values.yaml, emergency-webhook-cleanup.sh) | 18 reads | ~46306 tok |
 | 21:45 | Edited CLAUDE.md | 1→3 lines | ~197 |
+| 21:45 | Edited infrastructure/controllers/kyverno/CLAUDE.md | expanded (+57 lines) | ~707 |
+| 21:45 | Edited CLAUDE.md | 1→2 lines | ~77 |

CLAUDE.md

@@ -128,6 +128,8 @@ docs/                   # Documentation
 - Omit Kyverno canonical defaults (`emitWarning`, `validationFailureAction`, `skipBackgroundRequests`) from policy YAML — **Kyverno webhook adds them, ArgoCD detects the diff, app shows OutOfSync**
 - Create external HTTPRoutes without the three required pieces: `external-dns: "true"` label, `external-dns.alpha.kubernetes.io/target: vanillax.me` annotation, and `sectionName: https` — **DNS won't be created and Cloudflare tunnel routing fails silently**
 - Use `Replace=true,Force=true` sync-options on Jobs — causes duplicate Job execution bug ([#24005](https://github.com/argoproj/argo-cd/issues/24005)); use ArgoCD hooks instead
+- Auto-merge major Helm chart version bumps for critical infrastructure (kube-prometheus-stack, longhorn, kyverno, cilium) — **a kube-prometheus-stack v82→v83 auto-merge caused a full cluster outage on 2026-04-08 via Kyverno webhook deadlock**. Pin Renovate to minor/patch only for these charts.
+- Remove infrastructure namespaces from Kyverno webhook exclusions in `values.yaml` — **longhorn-system, argocd, volsync-system, etc. MUST be excluded or a Kyverno crash causes full cluster deadlock**. See `infrastructure/controllers/kyverno/CLAUDE.md` for details.
 
 ## Nested CLAUDE.md Files
 
@@ -182,3 +184,4 @@ Detailed instructions load automatically when working in these directories:
 - **[docs/argocd.md](docs/argocd.md)** - ArgoCD documentation
 - **[docs/vpa-resource-optimization.md](docs/vpa-resource-optimization.md)** - VPA auto-scaling
 - **[docs/plans/2026-03-22-alloy-otel-honeycomb-design.md](docs/plans/2026-03-22-alloy-otel-honeycomb-design.md)** - OTEL + Honeycomb observability design
+- **[scripts/emergency-webhook-cleanup.sh](scripts/emergency-webhook-cleanup.sh)** - Emergency recovery from Kyverno webhook deadlock

infrastructure/controllers/kyverno/CLAUDE.md

@@ -227,6 +227,63 @@ spec:
 
 **If you need to re-process existing resources after a policy change**, do a one-time ArgoCD sync or manually trigger resource re-admission — don't enable `mutateExistingOnPolicyUpdate`.
 
+## Critical: Webhook Deadlock Prevention
+
+**Incident: 2026-04-08 — Full cluster outage caused by Kyverno webhook deadlock.**
+
+### What happened
+
+1. Renovate auto-merged a kube-prometheus-stack v82→v83 Helm chart upgrade
+2. ArgoCD synced it, restarting many Prometheus pods simultaneously
+3. The restart flood overwhelmed Kyverno's admission controller — its internal cache sync failed
+4. Kyverno crashed with `"failed to wait for cache sync"` on v1beta1 informers
+5. Kyverno's webhook was still registered with `failurePolicy: Fail`
+6. **Every Deployment/StatefulSet/DaemonSet creation outside kube-system was rejected**
+7. Longhorn (longhorn-system) couldn't restart → no storage → ArgoCD couldn't mount PVCs → ArgoCD died
+8. Full cluster deadlock — even rebooting all nodes didn't fix it because webhook configs survive in etcd
+9. Only manual deletion of webhook configurations broke the deadlock
+
+### The fix
+
+Infrastructure namespaces are excluded from Kyverno's webhook `namespaceSelector` in `values.yaml`:
+
+```yaml
+config:
+  webhooks:
+    namespaceSelector:
+      matchExpressions:
+      - key: kubernetes.io/metadata.name
+        operator: NotIn
+        values:
+          - kube-system
+          - longhorn-system        # Wave 1 storage
+          - argocd                 # Wave 0 GitOps
+          - volsync-system         # Wave 1 backup controller
+          - snapshot-controller    # Wave 1 snapshots
+          - cert-manager           # Wave 4 but critical
+          - external-secrets       # Wave 0 secrets
+          - 1passwordconnect       # Wave 0 secrets
+```
+
+**Why this works:** The fail-closed PVC gate only needs to protect app namespaces (Waves 4-6). Infrastructure namespaces (Waves 0-2) must boot before Kyverno (Wave 3), so they should never be gated by Kyverno's webhook.
+
+### Emergency recovery
+
+If Kyverno causes a webhook deadlock again:
+
+```bash
+./scripts/emergency-webhook-cleanup.sh
+```
+
+This deletes all Kyverno webhook configurations. Kyverno recreates them once it's healthy. The script is safe to run — it only removes webhook registrations, not policies or generated resources.
+
+### Prevention rules
+
+- **Never remove infrastructure namespaces from the webhook exclusion list**
+- **Pin Renovate to minor/patch for critical charts** (kube-prometheus-stack, longhorn, kyverno, cilium) — major version bumps should be manually reviewed
+- **Monitor Kyverno admission controller restarts** — more than 2 restarts in 10 minutes indicates a potential deadlock forming
+- **If Kyverno shows `"failed to wait for cache sync"` in logs** — run the emergency cleanup script immediately, don't wait
+
 ## Debugging Backup/Restore
 
 ```bash