Add Kyverno ArgoCD app and Cilium preflight

Make Kyverno a standalone ArgoCD Application (sync-wave 3) so its webhooks register before any app PVCs are created; add infrastructure/controllers/argocd/apps/kyverno-app.yaml, remove Kyverno from the Infrastructure AppSet, and register kyverno-app.yaml in the kustomization manifest. Add preflight checks to scripts/bootstrap-argocd.sh to verify the cilium CLI and expected Cilium version (1.19.0), warn/prompt on mismatches, and document repair steps for Hubble Relay cert issues. Update documentation: README.md clarifies that the cilium install CLI version must match the Helm chart and includes Hubble cert cleanup steps; CLAUDE.md updates sync-wave ordering and rationale (PVC Plumber → Kyverno → Infrastructure AppSet). Also expand .claude local settings to allow additional kubectl bash commands used during bootstrap.

Author: mitchross

.claude/settings.local.json

@@ -13,7 +13,9 @@
       "Bash(kubectl api-resources:*)",
       "Bash(python3:*)",
       "Bash(helm repo list:*)",
-      "WebFetch(domain:raw.githubusercontent.com)"
+      "WebFetch(domain:raw.githubusercontent.com)",
+      "Bash(kubectl run:*)",
+      "Bash(kubectl rollout:*)"
     ],
     "deny": [],
     "ask": []

CLAUDE.md

@@ -117,15 +117,17 @@ Applications deploy in strict order to prevent race conditions:
 | **0** | Foundation | Cilium (CNI), ArgoCD, 1Password Connect, External Secrets, AppProjects |
 | **1** | Storage | Longhorn, VolumeSnapshot Controller, VolSync |
 | **2** | PVC Plumber | Backup existence checker (FAIL-CLOSED gate: PVC creation denied if Plumber is down) |
-| **4** | Infrastructure AppSet | Deploys from explicit path list: cert-manager, external-dns, GPU operators, Kyverno, gateway, databases, etc. |
+| **3** | Kyverno | Policy engine (standalone App, must register webhooks before app PVCs are created) |
+| **4** | Infrastructure AppSet | Deploys from explicit path list: cert-manager, external-dns, GPU operators, gateway, databases, etc. |
 | **5** | Monitoring AppSet | Discovers `monitoring/*` applications |
 | **6** | My-Apps AppSet | Discovers `my-apps/*/*` applications |
 
 **Why this matters**:
 - Longhorn won't deploy until Cilium + External Secrets are healthy
-- PVC Plumber (Wave 2) must run before Infrastructure AppSet (Wave 4) because Kyverno policies call PVC Plumber API
+- PVC Plumber (Wave 2) must run before Kyverno (Wave 3) because Kyverno policies call PVC Plumber API
+- Kyverno (Wave 3) is a **standalone Application** (not in the Infrastructure AppSet) to guarantee its webhooks are registered before any app PVCs are created. ApplicationSets are considered "healthy" immediately upon creation, so putting Kyverno in an AppSet would race with app deployment.
 - **FAIL-CLOSED**: If PVC Plumber is down, Kyverno denies creation of backup-labeled PVCs. Apps retry via ArgoCD backoff until Plumber is healthy. This prevents data loss during disaster recovery.
-- Kyverno, cert-manager, GPU operators etc. deploy via Infrastructure AppSet (Wave 4) before user apps (Wave 6)
+- cert-manager, GPU operators etc. deploy via Infrastructure AppSet (Wave 4) before user apps (Wave 6)
 - This prevents "chicken-and-egg" dependency issues and SSD thrashing
 
 **Important**: The Infrastructure AppSet uses an explicit list of paths (not glob discovery). To add a new infrastructure component, you must add its path to `infrastructure/controllers/argocd/apps/infrastructure-appset.yaml`.

README.md

@@ -80,6 +80,7 @@ Omni provisions Talos clusters without a CNI. Install Cilium to get networking f
 
 ```bash
 cilium install \
+    --version 1.19.0 \
     --set cluster.name=talos-prod-cluster \
     --set ipam.mode=kubernetes \
     --set kubeProxyReplacement=true \
@@ -94,9 +95,16 @@ cilium install \
     --set gatewayAPI.enableAppProtocol=true
 ```
 
-> **Important:** `cluster.name` must match `infrastructure/networking/cilium/values.yaml` for Hubble certificate SANs. After ArgoCD deploys, it takes over Cilium management at Wave 0.
+> **Important — version must match:** The `cilium install` CLI version must match the Helm chart version in `infrastructure/networking/cilium/kustomization.yaml` (currently **1.19.0**). Use `cilium install --version 1.19.0` to pin it. If versions differ, ArgoCD upgrades Cilium at Wave 0 and regenerates some Hubble certs but not others, causing TLS handshake failures (`x509: certificate signed by unknown authority`) that block all sync waves.
 >
-> If `cilium install` is run without `--set cluster.name=talos-prod-cluster`, certificates are generated for `default` or `kind-kind`. When ArgoCD later configures Cilium to expect `talos-prod-cluster`, the certificates will not match, causing TLS handshake failures in Hubble Relay (`x509: certificate signed by unknown authority`).
+> **Important — cluster name must match:** `cluster.name` must match `infrastructure/networking/cilium/values.yaml` for Hubble certificate SANs. If `cilium install` is run without `--set cluster.name=talos-prod-cluster`, certificates are generated for `default` or `kind-kind`, causing the same TLS failures.
+>
+> **If Hubble Relay is crash-looping after bootstrap**, delete stale certs and restart:
+> ```bash
+> kubectl delete secret hubble-relay-client-certs hubble-server-certs -n kube-system
+> kubectl rollout restart deployment hubble-relay -n kube-system
+> kubectl rollout restart ds cilium -n kube-system
+> ```
 
 ### Step 2: Install Gateway API CRDs
 

infrastructure/controllers/argocd/apps/infrastructure-appset.yaml

@@ -20,7 +20,6 @@ spec:
       - path: infrastructure/controllers/nvidia-gpu-operator
       - path: infrastructure/controllers/postgres-operator
       - path: infrastructure/controllers/reloader
-      - path: infrastructure/controllers/kyverno
       - path: infrastructure/networking/cloudflared
       - path: infrastructure/networking/coredns
       - path: infrastructure/networking/gateway

infrastructure/controllers/argocd/apps/kustomization.yaml

@@ -11,6 +11,7 @@ resources:
   - snapshot-controller-app.yaml # Wave 1 - VolumeSnapshot controller + CRDs
   - volsync-app.yaml       # Wave 1 - PVC backup and replication (Kopia + NFS)
   - pvc-plumber-app.yaml   # Wave 2 - Backup existence checker for restore
+  - kyverno-app.yaml       # Wave 3 - Policy engine (must be healthy before apps create PVCs)
   # ApplicationSets for automatic discovery
   - infrastructure-appset.yaml  # Wave 4
   - monitoring-appset.yaml      # Wave 5

infrastructure/controllers/argocd/apps/kyverno-app.yaml

@@ -0,0 +1,26 @@
+---
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: kyverno
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "3"
+  finalizers:
+    - resources-finalizer.argocd.argoproj.io
+spec:
+  project: infrastructure
+  source:
+    repoURL: https://github.com/mitchross/talos-argocd-proxmox.git
+    targetRevision: main
+    path: infrastructure/controllers/kyverno
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: kyverno
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=true
+      - ServerSideApply=true

scripts/bootstrap-argocd.sh

@@ -4,12 +4,75 @@ set -euo pipefail
 # Bootstrap ArgoCD Script
 # This script works around kustomize --enable-helm compatibility issues
 # by using Helm directly, then letting ArgoCD self-manage
+#
+# Prerequisites:
+#   1. Cilium must be installed FIRST (provides CNI networking)
+#   2. Gateway API CRDs must be applied
+#   3. 1Password secrets must be pre-seeded
+#
+# See README.md for the full bootstrap sequence.
 
 SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
 ROOT_DIR="$( cd "$SCRIPT_DIR/.." && pwd )"
 
+# Expected Cilium version — must match infrastructure/networking/cilium/kustomization.yaml
+EXPECTED_CILIUM_VERSION="1.19.0"
+
 echo "🚀 Bootstrapping ArgoCD with sync waves..."
 
+# Pre-flight: Verify Cilium is installed and healthy at the correct version
+echo ""
+echo "🔍 Pre-flight: Checking Cilium..."
+
+if ! command -v cilium &> /dev/null; then
+  echo "❌ cilium CLI not found. Install it first: https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/"
+  exit 1
+fi
+
+if ! cilium status --wait --wait-duration 30s &> /dev/null; then
+  echo "❌ Cilium is not healthy. Install Cilium first:"
+  echo ""
+  echo "   cilium install \\"
+  echo "       --version $EXPECTED_CILIUM_VERSION \\"
+  echo "       --set cluster.name=talos-prod-cluster \\"
+  echo "       --set ipam.mode=kubernetes \\"
+  echo "       --set kubeProxyReplacement=true \\"
+  echo "       --set k8sServiceHost=localhost \\"
+  echo "       --set k8sServicePort=7445 \\"
+  echo "       --set gatewayAPI.enabled=true"
+  echo ""
+  exit 1
+fi
+
+RUNNING_VERSION=$(kubectl get ds cilium -n kube-system -o jsonpath='{.spec.template.spec.containers[0].image}' 2>/dev/null | sed -E 's/.*:v([0-9]+\.[0-9]+\.[0-9]+).*/\1/' || true)
+
+if [ -n "$RUNNING_VERSION" ] && [ "$RUNNING_VERSION" != "$EXPECTED_CILIUM_VERSION" ]; then
+  echo "⚠️  WARNING: Cilium version mismatch!"
+  echo "   Running:  $RUNNING_VERSION"
+  echo "   Expected: $EXPECTED_CILIUM_VERSION (from Helm chart)"
+  echo ""
+  echo "   ArgoCD Wave 0 will upgrade Cilium $RUNNING_VERSION → $EXPECTED_CILIUM_VERSION"
+  echo "   This in-place upgrade can corrupt BPF state and break new pod networking."
+  echo ""
+  echo "   Recommended: Reinstall Cilium at the correct version first:"
+  echo "     cilium uninstall"
+  echo "     cilium install --version $EXPECTED_CILIUM_VERSION \\"
+  echo "         --set cluster.name=talos-prod-cluster \\"
+  echo "         --set ipam.mode=kubernetes \\"
+  echo "         --set kubeProxyReplacement=true \\"
+  echo "         --set k8sServiceHost=localhost \\"
+  echo "         --set k8sServicePort=7445 \\"
+  echo "         --set gatewayAPI.enabled=true"
+  echo ""
+  read -p "   Continue anyway? (y/N) " -n 1 -r
+  echo ""
+  if [[ ! $REPLY =~ ^[Yy]$ ]]; then
+    exit 1
+  fi
+else
+  echo "✅ Cilium $RUNNING_VERSION is healthy and matches Helm chart ($EXPECTED_CILIUM_VERSION)"
+fi
+
 # Step 1: Create namespace
 echo ""
 echo "📦 Creating argocd namespace..."
@@ -52,11 +115,13 @@ echo ""
 echo "✅ ArgoCD bootstrap complete!"
 echo ""
 echo "📊 ArgoCD will now sync applications in this order:"
-echo "   Wave 0: Cilium (networking) & Secrets"
-echo "   Wave 1: Longhorn (storage), Snapshot Controller & VolSync"
-echo "   Wave 2: Infrastructure (core services)"
-echo "   Wave 3: Monitoring (observability)"
-echo "   Wave 4: My-Apps (workloads)"
+echo "   Wave 0: Cilium (networking), 1Password Connect, External Secrets"
+echo "   Wave 1: Longhorn (storage), Snapshot Controller, VolSync"
+echo "   Wave 2: PVC Plumber (backup checker, FAIL-CLOSED gate)"
+echo "   Wave 3: Kyverno (policy engine, must register webhooks before app PVCs)"
+echo "   Wave 4: Infrastructure AppSet (cert-manager, GPU operators, gateway, etc.)"
+echo "   Wave 5: Monitoring AppSet (Prometheus, Grafana, Loki)"
+echo "   Wave 6: My-Apps AppSet (user workloads)"
 echo ""
 echo "🔍 Monitor progress with:"
 echo "   kubectl get applications -n argocd -w"