fix overload

Author: mitchross

CLAUDE.md

@@ -112,8 +112,10 @@ docs/                   # Documentation
 - Manually create or delete ReplicationSource/ReplicationDestination (Kyverno manages these)
 - Use legacy `nfs:` block for NFS PVs (mountOptions silently ignored — use CSI)
 - Use `RollingUpdate` strategy on Deployments with RWO PVCs (causes Multi-Attach deadlock)
-- Use `background: true` on Kyverno generate policies — **causes API server overload from continuous background scanning of all matching resources; use `background: false` and rely on admission-time generation**
-- Use `mutateExistingOnPolicyUpdate: true` on Kyverno generate policies — **re-evaluates ALL matching resources cluster-wide on any policy change, creating UpdateRequest storms**
+- Use `background: true` on Kyverno generate policies — **causes API server overload from continuous background scanning; use `background: false`**
+- Use `mutateExistingOnPolicyUpdate: true` on Kyverno generate policies — **re-evaluates ALL matching resources cluster-wide on any policy change**
+- Use `synchronize: true` on Kyverno generate policies — **drift watchers create UpdateRequests on every controller status update, hammering the API server; use `synchronize: false`**
+- Omit Kyverno canonical defaults (`emitWarning`, `validationFailureAction`, `skipBackgroundRequests`) from policy YAML — **Kyverno webhook adds them, ArgoCD detects the diff, app shows OutOfSync**
 
 ## Nested CLAUDE.md Files
 

infrastructure/controllers/argocd/apps/kyverno-app.yaml

@@ -36,24 +36,16 @@ spec:
   ignoreDifferences:
   - group: kyverno.io
     kind: ClusterPolicy
-    jqPathExpressions:
-    - .metadata.generation
-    # Kyverno webhook adds these defaults on admission
-    - .spec.emitWarning
-    - .spec.validationFailureAction
-    - .spec.rules[].skipBackgroundRequests
+    jsonPointers:
+    - /metadata/generation
   - group: kyverno.io
     kind: ClusterCleanupPolicy
-    jqPathExpressions:
-    - .metadata.generation
-    - .spec.validationFailureAction
+    jsonPointers:
+    - /metadata/generation
   - group: kyverno.io
     kind: Policy
-    jqPathExpressions:
-    - .metadata.generation
-    - .spec.emitWarning
-    - .spec.validationFailureAction
-    - .spec.rules[].skipBackgroundRequests
+    jsonPointers:
+    - /metadata/generation
   # Kyverno injects caBundle into webhooks after creation
   - group: admissionregistration.k8s.io
     kind: MutatingWebhookConfiguration

infrastructure/controllers/kyverno/CLAUDE.md

@@ -207,18 +207,23 @@ spec:
 
 **Never use `mutateExistingOnPolicyUpdate: true` on generate policies**. This re-evaluates ALL matching resources cluster-wide whenever the policy YAML changes — even a comment edit triggers it. Combined with background scanning, this caused a 23-hour API server overload incident (2026-03-25).
 
-**The safe pattern for all Kyverno generate policies:**
+**The safe pattern for all Kyverno generate policies (canonical form):**
 ```yaml
 spec:
   mutateExistingOnPolicyUpdate: false  # REQUIRED — prevents cluster-wide re-evaluation on policy change
   background: false                     # REQUIRED — prevents continuous background scanning
+  emitWarning: false                    # Kyverno default — include to match canonical form for ArgoCD sync
+  validationFailureAction: Audit        # Kyverno default — include to match canonical form for ArgoCD sync
   rules:
     - name: my-generate-rule
+      skipBackgroundRequests: true       # Kyverno default — include to match canonical form for ArgoCD sync
       generate:
-        synchronize: true               # OK — sync enforcement happens via admission webhook, not background scan
+        synchronize: false              # REQUIRED — prevents drift watchers that generate UpdateRequests on every controller status update
 ```
 
-`synchronize: true` still works with `background: false` because sync is enforced through the admission controller. Generated resources are created/recreated when the trigger resource goes through admission (e.g., PVC creation via ArgoCD sync).
+**Why `synchronize: false`**: With `synchronize: true`, Kyverno watches every generated resource (ExternalSecrets, ReplicationSources, etc.) and creates UpdateRequests whenever their controllers update status. With ~114 watched resources, this generates hundreds of thousands of API calls. Resources are still created on admission (PVC creation via ArgoCD sync) — they just aren't re-synced on drift.
+
+**Why canonical form**: Kyverno's admission webhook adds `emitWarning`, `validationFailureAction`, and `skipBackgroundRequests` as defaults. If these aren't in git, ArgoCD detects the diff and shows OutOfSync. Writing the defaults explicitly keeps ArgoCD happy.
 
 **If you need to re-process existing resources after a policy change**, do a one-time ArgoCD sync or manually trigger resource re-admission — don't enable `mutateExistingOnPolicyUpdate`.
 

infrastructure/controllers/kyverno/policies/volsync-pvc-backup-restore.yaml

@@ -14,11 +14,14 @@ metadata:
 spec:
   mutateExistingOnPolicyUpdate: false
   background: false
+  emitWarning: false
+  validationFailureAction: Audit
   rules:
     # Rule 0: Gate PVC creation on PVC Plumber availability (FAIL-CLOSED)
     # If PVC Plumber is unreachable, DENY PVC creation to prevent data loss
     # during disaster recovery. Apps retry via ArgoCD backoff until Plumber is healthy.
     - name: require-pvc-plumber-available
+      skipBackgroundRequests: true
       match:
         any:
           - resources:
@@ -60,6 +63,7 @@ spec:
     # Rule 1: Conditionally add dataSourceRef if backup exists in Kopia
     # IMPORTANT: Only trigger on CREATE to avoid race conditions during PVC deletion
     - name: add-datasource-if-backup-exists
+      skipBackgroundRequests: true
       match:
         any:
           - resources:
@@ -103,6 +107,7 @@ spec:
 
     # Rule 2: Generate ExternalSecret for Kopia repository credentials
     - name: generate-kopia-secret
+      skipBackgroundRequests: true
       match:
         any:
           - resources:
@@ -121,7 +126,7 @@ spec:
                 - volsync-system
                 - kyverno
       generate:
-        synchronize: true
+        synchronize: false
         apiVersion: external-secrets.io/v1
         kind: ExternalSecret
         name: "volsync-{{request.object.metadata.name}}"
@@ -160,6 +165,7 @@ spec:
     # Rule 3: Generate ReplicationSource (backup schedule)
     # IMPORTANT: Only create backup AFTER PVC is Bound to avoid conflicts with restore
     - name: generate-replication-source
+      skipBackgroundRequests: true
       match:
         any:
           - resources:
@@ -190,7 +196,7 @@ spec:
             operator: GreaterThanOrEquals
             value: "2h"
       generate:
-        synchronize: true
+        synchronize: false
         apiVersion: volsync.backube/v1alpha1
         kind: ReplicationSource
         name: "{{request.object.metadata.name}}-backup"
@@ -225,6 +231,7 @@ spec:
 
     # Rule 4: Generate ReplicationDestination (restore capability)
     - name: generate-replication-destination
+      skipBackgroundRequests: true
       match:
         any:
           - resources:
@@ -243,7 +250,7 @@ spec:
                 - volsync-system
                 - kyverno
       generate:
-        synchronize: true
+        synchronize: false
         apiVersion: volsync.backube/v1alpha1
         kind: ReplicationDestination
         name: "{{request.object.metadata.name}}-backup"

infrastructure/controllers/kyverno/policies/vpa-auto-generate.yaml

@@ -15,11 +15,14 @@ metadata:
 spec:
   mutateExistingOnPolicyUpdate: false
   background: false
+  emitWarning: false
+  validationFailureAction: Audit
   rules:
     # Rule 1: Infrastructure and monitoring — recommend only
     # Resources in these namespaces are manually tuned via GitOps.
     # VPA provides recommendations but does not auto-apply them.
     - name: generate-vpa-recommend-only
+      skipBackgroundRequests: true
       match:
         any:
           - resources:
@@ -64,7 +67,7 @@ spec:
                 - k8sgpt
                 - pod-cleanup
       generate:
-        synchronize: true
+        synchronize: false
         apiVersion: autoscaling.k8s.io/v1
         kind: VerticalPodAutoscaler
         name: "{{request.object.metadata.name}}"
@@ -91,6 +94,7 @@ spec:
     # running pods. This prevents OOM kills on bursty workloads (e.g. Kafka,
     # TubeSync) where steady-state usage is low but peak usage needs headroom.
     - name: generate-vpa-auto-tune
+      skipBackgroundRequests: true
       match:
         any:
           - resources:
@@ -137,7 +141,7 @@ spec:
                 - k8sgpt
                 - pod-cleanup
       generate:
-        synchronize: true
+        synchronize: false
         apiVersion: autoscaling.k8s.io/v1
         kind: VerticalPodAutoscaler
         name: "{{request.object.metadata.name}}"

infrastructure/controllers/kyverno/policies/vpa-min-allowed.yaml

@@ -15,8 +15,11 @@ metadata:
       handles the rest. Example: vpa.kubernetes.io/min-memory: "2Gi"
 spec:
   background: false
+  emitWarning: false
+  validationFailureAction: Audit
   rules:
     - name: inject-min-allowed
+      skipBackgroundRequests: true
       match:
         any:
           - resources:

scripts/validate-kyverno-policies.sh

@@ -2,9 +2,10 @@
 # Validates Kyverno generate policies for dangerous settings that cause API server overload.
 # Used by: CI pipeline, pre-commit hook
 #
-# background: true on generate policies → continuous background scanning (~30s loop)
+# background: true → continuous background scanning (~30s loop)
 # mutateExistingOnPolicyUpdate: true → re-evaluates ALL matching resources on policy change
-# Both caused a 23-hour API server overload incident (2026-03-25).
+# synchronize: true → drift watchers create UpdateRequests on every controller status update
+# All three caused a 23-hour API server overload incident (2026-03-25).
 set -euo pipefail
 
 ERRORS=0
@@ -27,6 +28,14 @@ for file in $(grep -rl 'kind: ClusterPolicy\|kind: Policy' infrastructure/contro
     echo ""
     ERRORS=$((ERRORS + 1))
   fi
+
+  if grep -q 'synchronize: true' "$file"; then
+    echo "ERROR: ${file}"
+    echo "  synchronize: true creates drift watchers that generate UpdateRequests on every controller update."
+    echo "  Use: synchronize: false"
+    echo ""
+    ERRORS=$((ERRORS + 1))
+  fi
 done
 
 if [ "$ERRORS" -gt 0 ]; then