fix argo

Author: mitchross

.github/instructions/argocd.instructions.md

@@ -76,7 +76,8 @@ Standard sync options for all applications:
 - `CreateNamespace=true`: Auto-create target namespaces
 - `ServerSideApply=true`: Use server-side apply for better conflict resolution
 - `RespectIgnoreDifferences=true`: Honor ignoreDifferences configurations
-- `ApplyOutOfSyncOnly=true`: Only apply resources that are out of sync
+- `Replace=false`: Use patch instead of full replace
+- **DO NOT use `ApplyOutOfSyncOnly=true`** — causes silent sync failures with SSA (configmaps not applied)
 
 ### Retry Strategy
 ```yaml
@@ -211,8 +212,12 @@ kubectl describe application <app-name> -n argocd
 ### Sync Policies
 - ✅ Use automated sync with prune and selfHeal
 - ✅ Set appropriate sync waves for dependencies
-- ✅ Use server-side apply for complex resources
+- ✅ Use server-side apply AND server-side diff together (both enabled globally)
+- ✅ Keep `RespectIgnoreDifferences=true` for PVC/HTTPRoute/ExternalSecret diffs
+- ❌ Don't use `ApplyOutOfSyncOnly=true` — causes silent sync failures with SSA
+- ❌ Don't use `IgnoreMissingTemplate=true` — masks real template errors
 - ❌ Don't disable automated sync without good reason
+- ❌ Don't use client-side diff with server-side apply (diff/apply mismatch)
 
 ### Resource Management
 - ✅ Use ignoreDifferences for known noisy fields

.github/instructions/standards.instructions.md

@@ -33,7 +33,7 @@ Applications are categorized into `infrastructure`, `monitoring`, and `my-apps`,
   - `CreateNamespace=true`
   - `ServerSideApply=true`
   - `RespectIgnoreDifferences=true`
-  - `ApplyOutOfSyncOnly=true`
+  - `Replace=false`
 - Retry: exponential backoff, max 5 attempts, 3m cap
 
 ## ApplicationSet: `monitoring`
@@ -74,6 +74,18 @@ Applications are categorized into `infrastructure`, `monitoring`, and `my-apps`,
 - All apps must support `kustomize build` locally
 - Use `ignoreDifferences` for Helm-managed labels, CRDs, and known noisy fields
 
+# Server-Side Diff & Apply (CRITICAL)
+
+This cluster uses **Server-Side Diff** globally (`resource.server-side-diff: "true"` in `argocd-cm`) paired with **Server-Side Apply** (`ServerSideApply=true` in syncOptions). These MUST be aligned:
+
+- **Server-Side Apply** sends manifests to K8s API for patching with field ownership tracking
+- **Server-Side Diff** asks K8s API "what would change?" via dry-run SSA, then compares the result
+- Without Server-Side Diff, ArgoCD uses client-side diff which guesses — and can miss changes that SSA would detect (especially ConfigMap data), causing resources to silently not get applied
+
+**Forbidden sync options:**
+- `ApplyOutOfSyncOnly=true` — relies on diff accuracy; with SSA edge cases, resources can be skipped silently
+- `IgnoreMissingTemplate=true` — masks template rendering errors in ApplicationSets
+
 # Helm + Kustomize Integration
 
 Refer to `1password-connect` as a pattern:

docs/argocd.md

@@ -94,3 +94,76 @@ resource.customizations.health.argoproj.io_Application: |
 6.  **Completion**: The process continues until all waves are healthy.
 
 This ensures a deterministic, reliable boot sequence every time.
+
+## Server-Side Diff vs Client-Side Diff
+
+This cluster uses **Server-Side Diff** (`resource.server-side-diff: "true"` in `argocd-cm`) paired with **Server-Side Apply** (`ServerSideApply=true` in syncOptions). These must be aligned — using one without the other causes silent sync failures.
+
+### Client-Side Diff (legacy, DO NOT USE with SSA)
+
+ArgoCD downloads the live resource from the cluster, then compares it against the Git manifest **locally in the ArgoCD controller**. It's essentially doing `diff manifest.yaml live-resource.yaml` on its own.
+
+**Problem**: ArgoCD doesn't know what Kubernetes would actually do with the manifest. Kubernetes adds defaults, mutating webhooks modify fields, and SSA has field ownership rules. ArgoCD is guessing — and sometimes guesses wrong (thinks it's "in-sync" when it's not).
+
+### Server-Side Diff (modern, REQUIRED with SSA)
+
+ArgoCD sends the Git manifest to the Kubernetes API as a **dry-run server-side apply** and gets back what the result *would* look like. Then it compares *that* against the live resource.
+
+**Why it's better**: Kubernetes itself tells ArgoCD "here's what would change if you applied this" — accounting for defaults, field ownership, webhooks, everything. No guessing.
+
+### Why the Mismatch Breaks ConfigMaps
+
+Without Server-Side Diff, using Server-Side Apply + `ApplyOutOfSyncOnly`:
+
+```
+Git: configmap data = NEW content
+                ↓
+Client-side diff: "managed fields metadata looks the same..." → IN SYNC (wrong!)
+                ↓
+ApplyOutOfSyncOnly: "it's in-sync, skip it"
+                ↓
+Result: configmap never applied, ArgoCD says "Synced" ✓ (LIE)
+```
+
+With Server-Side Diff:
+
+```
+Git: configmap data = NEW content
+                ↓
+K8s API dry-run: "this would change .data.presets.ini" → OUT OF SYNC
+                ↓
+Sync: applies the configmap
+                ↓
+Result: configmap actually updated ✓
+```
+
+### Configuration
+
+Enabled globally in `infrastructure/controllers/argocd/values.yaml`:
+
+```yaml
+configs:
+  cm:
+    resource.server-side-diff: "true"
+```
+
+### Sync Options (CRITICAL — do not add ApplyOutOfSyncOnly)
+
+Standard sync options for all ApplicationSets:
+
+```yaml
+syncOptions:
+- CreateNamespace=true
+- ServerSideApply=true          # Server-side apply for better conflict resolution
+- RespectIgnoreDifferences=true # Honor ignoreDifferences for PVC, HTTPRoute, etc.
+- Replace=false                 # Use patch, not full replace
+```
+
+**DO NOT add these options:**
+- `ApplyOutOfSyncOnly=true` — Even with ServerSideDiff, has [known edge cases with key removal](https://github.com/argoproj/argo-cd/issues/24882). Not worth the risk for a homelab-scale cluster.
+- `IgnoreMissingTemplate=true` — Can mask real template errors in ApplicationSets.
+
+### References
+- [ArgoCD Diff Strategies](https://argo-cd.readthedocs.io/en/stable/user-guide/diff-strategies/)
+- [ArgoCD SSA ConfigMap sync failure (#22687)](https://github.com/argoproj/argo-cd/issues/22687)
+- [ArgoCD key removal not detected (#24882)](https://github.com/argoproj/argo-cd/issues/24882)

infrastructure/controllers/argocd/apps/infrastructure-appset.yaml

@@ -60,9 +60,7 @@ spec:
         - CreateNamespace=true
         - ServerSideApply=true
         - RespectIgnoreDifferences=true
-        - ApplyOutOfSyncOnly=true
         - Replace=false
-        - IgnoreMissingTemplate=true
         retry:
           limit: 5
           backoff:

infrastructure/controllers/argocd/apps/monitoring-appset.yaml

@@ -36,9 +36,7 @@ spec:
         - CreateNamespace=true
         - ServerSideApply=true
         - RespectIgnoreDifferences=true
-        - ApplyOutOfSyncOnly=true
         - Replace=false
-        - IgnoreMissingTemplate=true
         retry:
           limit: 5
           backoff:

infrastructure/controllers/argocd/apps/my-apps-appset.yaml

@@ -36,9 +36,7 @@ spec:
         - CreateNamespace=true
         - ServerSideApply=true
         - RespectIgnoreDifferences=true
-        - ApplyOutOfSyncOnly=true
         - Replace=false
-        - IgnoreMissingTemplate=true
         retry:
           limit: 5
           backoff:

infrastructure/controllers/argocd/root.yaml

@@ -22,7 +22,6 @@ spec:
     - CreateNamespace=true
     - ServerSideApply=true
     - RespectIgnoreDifferences=true
-    - ApplyOutOfSyncOnly=true
   ignoreDifferences:
   - group: argoproj.io
     kind: Application

infrastructure/controllers/argocd/values.yaml

@@ -18,6 +18,10 @@ configs:
   cm:
     url: "https://argocd.vanillax.me"
     kustomize.buildOptions: "--enable-helm"
+    # Use server-side diff to match ServerSideApply - prevents false "in-sync" detection
+    # Without this, client-side diff + server-side apply disagree on changes, causing
+    # configmaps and other resources to silently not get applied
+    resource.server-side-diff: "true"
     # Performance: add jitter to prevent all 60+ apps reconciling simultaneously
     timeout.reconciliation: "60s"
     timeout.reconciliation.jitter: "30s"