cleanup

Author: mitchross

README.md

@@ -72,6 +72,18 @@ graph TD;
 - **GPU Integration**: Full NVIDIA GPU support via Talos system extensions and GPU Operator
 - **Zero SSH**: All node management via Talosctl API
 
+### 🌊 Sync Wave Architecture
+
+The cluster uses **ArgoCD Sync Waves** to strictly order deployments, preventing "chicken-and-egg" dependency issues:
+
+1.  **Wave 0 (Foundation)**: Networking (Cilium) & Secrets (1Password/External Secrets)
+2.  **Wave 1 (Storage)**: Persistent Storage (Longhorn) & Object Storage (Garage)
+3.  **Wave 2 (System)**: Core Infrastructure (Cert-Manager, Databases, GPU)
+4.  **Wave 3 (Monitoring)**: Observability Stack (Prometheus, Grafana)
+5.  **Wave 4 (Apps)**: User Workloads
+
+*See [docs/argocd.md](docs/argocd.md) for the deep dive on health checks and dependency management.*
+
 ## 🚀 Quick Start (Manual Talos Method)
 
 > **Note:** If you're using Omni + Sidero Proxmox Provider, see **[BOOTSTRAP.md](BOOTSTRAP.md)** instead.

docs/argocd.md

@@ -0,0 +1,95 @@
+# ArgoCD & GitOps Architecture
+
+This document details the "App of Apps" GitOps architecture used in this cluster, specifically focusing on the **Sync Wave** strategy and **Health Check Customizations** that enable a fully self-managing cluster.
+
+## 🏗️ The "App of Apps" Pattern
+
+We use a hierarchical "App of Apps" pattern to manage the entire cluster state.
+
+```mermaid
+graph TD;
+    RootApp[Root Application] -->|Manages| AppSets[ApplicationSets];
+    AppSets -->|Generates| Apps[Applications];
+    Apps -->|Deploys| Resources[Kubernetes Resources];
+```
+
+### The Root Application
+The entry point is `infrastructure/controllers/argocd/root.yaml`. This application:
+1.  Points to `infrastructure/controllers/argocd/apps/`
+2.  Deploys the `ApplicationSet` definitions found there.
+3.  Is the *only* thing applied manually (during bootstrap).
+
+### ApplicationSets
+We use three primary ApplicationSets to categorize workloads:
+1.  **Infrastructure** (`infrastructure-appset.yaml`): Core system components (Cilium, Longhorn, Cert-Manager).
+2.  **Monitoring** (`monitoring-appset.yaml`): Observability stack (Prometheus, Grafana).
+3.  **My Apps** (`my-apps-appset.yaml`): User workloads.
+
+## 🌊 Sync Waves & Dependency Management
+
+To solve the "chicken-and-egg" problem of bootstrapping a cluster (e.g., needing storage for apps, but networking for storage), we use **ArgoCD Sync Waves**.
+
+### The Wave Strategy
+
+| Wave | Phase | Components | Description |
+|------|-------|------------|-------------|
+| **0** | **Foundation** | `cilium`, `1password-connect`, `external-secrets` | **Networking & Secrets**. The absolute minimum required for other pods to start and pull credentials. |
+| **1** | **Storage** | `longhorn`, `garage` | **Persistence**. Depends on Wave 0 for Pod-to-Pod communication and S3 backup credentials. |
+| **2** | **System** | `cert-manager`, `gpu-operator`, `databases` | **Core Services**. Depends on Storage (PVCs) and Networking (Ingress/Gateway). |
+| **3** | **Observability** | `kube-prometheus-stack`, `loki` | **Monitoring**. Monitors the healthy stack. |
+| **4** | **User** | `my-apps/*` | **Workloads**. The actual applications running on the cluster. |
+
+### How It Works
+Each `Application` resource in `infrastructure/controllers/argocd/apps/` is annotated with a sync wave:
+
+```yaml
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: cilium
+  annotations:
+    argocd.argoproj.io/sync-wave: "0"
+```
+
+ArgoCD processes these waves sequentially. **Wave 1 will NOT start until Wave 0 is healthy.**
+
+## 🏥 Health Check Customizations
+
+Standard ArgoCD behavior is to mark a parent Application as "Healthy" as soon as the child Application resource is created, *even if the child app is still syncing or degraded*. This breaks the Sync Wave logic for App-of-Apps.
+
+To fix this, we inject a custom Lua health check in `infrastructure/controllers/argocd/values.yaml`.
+
+### The "Wait for Child" Script
+
+```lua
+resource.customizations.health.argoproj.io_Application: |
+  hs = {}
+  hs.status = "Progressing"
+  hs.message = ""
+  if obj.status ~= nil then
+    if obj.status.health ~= nil then
+      hs.status = obj.status.health.status
+      if obj.status.health.message ~= nil then
+        hs.message = obj.status.health.message
+      end
+    end
+  end
+  return hs
+```
+
+**What this does:**
+1.  It overrides the health assessment of `Application` resources.
+2.  It forces the parent (Root App) to report the *actual status* of the child Application.
+3.  If `cilium` (Wave 0) is "Progressing", the Root App sees it as "Progressing".
+4.  The Root App **pauses** processing Wave 1 until all Wave 0 apps report "Healthy".
+
+## 🔄 Self-Management Loop
+
+1.  **Bootstrap**: You apply `root.yaml`.
+2.  **Adoption**: ArgoCD sees `cilium` defined in Git (Wave 0). It adopts the running Cilium instance.
+3.  **Expansion**: ArgoCD deploys `external-secrets` (Wave 0).
+4.  **Wait**: ArgoCD waits for Cilium and External Secrets to be green.
+5.  **Storage**: ArgoCD deploys `longhorn` (Wave 1).
+6.  **Completion**: The process continues until all waves are healthy.
+
+This ensures a deterministic, reliable boot sequence every time.

infrastructure/controllers/argocd/apps/infrastructure-appset.yaml

@@ -11,27 +11,21 @@ spec:
       repoURL: https://github.com/mitchross/talos-argocd-proxmox.git
       revision: main
       directories:
-      # Controllers (argocd managed via Helm, cilium/external-secrets/1password have their own apps)
-      - path: infrastructure/controllers/cert-manager
-      - path: infrastructure/controllers/gpu-priority-classes
-      - path: infrastructure/controllers/intel-device-plugins
-      - path: infrastructure/controllers/node-feature-discovery
-      - path: infrastructure/controllers/nvidia-device-plugin
-      - path: infrastructure/controllers/nvidia-gpu-operator
-      # Databases
+      - path: infrastructure/controllers/*
+      - path: infrastructure/networking/*
+      - path: infrastructure/storage/*
       - path: infrastructure/database/*/*
-      # Networking (cilium has its own app)
-      - path: infrastructure/networking/cloudflared
-      - path: infrastructure/networking/coredns
-      - path: infrastructure/networking/gateway
-      # Storage (longhorn has its own app)
-      - path: infrastructure/storage/container-registry
-      - path: infrastructure/storage/csi-driver-nfs
-      - path: infrastructure/storage/csi-driver-smb
-      - path: infrastructure/storage/local-storage
-      - path: infrastructure/storage/openebs
-      # CRDs
       - path: infrastructure/crds
+
+      # EXCLUDES: These are managed by standalone Apps for Sync Wave control (Wave 0/1)
+      # or are the ArgoCD controller itself.
+      exclude:
+      - path: infrastructure/controllers/argocd
+      - path: infrastructure/controllers/1passwordconnect
+      - path: infrastructure/controllers/external-secrets
+      - path: infrastructure/networking/cilium
+      - path: infrastructure/storage/longhorn
+      - path: infrastructure/storage/garage
   template:
     metadata:
       name: '{{path.basename}}'

scripts/bootstrap-argocd.sh

@@ -52,8 +52,8 @@ echo ""
 echo "✅ ArgoCD bootstrap complete!"
 echo ""
 echo "📊 ArgoCD will now sync applications in this order:"
-echo "   Wave 0: Cilium (networking)"
-echo "   Wave 1: Longhorn (storage)"
+echo "   Wave 0: Cilium (networking) & Secrets"
+echo "   Wave 1: Longhorn (storage) & Garage (S3)"
 echo "   Wave 2: Infrastructure (core services)"
 echo "   Wave 3: Monitoring (observability)"
 echo "   Wave 4: My-Apps (workloads)"