up

Author: mitchross

.claude/settings.local.json

@@ -17,7 +17,8 @@
       "Bash(helm:*)",
       "Bash(gh:*)",
       "WebFetch(domain:*)",
-      "WebSearch"
+      "WebSearch",
+      "WebFetch(domain:github.com)"
     ],
     "deny": [
       "Bash(git push:*)",
@@ -35,4 +36,4 @@
       "Bash(git cherry-pick:*)"
     ]
   }
-}
+}

CLAUDE.md

@@ -628,10 +628,12 @@ kubectl apply -f pvc.yaml
 - Keep PVC names consistent for restore to work
 - Test restores periodically
 
+**Removing backups**: Just remove the `backup` label from the PVC. The `volsync-orphan-cleanup` ClusterCleanupPolicy runs every 15 minutes and automatically deletes orphaned ReplicationSource, ReplicationDestination, and ExternalSecret resources when the PVC no longer has a backup label.
+
 **DON'T**:
 - Add backup labels to system namespace PVCs (auto-excluded)
 - Change PVC name if you want automatic restore
-- Delete ReplicationSource/ReplicationDestination manually (Kyverno will recreate)
+- Delete ReplicationSource/ReplicationDestination manually (Kyverno will recreate them if label still present)
 - Use backup labels on non-Longhorn PVCs (snapshot support required)
 
 ## Debugging & Troubleshooting
@@ -806,6 +808,7 @@ kubectl exec -it gpu-pod -n app-name -- nvidia-smi
 | **Complex app with storage** | `my-apps/media/immich/` |
 | **PVC with automatic backup** | `my-apps/ai/khoj/pvc.yaml` (see backup label) |
 | **Kyverno backup policies** | `infrastructure/controllers/kyverno/policies/volsync-pvc-backup-restore.yaml` |
+| **Kyverno orphan cleanup** | `infrastructure/controllers/kyverno/policies/volsync-orphan-cleanup.yaml` |
 | **PVC Plumber (restore checker)** | `infrastructure/controllers/pvc-plumber/` |
 | **Full backup/restore flow diagram** | `docs/pvc-plumber-full-flow.md` |
 | **VolSync configuration** | `infrastructure/storage/volsync/` |

docs/backup-restore.md

@@ -95,6 +95,13 @@ PVC Created ──▶ Kyverno Rule 0 ──▶ Calls pvc-plumber ──▶ Backu
 - **Rule 3 (generate):** Creates ReplicationSource (backup schedule) - only after PVC is Bound
 - **Rule 4 (generate):** Creates ReplicationDestination (restore capability)
 
+### 4a. Kyverno ClusterCleanupPolicy (Orphan Cleanup)
+- **Runs every 15 minutes** to clean up orphaned backup resources
+- Targets ReplicationSource, ReplicationDestination, and ExternalSecret with labels `app.kubernetes.io/managed-by=kyverno` and `volsync.backup/pvc`
+- Checks if the corresponding PVC still has `backup: hourly` or `backup: daily` label
+- If label was removed or PVC no longer exists, **deletes the orphaned resources**
+- Prevents stale backup/restore jobs from running after backups are disabled
+
 ### 5. VolSync
 - Performs actual backup/restore operations using **Kopia**
 - Uses Longhorn snapshots for consistent backups
@@ -105,6 +112,20 @@ PVC Created ──▶ Kyverno Rule 0 ──▶ Calls pvc-plumber ──▶ Backu
 - Accessible at `kopia-ui.{domain}`
 - Mounts same NFS share as VolSync
 
+## How to Disable Backup for a PVC
+
+Remove the `backup` label from the PVC:
+
+```yaml
+metadata:
+  labels:
+    # backup: "hourly"   # Comment out or remove to disable backup
+```
+
+The `volsync-orphan-cleanup` ClusterCleanupPolicy runs every 15 minutes and automatically deletes the orphaned ReplicationSource, ReplicationDestination, and ExternalSecret. No manual cleanup needed.
+
+**Note:** Removing the label does NOT delete existing backups from NFS. The Kopia repository on TrueNAS retains all previous snapshots. To re-enable backups later, simply re-add the label.
+
 ## How to Enable Backup for a PVC
 
 Add a backup label to your PVC:
@@ -252,5 +273,6 @@ The following namespaces are excluded from automatic backup:
 | `infrastructure/storage/volsync/` | VolSync Helm chart |
 | `infrastructure/controllers/kyverno/policies/volsync-nfs-inject.yaml` | Injects NFS mount into mover pods |
 | `infrastructure/storage/kopia-ui/` | Kopia web UI for browsing backups |
-| `infrastructure/controllers/kyverno/policies/volsync-pvc-backup-restore.yaml` | Kyverno policy |
+| `infrastructure/controllers/kyverno/policies/volsync-pvc-backup-restore.yaml` | Kyverno backup/restore policy |
+| `infrastructure/controllers/kyverno/policies/volsync-orphan-cleanup.yaml` | Cleanup orphaned backup resources |
 | `monitoring/prometheus-stack/volsync-alerts.yaml` | Prometheus alerting rules |

monitoring/prometheus-stack/values.yaml

@@ -205,6 +205,10 @@ grafana:
         gnetId: 13032
         revision: 6
         datasource: Prometheus
+      cloudnative-pg:
+        gnetId: 20417
+        revision: 3
+        datasource: Prometheus
   # Additional data sources configuration
   additionalDataSources:
     - name: Loki