docs: update storage architecture for tiered backup and volume populator

- Update backup schedules: critical (hourly) vs non-critical (daily)
- Document Volume Populator pattern for automatic restore
- Update secrets doc for RustFS (replaced MinIO references)
- Add secret structure example for ExternalSecrets

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Author: mitchross

docs/secrets/volsync-secrets.md

@@ -2,19 +2,20 @@
 
 ## Required 1Password Items
 
-Before VolSync backups will work, you need to create one new item in 1Password.
+Before VolSync backups will work, you need the `rustfs` item in 1Password.
 
-### volsync-kopia
+### rustfs
 
-Create a new **Password** item in your 1Password vault:
+Create a **Password** item in your 1Password vault:
 
 | Field | Value |
 |-------|-------|
-| **Item name** | `volsync-kopia` |
-| **Field name** | `password` |
-| **Value** | A strong random password (32+ characters) |
+| **Item name** | `rustfs` |
+| **access_key** | RustFS access key |
+| **secret_key** | RustFS secret key |
+| **restic_password** | A strong random password (32+ characters) |
 
-This password encrypts all Kopia/Restic backup repositories stored in S3.
+The `restic_password` encrypts all backup repositories stored in S3.
 
 **Generate a secure password:**
 ```bash
@@ -23,41 +24,46 @@ openssl rand -base64 32
 
 Example output: `K7x9mP2nL4qR8vT1wY5zA3cF6hJ0bN+dG=`
 
-### Existing Items Used
-
-The VolSync configuration also uses your existing `minio` item:
-
-| Item | Fields Used |
-|------|-------------|
-| `minio` | `minio_access_key`, `minio_secret_key` |
-
-These should already exist from your Longhorn backup configuration.
-
 ## Verification
 
-After creating the `volsync-kopia` item, verify the ExternalSecrets are syncing:
+After creating the `rustfs` item, verify the ExternalSecrets are syncing:
 
 ```bash
-# Check VolSync system secret
-kubectl get externalsecret -n volsync-system
-
 # Check app-level secrets (example)
 kubectl get externalsecret -n home-assistant
+
+# View secret details
+kubectl get externalsecret home-assistant-volsync-secret -n home-assistant -o yaml
 ```
 
 All ExternalSecrets should show `SecretSynced` status.
 
 ## S3 Bucket Setup
 
-Ensure these buckets exist in your RustFS/MinIO (192.168.10.133):
+Ensure the `volsync` bucket exists in RustFS (192.168.10.133:30292):
 
 | Bucket | Purpose |
 |--------|---------|
-| `volsync-backups` | VolSync PVC backups (Kopia repositories) |
-| `postgres-backups` | CNPG and Crunchy database backups |
+| `volsync` | VolSync PVC backups (Restic repositories) |
 
-Create them if they don't exist:
+Create it if it doesn't exist:
 ```bash
-mc mb truenas/volsync-backups
-mc mb truenas/postgres-backups
+mc alias set rustfs http://192.168.10.133:30292 <access_key> <secret_key>
+mc mb rustfs/volsync
 ```
+
+## Secret Structure
+
+Each app has an ExternalSecret that creates a Secret with this structure:
+
+```yaml
+apiVersion: v1
+kind: Secret
+metadata:
+  name: <app>-volsync-secret
+type: Opaque
+stringData:
+  RESTIC_REPOSITORY: s3:http://192.168.10.133:30292/volsync/<app>
+  RESTIC_PASSWORD: <from 1Password rustfs.restic_password>
+  AWS_ACCESS_KEY_ID: <from 1Password rustfs.access_key>
+  AWS_SECRET_ACCESS_KEY: <from 1Password rustfs.secret_key>

docs/storage-architecture.md

@@ -74,20 +74,37 @@ graph LR
 
 ### PVC Backups (VolSync)
 
-All application PVCs are backed up daily at 2 AM using VolSync with Restic:
+Application PVCs are backed up using VolSync with Restic, with a tiered schedule:
+
+**Critical Apps (Hourly):**
+home-assistant, paperless-ngx, karakeep, meilisearch, n8n, immich, open-webui, khoj
+
+| Setting | Value |
+|---------|-------|
+| Schedule | `0 * * * *` (hourly) |
+| Retention | 24 hourly + 7 daily |
+
+**Non-Critical Apps (Daily):**
+container-registry, redis, mqtt, searxng, fizzy, nginx, jellyfin, nestmtx, homepage-dashboard, plex
 
 | Setting | Value |
 |---------|-------|
 | Schedule | `0 2 * * *` (daily at 2 AM) |
 | Retention | 14 days |
+
+**Common Settings:**
+
+| Setting | Value |
+|---------|-------|
 | Backend | Restic |
 | Target | RustFS S3 on TrueNAS (192.168.10.133:30292) |
 | Bucket | `volsync` |
 | Copy Method | Snapshot |
 
 Each app has:
 - `ReplicationSource` - Defines backup schedule and retention
-- `ReplicationDestination` - Pre-provisioned for restore capability
+- `ReplicationDestination` - Dormant restore definition (no trigger)
+- `PVC` - References ReplicationDestination via `dataSourceRef`
 - `ExternalSecret` - Pulls S3 credentials from 1Password
 
 ### Database Backups (Native)
@@ -107,31 +124,44 @@ PostgreSQL databases use their native backup tools:
 
 ## 3. Disaster Recovery
 
-### Restoring a PVC (VolSync)
-
-When you need to restore a PVC from backup:
+### Volume Populator Pattern (Automatic Restore)
 
-1. **Trigger the ReplicationDestination**:
-```bash
-kubectl patch replicationdestination <app>-restore -n <namespace> \
-  --type merge \
-  -p '{"spec":{"trigger":{"manual":"restore-'$(date +%s)'"}}}'
-```
+PVCs use the **Volume Populator** pattern for automatic restore:
 
-2. **Wait for restore to complete**:
-```bash
-kubectl get replicationdestination <app>-restore -n <namespace> -w
-```
-
-3. **Update PVC to use restored data** (if needed):
 ```yaml
+# In each app's PVC
 spec:
   dataSourceRef:
     kind: ReplicationDestination
     apiGroup: volsync.backube
     name: <app>-restore
 ```
 
+**How it works:**
+1. PVC references a dormant ReplicationDestination (no trigger)
+2. When PVC is newly created (no existing Longhorn volume), Kubernetes uses the dataSourceRef
+3. VolSync automatically restores from S3 backup to populate the new PVC
+4. If Longhorn already has the data, dataSourceRef is ignored
+
+**This enables zero-intervention restore:**
+- Deploy app → PVC created → auto-restore from S3 → app starts with data
+- Longhorn replication is Layer 1 (node failure)
+- S3 backup is Layer 2 (cluster loss)
+
+### Manual Restore (if needed)
+
+For manual restore scenarios, you can trigger the ReplicationDestination:
+
+```bash
+# Add manual trigger to force restore
+kubectl patch replicationdestination <app>-restore -n <namespace> \
+  --type merge \
+  -p '{"spec":{"trigger":{"manual":"restore-'$(date +%s)'"}}}'
+
+# Wait for restore to complete
+kubectl get replicationdestination <app>-restore -n <namespace> -w
+```
+
 ### Restoring a Database
 
 **CloudNativePG:**
@@ -159,11 +189,11 @@ After a complete cluster rebuild:
 
 | Feature | Before (Longhorn) | Now (VolSync) |
 |---------|-------------------|---------------|
-| Backup tool | Longhorn built-in | VolSync + Kopia |
-| Backup schedule | RecurringJobs (tiered) | Single daily schedule |
-| Restore method | Hardcoded restore-job.yaml | Declarative ReplicationDestination |
+| Backup tool | Longhorn built-in | VolSync + Restic |
+| Backup schedule | RecurringJobs (tiered) | Tiered: hourly (critical) + daily (non-critical) |
+| Restore method | Hardcoded restore-job.yaml | Volume Populator (automatic on PVC create) |
 | Database backups | PVC snapshots (inconsistent) | Native WAL archiving (consistent) |
-| Complexity | Multiple tiers, shell scripts | Simple, uniform config |
+| Complexity | Multiple tiers, shell scripts | Declarative YAML per app |
 
 ## 5. Monitoring