Skip to content

Commit 272f5bd

Browse files
committed
goldilocks dash vpa
1 parent f5cc6e2 commit 272f5bd

3 files changed

Lines changed: 44 additions & 1 deletion

File tree

docs/cnpg-disaster-recovery.md

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,15 @@ s3://postgres-backups/cnpg/immich/
5151

5252
**Each recovery bumps the version**: `-v2``-v3``-v4`, etc.
5353

54+
### Restore Source vs Backup Target (Critical)
55+
56+
During recovery, treat these as two different values:
57+
58+
- `externalClusters[].barmanObjectStore.serverName` = **restore source** (existing lineage, e.g. `immich-database-v2`)
59+
- `backup.barmanObjectStore.serverName` = **new backup target** (next lineage, e.g. `immich-database-v3`)
60+
61+
After recovery succeeds, keep backups on the new lineage (`v3`). Do **not** switch backup target back to `v2`.
62+
5463
## Recovery Procedure
5564

5665
### Prerequisites
@@ -130,7 +139,7 @@ In `cluster.yaml`:
130139
- Uncomment `initdb` bootstrap
131140
- Comment out `recovery` bootstrap + `externalClusters`
132141
- Keep the new `serverName` in the backup section (e.g. `immich-database-v3`)
133-
- Update the commented recovery `externalClusters.serverName` to match the new backup serverName
142+
- Update the commented recovery `externalClusters.serverName` for next time (e.g. `immich-database-v3`)
134143

135144
```bash
136145
git add infrastructure/database/cloudnative-pg/immich/cluster.yaml
@@ -140,6 +149,13 @@ git push
140149

141150
ArgoCD syncs. CNPG ignores `initdb` bootstrap on existing clusters — your data is safe.
142151

152+
### Quick Example Timeline (Immich)
153+
154+
- Before nuke: backups writing to `immich-database-v2`
155+
- Recovery manifest: restore from `v2`, write new backups to `v3`
156+
- After recovery: normal manifest with `initdb` active, backup still on `v3`
157+
- Next DR event: restore from `v3`, then bump backup target to `v4`
158+
143159
## Troubleshooting
144160

145161
### "Expected empty archive"

docs/vpa-resource-optimization.md

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -438,6 +438,25 @@ kubectl logs -n goldilocks -l app.kubernetes.io/name=goldilocks,app.kubernetes.i
438438
kubectl logs -n vertical-pod-autoscaler -l app.kubernetes.io/component=recommender
439439
```
440440

441+
## Grafana Dashboards
442+
443+
Two community dashboards are auto-provisioned in Grafana under the **Infrastructure** folder:
444+
445+
| Dashboard | Grafana.com ID | What It Shows |
446+
|-----------|---------------|---------------|
447+
| **VPA Recommendations** | [14588](https://grafana.com/grafana/dashboards/14588) | Table of target/lower/upper bounds per container, namespace summary |
448+
| **K8s Autoscaling VPA** | [22168](https://grafana.com/grafana/dashboards/22168) | Cluster overview with drill-down to pod-level VPA details |
449+
450+
**URL**: https://grafana.vanillax.me → search for "VPA"
451+
452+
These dashboards read VPA metrics exposed by kube-state-metrics (`kube_verticalpodautoscaler_*`). Combined with Goldilocks and `vpa-report.sh`, you have three ways to view VPA data:
453+
454+
1. **Goldilocks dashboard** — per-namespace cards with copy-paste YAML
455+
2. **Grafana VPA dashboards** — time-series graphs and historical trends
456+
3. **CLI**`./scripts/vpa-report.sh` for quick terminal output
457+
458+
---
459+
441460
## Related Docs
442461

443462
- [Monitoring README](../monitoring/README.md) — metrics-server vs Prometheus pipelines

monitoring/prometheus-stack/values.yaml

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -209,6 +209,14 @@ grafana:
209209
gnetId: 20417
210210
revision: 3
211211
datasource: Prometheus
212+
vpa-recommendations:
213+
gnetId: 14588
214+
revision: 5
215+
datasource: Prometheus
216+
vpa-autoscaling:
217+
gnetId: 22168
218+
revision: 1
219+
datasource: Prometheus
212220
# Additional data sources configuration
213221
additionalDataSources:
214222
- name: Loki

0 commit comments

Comments
 (0)