Roll Grafana on secret rotation, release-gate monitoring deploys - #151
Merged
giang-h-vu merged 4 commits intoJul 20, 2026
Merged
Conversation
Env vars from secretKeyRef resolve once at container start, so a rotated grafana-admin Secret never reached the running pod (live password drifted from the vault). Ansible now passes the Secret's resourceVersion into the chart, where it lands as a pod-template annotation — any rotation changes the template and triggers a rollout. resourceVersion instead of a password hash so nothing password-derived appears in the pod spec.
Monitoring previously deployed from main on merge while the app shipped at release, so dashboards/alerts could reference metrics prod did not emit yet. cd-prod now calls cd-monitoring as a reusable workflow after the app deploy; workflow_call runs at the caller's ref, so monitoring deploys from the same release tag as the app. workflow_dispatch remains as the manual escape hatch, sharing the deploy-prod concurrency group since both paths upgrade the same Helm release.
Root README gains a live-environments table (dev AKS, prod Rancher, Grafana; Prometheus is deliberately not exposed — NetworkPolicy admits only Grafana). Update monitoring README, CLAUDE.md, and the Grafana deployment header to match the new deploy model: monitoring ships with prod releases via cd-prod calling cd-monitoring, and a rotated admin password now rolls the pod.
giang-h-vu
added
infra
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (9)
📝 WalkthroughWalkthroughMonitoring deployment is now release-gated through ChangesMonitoring release and rotation
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant CDProd
participant CDMonitoring
participant MonitoringPlaybook
participant GrafanaSecret
participant HelmMonitoring
participant Grafana
CDProd->>CDMonitoring: invoke after application deploy
CDMonitoring->>MonitoringPlaybook: run monitoring deployment
MonitoringPlaybook->>GrafanaSecret: create or update admin Secret
GrafanaSecret-->>MonitoringPlaybook: return resourceVersion
MonitoringPlaybook->>HelmMonitoring: pass adminSecretVersion
HelmMonitoring->>Grafana: render rollout annotation
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
giang-h-vu
deleted the
PROJ-A/roll-grafana-on-secret-rotation_update-documentation
branch
July 20, 2026 06:48
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes Grafana password rotation actually take effect (pod rolls automatically when the
grafana-adminSecret changes), release-gates the monitoring deploy so it always ships from the same tag as the app, rotates the prod Grafana admin password, and updates docs incl. live-environment links in the root README.Motivation & Context
The live Grafana password had drifted from the vault: env vars from
secretKeyRefresolve once at container start, so rotating the Secret never reached the running pod. Separately, monitoring deployed frommainon merge while the app shipped at release, so dashboards/alerts could reference metrics prod didn't emit yet.Changes
grafana-adminSecret'sresourceVersioninto the monitoring chart; it lands as a pod-template annotation, so any rotation rolls the Grafana pod (resourceVersion, not a password hash — nothing password-derived in the pod spec)cd-prodcallscd-monitoringas a reusable workflow after the app deploy (workflow_callruns at the caller's ref = the release tag);cd-monitoringkeepsworkflow_dispatchas the manual escape hatch and now shares thedeploy-prodconcurrency groupvault_grafana_admin_passwordin the encrypted prod vaultSteps for Testing
helm template infra/helm/monitoring --set grafana.adminSecretVersion=123→ deployment renders asecret-version/admin: "123"pod annotation; without the value, no annotationcd-prodrun shows thedeploy-monitoringjob;kubectl -n genops-monitoring get podshows Grafana re-rolled and login works with the new vault passwordChecklist
Summary by CodeRabbit
New Features
Bug Fixes
Documentation