Roll Grafana on secret rotation, release-gate monitoring deploys

.github/workflows/cd-monitoring.yml

@@ -2,30 +2,21 @@ name: CD - Monitoring (Rancher)
 
 # Deploys the Prometheus/Grafana chart to the TUM Rancher monitoring namespace.
 #
-# Path-triggered on main rather than release-gated (unlike cd-prod): the chart
-# ships config only — dashboards, alert rules, operator CRs — no application
-# images, so a merged dashboard edit should reach prod immediately without
-# cutting a release. The blast radius is the monitoring namespace, never the
-# app.
+# Monitoring is RELEASE-GATED: cd-prod calls this workflow right after the app
+# deploy, and a workflow_call runs at the caller's ref — the release tag — so
+# dashboards/alerts always match the shipped app version.
 on:
-  push:
-    branches: [main]
-    paths:
-      - "infra/helm/monitoring/**"
-      - "infra/ansible/playbooks/monitoring.yml"
-      # The playbook + summary read monitoring_namespace/grafana_url from the
-      # prod inventory — an edit there must redeploy too.
-      - "infra/ansible/inventories/prod/**"
-      - ".github/workflows/cd-monitoring.yml"
+  workflow_call:
   workflow_dispatch:
 
 permissions:
   contents: read
 
-# One monitoring deploy at a time; queued, never cancelled. Separate group from
-# deploy-prod — the two releases touch disjoint namespaces.
+# Same group as cd-prod/destroy-prod: cd-prod now upgrades the `monitoring`
+# Helm release too, and two concurrent upgrades of one release conflict —
+# queued, never cancelled.
 concurrency:
-  group: deploy-monitoring
+  group: deploy-prod
   cancel-in-progress: false
 
 jobs:
@@ -58,9 +49,6 @@ jobs:
         with:
           version: v3.16.0
 
-      # The vault password is needed even though this playbook reads no vault
-      # secrets: the prod inventory auto-loads group_vars/all/vault.yml, and
-      # ansible refuses to parse an encrypted file it cannot decrypt.
       - name: Deploy monitoring via Ansible
         working-directory: infra/ansible
         env:

.github/workflows/cd-prod.yml

@@ -113,3 +113,12 @@ jobs:
           image_tag: ${{ steps.tag.outputs.tag }}
           vault_password: ${{ secrets.ANSIBLE_VAULT_PASSWORD_PROD }}
           app_url: ${{ steps.host.outputs.app_url }}
+
+  # Release-gate monitoring: run cd-monitoring as a called workflow AFTER the
+  # app deploy. A workflow_call runs at the CALLER's ref — the release tag —
+  # so dashboards/alerts deploy from the same tagged commit as the app and
+  # never reference metrics that haven't shipped yet.
+  deploy-monitoring:
+    needs: deploy
+    uses: ./.github/workflows/cd-monitoring.yml
+    secrets: inherit

CLAUDE.md

@@ -224,7 +224,7 @@ Service implementation status:
 - `web-client` — ✅ product shell (React + Vite + RTK Query): auth, dashboard with pipeline, applications board with detail drawer + generated-document tabs, profile editor, onboarding wizard, assistant chat. Jobs page is a static placeholder. No client tests yet.
 - `gateway` — ✅ Spring Cloud Gateway (JWT auth + routing), in compose/Helm.
 - `document` — ✅ functional (Spring Boot): profile aggregate + sub-resource CRUD (work experiences, educations, skills, languages), generated-document storage (cover letters/resumes per application), Flyway schema (`ddl-auto=validate`), verify-only JWT (owner from `sub`); port 8083 local.
-- **Monitoring** — ✅ Prometheus/Grafana in place: all four Spring services emit RED metrics via Micrometer on the management port 8090 (`/actuator/prometheus`), `genai` via prometheus-fastapi-instrumentator; custom auth security counters from `SecurityAuditLog`; `app_info{version}` everywhere. Local stack via the `monitoring` compose profile (Grafana :3001); prod stack in namespace `genops-monitoring` via the platform Prometheus Operator, deployed with the `infra/helm/monitoring` chart. Rules + dashboards live once in `infra/helm/monitoring/files/` (chart renders them; local compose mounts the same files). 4 alert rules; Alertmanager routing deferred. LLM observability via Langfuse.
+- **Monitoring** — ✅ Prometheus/Grafana in place: all four Spring services emit RED metrics via Micrometer on the management port 8090 (`/actuator/prometheus`), `genai` via prometheus-fastapi-instrumentator; custom auth security counters from `SecurityAuditLog`; `app_info{version}` everywhere. Local stack via the `monitoring` compose profile (Grafana :3001); prod stack in namespace `genops-monitoring` via the platform Prometheus Operator, deployed with the `infra/helm/monitoring` chart (release-gated: `cd-prod` calls `cd-monitoring` after the app deploy so monitoring ships from the same release tag; manual `workflow_dispatch` as escape hatch). Rules + dashboards live once in `infra/helm/monitoring/files/` (chart renders them; local compose mounts the same files). 4 alert rules; Alertmanager routing deferred. LLM observability via Langfuse.
 
 PostgreSQL schemas designed and migration files written for all services:
 - `auth` — schema auto-generated by Hibernate (`ddl-auto=update`) from `@Entity` classes
@@ -233,4 +233,4 @@ PostgreSQL schemas designed and migration files written for all services:
 - `email` — Flyway migration at `src/main/resources/db/migration/` (`ddl-auto=validate`)
 - `genai` — Alembic migration at `alembic/versions/`
 
-`docker-compose.yml` runs `postgres` + `pgadmin` (dev) + `redis` + `web-client` + `auth` + `application` + `document` + `email` + `genai` (`gateway` not yet added). Set `INTERNAL_SERVICE_TOKEN` in `.env` to activate the email application-detection pipeline (blank = disabled, fail-closed). In Kubernetes the email service deploys behind `email.enabled`: CD flips it automatically once the Gmail vault vars are set (which also create the `email-secrets` Secret), the gateway routes `/api/v1/email/**` via `EMAIL_URI`, and `internalServiceTokenSecret`/the `internal-service-token` Secret activate the pipeline — see `services/email/README.md` for the per-host manual steps (vault vars + Google redirect-URI registration). A Helm chart at `infra/helm/jobready/` deploys the system to Kubernetes — Azure AKS (auto on merge to main) and TUM Rancher (release-gated). LLM observability is in place via **Langfuse** — self-hosted locally under `monitoring/langfuse/` (opt-in `monitoring` compose profile) and Langfuse Cloud for deployed envs (wired through the Helm chart + Ansible vault). Prometheus/Grafana metrics config (dashboards/alerts) is still pending — see the readiness plan.
+`docker-compose.yml` runs `postgres` + `pgadmin` (dev) + `redis` + `web-client` + `auth` + `application` + `document` + `email` + `genai` (`gateway` not yet added). Set `INTERNAL_SERVICE_TOKEN` in `.env` to activate the email application-detection pipeline (blank = disabled, fail-closed). In Kubernetes the email service deploys behind `email.enabled`: CD flips it automatically once the Gmail vault vars are set (which also create the `email-secrets` Secret), the gateway routes `/api/v1/email/**` via `EMAIL_URI`, and `internalServiceTokenSecret`/the `internal-service-token` Secret activate the pipeline — see `services/email/README.md` for the per-host manual steps (vault vars + Google redirect-URI registration). A Helm chart at `infra/helm/jobready/` deploys the system to Kubernetes — Azure AKS (auto on merge to main) and TUM Rancher (release-gated). LLM observability is in place via **Langfuse** — self-hosted locally under `monitoring/langfuse/` (opt-in `monitoring` compose profile) and Langfuse Cloud for deployed envs (wired through the Helm chart + Ansible vault). Prometheus/Grafana dashboards and alert rules are in place (see the Monitoring bullet above); Alertmanager routing is the remaining gap — see the readiness plan.

README.md

@@ -7,6 +7,17 @@
 [![CD — Dev (AKS)](https://github.com/AET-DevOps26/team-genops/actions/workflows/cd-dev.yml/badge.svg?branch=main)](https://github.com/AET-DevOps26/team-genops/actions/workflows/cd-dev.yml?query=branch%3Amain)
 [![CodeQL](https://github.com/AET-DevOps26/team-genops/actions/workflows/codeql.yml/badge.svg?branch=main)](https://github.com/AET-DevOps26/team-genops/actions/workflows/codeql.yml?query=branch%3Amain)
 
+**Live environments**
+
+| Environment | URL | Deployed by |
+|---|---|---|
+| Dev (Azure AKS) | https://jobready-development.austriaeast.cloudapp.azure.com | every merge to `main` |
+| Prod (TUM Rancher) | https://jobready.stud.k8s.aet.cit.tum.de | GitHub release |
+| Grafana (prod monitoring) | https://genops-grafana.stud.k8s.aet.cit.tum.de | with prod releases |
+
+> The prod Prometheus UI is deliberately **not** exposed — a NetworkPolicy admits only Grafana.
+> Query metrics through Grafana (Explore) instead.
+
 JobReady consolidates the job-search journey into a single intelligent platform:
 
 - **Structured candidate profile** — work experience, education, skills, languages
@@ -289,8 +300,9 @@ pytest e2e_tests/ -v
 - **CD — Dev** (`cd-dev.yml`) — merge to `main` deploys automatically to **Azure AKS** via the
   [`infra/helm/jobready`](infra/helm/jobready) chart.
 - **CD — Prod** (`cd-prod.yml`) — release-gated deployment to **TUM Rancher**.
-- **CD — Monitoring** (`cd-monitoring.yml`) — changes under `infra/helm/monitoring/` deploy the
-  Prometheus/Grafana stack to the `genops-monitoring` namespace.
+- **CD — Monitoring** (`cd-monitoring.yml`) — release-gated: `cd-prod` calls it after every app
+  deploy, so the Prometheus/Grafana stack in `genops-monitoring` ships from the same release tag
+  as the app. `workflow_dispatch` is the escape hatch for urgent dashboard/alert fixes.
 - **CodeQL** (`codeql.yml`) — static security analysis.
 
 Infrastructure is provisioned with **Terraform** (`infra/terraform/`) and configured/deployed with