Support Kubernetes Secret MEK rotation (#1307)

* Support Kubernetes Secret MEK rotation

Read MEKs from operator-owned Kubernetes Secrets, with an opt-in create-only bootstrap for disposable installs. Coordinate PREPARE and ACTIVATE through Kubernetes Leases and exact pod startup descriptors, then rewrap UEKs and direct-MEK configuration only through the explicit lifecycle Job. Keep historical MEKs loaded and reject removal. Persist no MEK lifecycle tables, triggers, functions, or key material in PostgreSQL.

* Fix MEK CI validation

* Fix KIND quick-start MEK bootstrap race

* Improve MEK startup diagnostics

* Handle empty MEK-encrypted config secrets

* Fix MEK KIND Helm lifecycle tests

Author: vvnpn-nv

.github/workflows/oetf-kind.yaml

@@ -189,7 +189,13 @@ jobs:
             df -h /
             echo "::endgroup::"
           ' EXIT
+          # Keep this below the job's 60-minute ceiling. Without an inner
+          # deadline, a stuck Helm/Kubernetes wait consumes the whole job and
+          # GitHub kills it before the diagnostic and artifact steps can run.
+          # `timeout` returns 124, which is an ordinary step failure, leaving
+          # roughly ten minutes for cluster state and logs to be collected.
           /usr/bin/time -p -o /tmp/oetf.time \
+            timeout --signal=TERM --kill-after=30s 40m \
             bazel run //test/oetf:deploy_and_run -- \
               --env kind --tags kind \
               --target-pattern //test/smoke/... \
@@ -242,6 +248,13 @@ jobs:
           done
           echo "::endgroup::"
 
+          echo "::group::All OSMO pod logs (last 80 lines)"
+          for pod in $(kubectl get pods -n osmo -o name 2>/dev/null || true); do
+            echo "--- osmo/$pod ---"
+            kubectl logs -n osmo "$pod" --all-containers --tail=80 2>/dev/null || true
+          done
+          echo "::endgroup::"
+
           echo "::group::Helm releases (helm list -A)"
           helm list -A || true
           echo "::endgroup::"

deployments/README.md

@@ -161,18 +161,6 @@ kubectl create secret generic local-admin-password \
   --from-literal=password="$LOCAL_ADMIN_PASSWORD" \
   --dry-run=client -o yaml | kubectl apply -f -
 
-if ! kubectl get configmap mek-config --namespace osmo >/dev/null 2>&1; then
-  MEK_KEY=$(dd if=/dev/urandom bs=32 count=1 2>/dev/null | base64 | tr -d '\n')
-  MEK_JWK=$(printf '{"k":"%s","kid":"key1","kty":"oct"}' "$MEK_KEY" | base64 | tr -d '\n')
-  MEK_FILE=$(mktemp)
-  printf 'currentMek: key1\nmeks:\n  key1: %s\n' "$MEK_JWK" > "$MEK_FILE"
-  kubectl create configmap mek-config \
-    --namespace osmo \
-    --from-file=mek.yaml="$MEK_FILE" \
-    --dry-run=client -o yaml | kubectl apply -f -
-  rm -f "$MEK_FILE"
-fi
-
 helm repo add osmo https://helm.ngc.nvidia.com/nvidia/osmo
 helm repo update osmo
 
@@ -187,10 +175,10 @@ helm upgrade --install osmo-backend-operator osmo/backend-operator \
   --wait
 ```
 
-For the local quick-start only, the service values generate
-`backend-operator-token` during the first Helm install. Because the backend
-operator is installed in the same namespace, it consumes that Secret directly;
-no pre-created backend Secret is required.
+For the local quick-start only, the service values generate `osmo-mek` and
+`backend-operator-token` during the first Helm install. Both are generated
+inside Kubernetes and preserved across upgrades; no pre-created MEK or backend
+Secret is required. Production values keep both bootstrap modes disabled.
 
 After installing the CLI and logging in, set the demo pool and LocalStack data credential:
 

deployments/charts/osmo/README.md

@@ -195,7 +195,9 @@ secrets:
   objectStorage:
     existingSecret: osmo-object-storage
   masterEncryptionKey:
-    existingSecret: osmo-master-encryption-key
+    existingSecret:
+      name: osmo-master-encryption-key
+      key: mek.yaml
 ```
 
 Keep `embeddedDependencies.postgresql.enabled: false` (the default), then
@@ -462,14 +464,96 @@ postgresql:
         name: osmo-postgresql-credentials
 ```
 
-When an external PostgreSQL or Valkey service uses a private CA, enable TLS in
-the matching `externalDependencies` block and reference the CA Secret there.
-The Valkey `caKey` must hold a complete PEM trust bundle, including the public
-or system roots used by other HTTPS endpoints; OSMO's Python services consume
-that bundle through `SSL_CERT_FILE`. The default Valkey key is `ca-bundle.crt`.
+The MEK is mounted through the typed
+`secrets.masterEncryptionKey.existingSecret.{name,key}` reference. Use
+`managementMode: external` for an operator-owned read-only Secret. Use
+`managementMode: osmo` when this release should create and update that exact
+Secret through explicitly requested lifecycle Jobs.
+
+For a disposable install backed by a new database, enable `bootstrap`. Helm
+renders no MEK Secret data. A namespace-scoped create-only lifecycle Job waits
+for PostgreSQL, proves that the database has no users, UEKs, or dynamic
+configuration, verifies that every chart consumer is blocked before its writer
+container starts, and atomically creates the full Secret. Key material never
+enters Helm output or release state. A retry accepts only the exact Secret owned
+by this installation and authenticates the retained database before succeeding;
+it never overwrites or deletes a Secret. Non-chart database writers must be
+stopped for initial bootstrap.
+
+After the bootstrap Job succeeds, commit and sync
+`secrets.masterEncryptionKey.bootstrap.enabled: false`. This mandatory second
+Helm/GitOps transaction removes bootstrap Secret-creation RBAC from desired
+state. The chart rejects a rotation phase while bootstrap remains enabled.
+If bootstrap fails or its database credentials are corrected under Argo CD or
+Flux, increment the non-secret `bootstrap.attempt` before syncing again; this
+creates a new immutable retry Job without deriving public names from credential
+bytes.
+
+Every consumer loads its keyring once at process startup. Before becoming
+ready, it performs a bounded authenticated inventory of every UEK wrapper and
+registered direct-MEK configuration value. It then logs one machine-readable
+`OSMO_MEK_DESCRIPTOR` containing only the current key ID, loaded key IDs,
+generation, and non-secret bundle digest. There are no MEK database tables,
+triggers, polling loops, or hot reloads.
+
+Rotation is an explicit three-phase operation. Use one unique request ID for
+the whole rotation and keep every previous key in the Secret:
+
+1. Set `rotation.phase=prepare`. The managed Job adds exactly one key and leaves
+   `currentMek` unchanged. After the Job succeeds, clear the phase, change
+   `rotation.rolloutRevision`, and sync again to roll every consumer.
+2. Set `rotation.phase=activate`. The Job first verifies that the complete,
+   Ready Pod cohort belongs to the expected Deployments and logged the PREPARE
+   descriptor, then selects the new key. Clear the phase, change
+   `rolloutRevision` again, and sync to perform the second rollout.
+3. Set `rotation.phase=rewrap`. The Job verifies the ACTIVATE cohort, then
+   compare-and-swap rewraps all UEKs and registered direct-MEK configuration
+   from the beginning and runs two authenticated inventories. Clear the phase
+   after success.
+
+For example, the same values changes work with Helm upgrades or separate Argo
+CD syncs:
 
-For an external Valkey endpoint signed by a public CA, leave
-`caExistingSecret` empty to use the image's system trust store.
+```yaml
+secrets:
+  masterEncryptionKey:
+    managementMode: osmo
+    bootstrap:
+      enabled: false
+      attempt: "1" # increment only to retry a failed bootstrap
+    rotation:
+      requestId: rotate-2026-08-21
+      phase: prepare       # then "", activate, "", rewrap, ""
+      rolloutRevision: "1" # change to "2" after PREPARE and "3" after ACTIVATE
+```
+
+Each Job creates or reuses a release-scoped Kubernetes Lease directly. The
+Lease is intentionally absent from Helm desired state, so GitOps self-heal
+cannot clear a live holder. A Lease held by another attempt is never stolen,
+even after its timestamp expires. If an attempt dies, delete its old Job/Pod,
+verify it is gone, clear the Lease holder, increment `rotation.attempt`, and
+retry the same phase. Jobs never delete Pods or patch Deployments; Helm or the
+GitOps controller owns both rollouts. Clear a completed phase promptly so its
+narrowly scoped ServiceAccount and RoleBinding leave the desired state.
+
+In `managementMode=external`, the operator performs PREPARE and ACTIVATE by
+updating the existing Secret, with one rollout after each update. Then set only
+`rotation.phase=rewrap`. The rewrap Job has exact-name Secret `get` permission,
+not `patch` or `update`, and enforces the same ACTIVATE Pod attestation before
+touching ciphertext.
+
+Rewrap completion is point-in-time evidence, not permission to remove an old
+key. Because this design deliberately has no database write fence, all old MEKs
+remain mandatory. User plaintext and UEK key material do not change; only their
+encrypted wrappers change. Normal application reads never perform MEK rewrap
+writes; the explicit Job is the sole orchestrator.
+
+For an external Valkey endpoint signed by a public CA, enable
+`externalDependencies.valkey.tls.enabled` and leave `caExistingSecret` empty to
+use the image's system trust store. For a private CA, set `caExistingSecret` and
+`caKey` in the same block. The selected key must contain the complete trust
+bundle because clients use it through `SSL_CERT_FILE`. PostgreSQL private CAs
+are also configured in its `externalDependencies` TLS block.
 
 ## Exposure
 

deployments/charts/osmo/profiles/kind-self-contained.yaml

@@ -44,8 +44,12 @@ secrets:
       managedSecret:
         name: osmo-backend-token
   masterEncryptionKey:
-    generate: true
-    existingSecret: ''
+    managementMode: osmo
+    existingSecret:
+      name: osmo-master-encryption-key
+      key: mek.yaml
+    bootstrap:
+      enabled: true
 
 services:
   mcp:

deployments/charts/osmo/profiles/split-plane-control.yaml

@@ -115,4 +115,6 @@ secrets:
   objectStorage:
     existingSecret: osmo-object-storage
   masterEncryptionKey:
-    existingSecret: osmo-master-encryption-key
+    existingSecret:
+      name: osmo-master-encryption-key
+      key: mek.yaml

deployments/charts/osmo/templates/_helpers.tpl

@@ -393,26 +393,23 @@ data:
 {{- end -}}
 
 {{- define "osmo.secrets.mekVolume" -}}
-{{- with (include "osmo.masterEncryptionKey.secretName" .) }}
+{{- with .Values.secrets.masterEncryptionKey.existingSecret.name }}
 - name: mek-volume
   secret:
-    secretName: {{ . }}
+    secretName: {{ . | quote }}
     items:
-    - key: {{ $.Values.secrets.masterEncryptionKey.keys.config }}
-      path: mek.yaml
+    - key: {{ required "secrets.masterEncryptionKey.existingSecret.key is required" $.Values.secrets.masterEncryptionKey.existingSecret.key | quote }}
+      path: "mek.yaml"
 {{- end }}
 {{- end -}}
 
-{{- define "osmo.masterEncryptionKey.secretName" -}}
-{{- if .Values.secrets.masterEncryptionKey.generate -}}
-{{- printf "%s-master-encryption-key" (include "osmo.fullname" .) | trunc 63 | trimSuffix "-" -}}
-{{- else -}}
-{{- .Values.secrets.masterEncryptionKey.existingSecret -}}
-{{- end -}}
-{{- end -}}
+{{- define "osmo.secrets.mekFile" -}}/opt/osmo/mek/mek.yaml{{- end -}}
+{{- define "osmo.secrets.mekMountPath" -}}/opt/osmo/mek{{- end -}}
 
-{{- define "osmo.masterEncryptionKey.bootstrapName" -}}
-{{- printf "%s-mek-bootstrap" (include "osmo.fullname" .) | trunc 63 | trimSuffix "-" -}}
+{{- define "osmo.secrets.mekRolloutAnnotation" -}}
+{{- with .Values.secrets.masterEncryptionKey.rotation.rolloutRevision }}
+osmo.nvidia.com/mek-rollout: {{ . | quote }}
+{{- end }}
 {{- end -}}
 
 {{- define "osmo.valkey.fullname" -}}

deployments/charts/osmo/templates/agent-service.yaml

@@ -46,6 +46,7 @@ spec:
         {{- include "osmo.pod.labels" (dict "root" . "component" .Values.services.agent "componentName" "agent") | nindent 8 }}
       annotations:
         {{- include "osmo.pod.annotations" (dict "root" . "component" .Values.services.agent) | nindent 8 }}
+        {{- include "osmo.secrets.mekRolloutAnnotation" . | nindent 8 }}
     spec:
       automountServiceAccountToken: {{ .Values.services.agent.serviceAccount.automountServiceAccountToken }}
       securityContext:
@@ -107,9 +108,9 @@ spec:
         - {{ include "osmo.postgresql.port" . | quote }}
         - --postgres_database_name
         - {{ include "osmo.postgresql.database" . }}
-        {{- if (include "osmo.masterEncryptionKey.secretName" .) }}
+        {{- if .Values.secrets.masterEncryptionKey.existingSecret.name }}
         - --mek_file
-        - {{ .Values.secrets.masterEncryptionKey.mountPath }}
+        - {{ include "osmo.secrets.mekFile" . | quote }}
         {{- end}}
         {{- with (include "osmo.postgresql.username" .) }}
         - --postgres_user
@@ -138,10 +139,10 @@ spec:
           mountPath: /tmp
         - name: osmo-progress-files
           mountPath: /var/run/osmo
-        {{- if (include "osmo.masterEncryptionKey.secretName" .) }}
-        - mountPath: {{ .Values.secrets.masterEncryptionKey.mountPath }}
+        {{- if .Values.secrets.masterEncryptionKey.existingSecret.name }}
+        - mountPath: {{ include "osmo.secrets.mekMountPath" . | quote }}
           name: mek-volume
-          subPath: mek.yaml
+          readOnly: true
         {{- end }}
         {{- include "osmo.configuration.volumeMounts" . | nindent 8 }}
         {{- include "osmo.externalDependencies.caVolumeMounts" . | nindent 8 }}

deployments/charts/osmo/templates/api-service.yaml

@@ -47,6 +47,7 @@ spec:
         {{- include "osmo.pod.labels" (dict "root" . "component" .Values.services.api "componentName" "api") | nindent 8 }}
       annotations:
         {{- include "osmo.pod.annotations" (dict "root" . "component" .Values.services.api "protectedAnnotations" $protectedPodAnnotations) | nindent 8 }}
+        {{- include "osmo.secrets.mekRolloutAnnotation" . | nindent 8 }}
     spec:
       automountServiceAccountToken: {{ .Values.services.api.serviceAccount.automountServiceAccountToken }}
       securityContext:
@@ -102,9 +103,9 @@ spec:
         - {{ include "osmo.postgresql.port" . | quote }}
         - --postgres_database_name
         - {{ include "osmo.postgresql.database" . }}
-        {{- if (include "osmo.masterEncryptionKey.secretName" .) }}
+        {{- if .Values.secrets.masterEncryptionKey.existingSecret.name }}
         - --mek_file
-        - {{ .Values.secrets.masterEncryptionKey.mountPath }}
+        - {{ include "osmo.secrets.mekFile" . | quote }}
         {{- end }}
         {{- with (include "osmo.postgresql.username" .) }}
         - --postgres_user
@@ -180,10 +181,10 @@ spec:
         volumeMounts:
         - name: osmo-runtime-tmp
           mountPath: /tmp
-        {{- if (include "osmo.masterEncryptionKey.secretName" .) }}
-        - mountPath: {{ .Values.secrets.masterEncryptionKey.mountPath }}
+        {{- if .Values.secrets.masterEncryptionKey.existingSecret.name }}
+        - mountPath: {{ include "osmo.secrets.mekMountPath" . | quote }}
           name: mek-volume
-          subPath: mek.yaml
+          readOnly: true
         {{- end }}
         {{- if .Values.secrets.backendApiTokens.enabled }}
         {{- range .Values.secrets.backendApiTokens.credentials }}

deployments/charts/osmo/templates/delayed-job-monitor.yaml

@@ -40,7 +40,8 @@ spec:
       labels:
         {{- include "osmo.pod.labels" (dict "root" . "component" .Values.services.delayedJobMonitor "componentName" "delayed-job-monitor") | nindent 8 }}
       annotations:
-      {{- include "osmo.pod.annotations" (dict "root" . "component" .Values.services.delayedJobMonitor) | nindent 8 }}
+        {{- include "osmo.pod.annotations" (dict "root" . "component" .Values.services.delayedJobMonitor) | nindent 8 }}
+        {{- include "osmo.secrets.mekRolloutAnnotation" . | nindent 8 }}
     spec:
       automountServiceAccountToken: {{ .Values.services.delayedJobMonitor.serviceAccount.automountServiceAccountToken }}
       securityContext:
@@ -98,9 +99,9 @@ spec:
         - {{ include "osmo.postgresql.port" . | quote }}
         - --postgres_database_name
         - {{ include "osmo.postgresql.database" . }}
-        {{- if (include "osmo.masterEncryptionKey.secretName" .) }}
+        {{- if .Values.secrets.masterEncryptionKey.existingSecret.name }}
         - --mek_file
-        - {{ .Values.secrets.masterEncryptionKey.mountPath }}
+        - {{ include "osmo.secrets.mekFile" . | quote }}
         {{- end}}
         {{- with (include "osmo.postgresql.username" .) }}
         - --postgres_user
@@ -127,10 +128,10 @@ spec:
         volumeMounts:
         - name: osmo-progress-files
           mountPath: /var/run/osmo
-        {{- if (include "osmo.masterEncryptionKey.secretName" .) }}
-        - mountPath: {{ .Values.secrets.masterEncryptionKey.mountPath }}
+        {{- if .Values.secrets.masterEncryptionKey.existingSecret.name }}
+        - mountPath: {{ include "osmo.secrets.mekMountPath" . | quote }}
           name: mek-volume
-          subPath: mek.yaml
+          readOnly: true
         {{- end }}
         {{- include "osmo.component.extraVolumeMounts" .Values.services.delayedJobMonitor | nindent 8 }}
         {{- include "osmo.externalDependencies.caVolumeMounts" . | nindent 8 }}

deployments/charts/osmo/templates/logger-service.yaml

@@ -46,6 +46,7 @@ spec:
         {{- include "osmo.pod.labels" (dict "root" . "component" .Values.services.logger "componentName" "logger") | nindent 8 }}
       annotations:
         {{- include "osmo.pod.annotations" (dict "root" . "component" .Values.services.logger) | nindent 8 }}
+        {{- include "osmo.secrets.mekRolloutAnnotation" . | nindent 8 }}
     spec:
       automountServiceAccountToken: {{ .Values.services.logger.serviceAccount.automountServiceAccountToken }}
       securityContext:
@@ -102,9 +103,9 @@ spec:
         - {{ include "osmo.postgresql.port" . | quote }}
         - --postgres_database_name
         - {{ include "osmo.postgresql.database" . }}
-        {{- if (include "osmo.masterEncryptionKey.secretName" .) }}
+        {{- if .Values.secrets.masterEncryptionKey.existingSecret.name }}
         - --mek_file
-        - {{ .Values.secrets.masterEncryptionKey.mountPath }}
+        - {{ include "osmo.secrets.mekFile" . | quote }}
         {{- end}}
         {{- with (include "osmo.postgresql.username" .) }}
         - --postgres_user
@@ -133,10 +134,10 @@ spec:
           mountPath: /tmp
         - name: osmo-progress-files
           mountPath: /var/run/osmo
-        {{- if (include "osmo.masterEncryptionKey.secretName" .) }}
-        - mountPath: {{ .Values.secrets.masterEncryptionKey.mountPath }}
+        {{- if .Values.secrets.masterEncryptionKey.existingSecret.name }}
+        - mountPath: {{ include "osmo.secrets.mekMountPath" . | quote }}
           name: mek-volume
-          subPath: mek.yaml
+          readOnly: true
         {{- end }}
         {{- include "osmo.configuration.volumeMounts" . | nindent 8 }}
         {{- include "osmo.externalDependencies.caVolumeMounts" . | nindent 8 }}