CI flakiness: CloudNativePG postgresql-cluster fails to reach Ready on GKE CI (operator not reconciling)

[eamonnmoloney]

Summary

Integration scenarios that provision a per-namespace CloudNativePG Cluster (postgresql-cluster) intermittently fail because the cluster never reaches Ready. Root indicator: the Cluster CR applies successfully but the CNPG operator does not reconcile it — no postgresql-cluster-1 instance pod is ever created. This is an infrastructure/operator fault on the shared GKE CI cluster, not a chart or test-code bug.

It surfaced as repeated red CI on PR #6318 (CI scenario-registry migration), but #6318 only changes test config + Go matrix loader — the same commit was green the day before.

Evidence

Workflow Test - Chart Version. Same PR commit (#6318): green 2026-06-03, red repeatedly 2026-06-04.

• Green only after retries (run_attempt: 3): https://github.com/camunda/camunda-platform-helm/actions/runs/26966949424
• Failed runs: https://github.com/camunda/camunda-platform-helm/actions/runs/26959405631 · https://github.com/camunda/camunda-platform-helm/actions/runs/26963855638 · https://github.com/camunda/camunda-platform-helm/actions/runs/26966029910

Two surface signatures, one root cause — they differ only in how each pre-install hook reacts to the missing DB:

1. Fixture-mode hooks (eske/elasticsearch, keyco/keycloak-original, esa0/auth0) have no readiness gate → web-modeler-restapi / identity crash-loop on java.net.UnknownHostException: postgresql-cluster-rw → helm upgrade --install --wait --timeout 1200s fails after ~20 min with the opaque context deadline exceeded.
2. Script-mode hooks (osss/osot, opensearch-self-signed) gate with kubectl wait --for=condition=Ready --timeout=300s cluster.postgresql.cnpg.io/postgresql-cluster → fail fast at 5 min: error: timed out waiting for the condition on clusters/postgresql-cluster.

Decisive detail (job osot/opensearch-self-signed-os-trust): the Cluster CR is serverside-applied successfully, then never becomes Ready, and no postgresql-cluster-1 instance pod is ever created. That rules out slow PVC-binding / image-pull (which would leave a Pending pod) — the operator simply was not reconciling the CR.

Provisioning source: the cnpg hook applies charts/camunda-platform-8.10/test/integration/scenarios/common/resources/postgresql-cluster.yaml (a postgresql.cnpg.io/v1 Cluster named postgresql-cluster). Components connect to the in-namespace postgresql-cluster-rw service the operator is supposed to create.

Why it looks "flaky"

The same scenario passes on retry / on 06-03 and fails on 06-04 purely because CNPG operator health on the shared CI cluster varies over time. Nothing in the test inputs changed between green and red runs of the same commit.

Track A — the fix (infra; removes the flake)

• Check the CloudNativePG operator pod health (Running, not crash-looping / OOM / rate-limited) on the CI clusters under *.ci.distro.ultrawombat.com; confirm postgresql.cnpg.io CRDs and the operator webhook are healthy.
• Review operator logs around 2026-06-04 14:00–17:30 UTC for reconcile errors / leader-election loss.
• Check node capacity / preemptible churn in that window.
• Add operator-health monitoring/alerting and (optionally) a nightly pre-flight gate that asserts CNPG is healthy before the matrix fans out — so a bad cluster fails once, fast, with one clear message instead of N scenario timeouts.

Track B — harness hardening (does NOT remove the flake; makes it fast & diagnosable)

• Give fixture-mode hooks the readiness gate that script-mode already has, so eske/keyco/esa0 fail in ~5 min with an explicit "CNPG cluster postgresql-cluster not Ready" error instead of a 20-min opaque helm timeout. Implement WaitForCNPGClusterReady(ctx, namespace, name, timeout) in scripts/camunda-core/pkg/kube (existing Client.dynamicClient, GVR postgresql.cnpg.io/v1, clusters, poll status.conditions[type=Ready]) and call it from scripts/deploy-camunda/deploy/lifecycle.go ApplyLifecycleManifests (guard with HasCRD).
• Dump CNPG operator + Cluster.status + namespace pod/event diagnostics on not-Ready.
• Wide blast radius (shared Go tooling): validate green across all supported chart versions.

Note: Track B alone will NOT make these scenarios pass when CNPG is down — the script-mode hooks already wait and still fail. It only converts slow/opaque failures into fast/diagnosable ones. Track A is required to actually stop the flakiness.

[distro-ci]

🧭 Triage Assessment

Field	Value
Work type	BUG
Kind	bug
Severity	⚠️ Mid
Likelihood	🔥 High
Confidence	0.89

📝 Rationale

Within scope: SM Platform Team Boundaries assign Helm Charts, the camunda/camunda-platform-helm repository, and automated scenario testing and deployments to the Distribution/SM Platform boundary, so this issue belongs here. I classify it as BUG with kind=bug because the issue describes incorrect existing behavior in an owned system: CNPG-backed GKE integration scenarios intermittently fail because the Cluster CR is applied but never reconciled, so the expected database service never appears; the Distro non-feature guide reserves chore/internal/research for maintenance, internal-only improvement, or investigation work rather than broken current behavior, and the bug guide expects kind/bug for bugs. Severity is mid: the bug-triage guide defines mid as noticeable impact without data loss, while high is for impact without a simple workaround; here the issue shows repeated red CI, long opaque timeouts, and broad disruption across several GKE scenarios, but it also shows at least one successful rerun and does not indicate a customer-facing production outage, data loss, or security problem. Likelihood is high because the bug-triage guide defines high as frequently observed or unknown scope, and the report documents multiple failures of the same PR across separate runs plus a wide blast radius across CNPG-backed scenarios on the shared GKE CI cluster. Report quality is sufficient, so missing_info is false: the bug guide asks for a clear problem statement, expected vs actual behavior, reproduction context, and relevant logs/environment details, and this issue includes the summary, evidence links, failure signatures, affected scenarios, root-cause hypothesis, and concrete remediation tracks. I did not find a strict duplicate in camunda-platform-helm; the closest related issues are #6192, #6212, and #5665, which all touch the same CNPG/RDBMS testing surface, but they cover missing coverage or scenario-specific defects rather than this shared-operator reconciliation flake.

🔍 Similar Issues

#6192, #6212, #5665

---

🤖 This issue was automatically triaged by an AI-based system.
✋ If you believe this assessment is incorrect, update the labels or add the triage:manual label for human review.

[eamonnmoloney]

Addressed by #6339 (draft).

#6339 removes the CloudNativePG dependency from the 8.10 CI scenarios entirely, routing Identity / Web Modeler / RDBMS at the internal-postgresql companion chart (the approach already used for the EKS scenarios) instead of provisioning a per-namespace CNPG Cluster. This eliminates the operator-reconcile dependency that causes the flakiness described above — the matrix no longer waits on cluster.postgresql.cnpg.io/postgresql-cluster becoming Ready.

This is an interim workaround: it should stand until we have a stable CNPG operator setup on the CI cluster (Track A), at which point CNPG can be reinstated if we still want operator coverage. Track B (the fail-fast readiness gate / diagnostics) is therefore not needed while #6339 is in place.