fix(8.9): orchestration startup blocks on unreachable secondary-storage schema init (intermittent nosec failures)

[eamonnmoloney]

Summary

While fixing the consistent 8.9 - nosec - install - gke (noSecondaryStorage) CI failure (see #6346), a deeper, latent robustness bug surfaced that is worth investigating on its own: when the orchestration app is configured with a secondary-storage exporter/schema target that is unreachable, the Spring bootstrap blocks on SchemaManager "init schema" retries and never binds the :9600 management port — so the kubelet startup probe gets connection refused, the container is killed and restart-loops, and Helm --wait times out.

This is intermittent, which is why it hid for a while:

• Run 26953963891 (2026-06-04): passed in 4m43s with the misconfigured values.
• Runs 26999716632 and 26998647996 (2026-06-05): failed with context deadline exceeded.

Controlled reproduction (GKE, back-to-back, same cluster)

	misconfigured (exporter → absent ES)	corrected (no exporter)
helm install	timeout, FAIL in 15m30s	success in 3m18s
broker pods	0/1 Running, restarts	1/1 Running, 0 restarts
startup probe	:9600 connection refused ×55 over 14m	healthy
init schema	retried to attempt 29	n/a

Broker signature:

io.camunda.search.schema.SchemaManager - Schema creation is enabled. Start Schema management.
RetryDecorator - Retrying operation for 'init schema': attempt 29. Message: Failed to check existence of index ...
io.camunda.zeebe.broker.exporter - Failed to open exporter 'camundaexporter'. Retrying...
Startup probe failed: dial tcp :9600: connect: connection refused
Container orchestration failed startup probe, will be restarted

#6346 fixes the immediate CI scenario by not pointing the exporter at a non-existent backend. But the underlying behavior is a bug regardless of that scenario.

Questions to investigate

1. Should SchemaManager init run on the bootstrap/startup path synchronously at all, or should it be async / retried in the background so the management server can bind :9600 and report an unhealthy (503) startup state instead of refusing connections?
2. Should an unreachable/misconfigured secondary storage fail fast with a clear error rather than retry indefinitely behind a blocked startup?
3. Is the startup probe budget (failureThreshold=30, period=10s, delay=30s ≈ 330s) appropriate, and should connection refused vs 503 be distinguished?
4. Why intermittent? Determine the timing/race that let 2026-06-04 pass with the same config (e.g., schema-check timeout vs probe budget, node CPU at startup).

Notes

• Likely spans the helm chart (probe config) and the orchestration app (io.camunda.search.schema.SchemaManager, exporter open lifecycle) in camunda-monorepo — may need a cross-repo issue/transfer.
• Immediate CI fix: test(8.9): disable secondary-storage exporter in noSecondaryStorage scenario #6346. This issue tracks the robustness/flakiness root cause behind it.

[distro-ci]

🧭 Triage Assessment

Field	Value
Work type	BUG
Kind	bug
Severity	🔴 High
Likelihood	🔶 Mid
Confidence	0.86

📝 Rationale

Within scope for the SM Platform Distribution team because the team-boundaries guidance assigns Helm Charts, the camunda/camunda-platform-helm repository, deployment readiness/periphery such as secondary-storage infrastructure, and issues related to Self-Managed Helm chart configurations to Distribution. This is a BUG with kind bug because the Distro triage defines kind/bug as incorrect existing behavior / broken functionality that must be assessed with severity and likelihood, and the issue reports that orchestration can block startup, never bind :9600, fail the startup probe, restart-loop, and cause Helm --wait timeouts when secondary storage is unreachable. I assess severity as high rather than mid because, when triggered, this breaks a core flow for the affected deployment (startup/install) and leaves the cluster unable to come up, but there is no evidence of data loss, privilege escalation, or other stop-the-world critical criteria. The behavior also appears inconsistent with the expected fail-fast readiness pattern documented for connectivity failures, which says connection errors typically cause the application not to become Ready rather than hiding behind long schema-init retries and connection refused on the management port. I assess likelihood as mid because it has been reproduced in controlled runs and observed in multiple failing CI runs, but it depends on a specific condition (misconfigured or unreachable secondary storage during startup) and is described as intermittent rather than broadly affecting all installs or environments. I did not find a clear duplicate issue in camunda/camunda-platform-helm; the closest related item is PR #6346, which fixes the immediate CI misconfiguration and explicitly treats this as a separate underlying robustness bug. Confidence is below very-high because the issue itself notes the likely fix may span the helm repo and the orchestration app in camunda/camunda, so final ownership/action may require cross-repo transfer even though initial triage is in scope here. The report has enough information for triage: it includes a clear summary, concrete reproduction/results, logs, and the relevant failure signatures, so missing-info follow-up is not required to classify it.

---

🤖 This issue was automatically triaged by an AI-based system.
✋ If you believe this assessment is incorrect, update the labels or add the triage:manual label for human review.

[eamonnmoloney]

Note: the immediate values fix referenced above landed in #6331 (not a standalone PR). #6346 was closed as a duplicate. This issue still tracks the underlying robustness bug (synchronous schema init blocking the :9600 bind on an unreachable secondary storage).