feat: add elasticsearch-self-signed-upgrade scenario for 8.7->8.8 TLS upgrade testing (#5974)

Author: eamonnmoloney

.gitignore

@@ -43,6 +43,7 @@ scripts/deploy-camunda/.env
 charts/camunda-platform*/test/e2e/package-lock.json
 
 diagnostics/
+deployment-logs/
 STATE.md
 
 **/.omc

AGENTS.md

@@ -161,6 +161,52 @@ make tools.asdf-install
 - `8.7` and older use separate component template directories.
 - Never assume paths/components are identical across versions.
 
+### Template Gating Patterns
+Many template blocks are gated behind value flags that create implicit coupling:
+
+- **`global.elasticsearch.external`** gates ES auth env var injection in most components (8.7 and 8.8). A hard constraint in `constraints.tpl` blocks setting `external=true` when the bundled subchart is active (`elasticsearch.enabled=true`). To inject auth when using the bundled subchart, use component-level `env` overrides instead.
+- **`global.elasticsearch.tls.existingSecret`** triggers TLS truststore volume mounts and `JAVA_TOOL_OPTIONS` injection in most components. This is NOT behind the `external` gate — it works with the bundled subchart.
+- Template blocks often differ between versions (e.g., 8.7 Operate init container renders `operate.env` with `toYaml` but NOT `tpl`, so `{{ .Release.Name }}` in `valueFrom.secretKeyRef.name` is literal). Always read the specific version's template before writing env overrides.
+
+### Helm Subchart Value Override Patterns
+Helm's values merge is a **deep merge for maps** but a **full replace for arrays**. This matters for subchart overrides:
+
+```yaml
+# Parent chart values.yaml default:
+elasticsearch:
+  master:
+    extraEnvVars:           # <-- array
+      - name: SOME_VAR
+        value: "default"
+
+# Your overlay:
+elasticsearch:
+  master:
+    extraEnvVars: []        # Replaces the entire array — parent default is gone
+```
+
+This is the correct way to neutralize a parent chart's default array value. Setting `extraEnvVars: []` removes the parent's entries entirely. Setting `extraEnvVars: [{name: SOME_VAR, value: "override"}]` replaces the array with your single entry.
+
+**Contrast with deploy-camunda's merge:** The `deploy-camunda` CLI uses name-keyed deep merge for `env` arrays (matching on `name` field). But Helm itself does NOT — Helm replaces arrays wholesale. Know which merge strategy applies at each layer.
+
+### Bitnami Subchart Env Var Chains
+Bitnami charts often set env vars from multiple sources in a fixed order within the statefulset template:
+
+1. Security helper (from `security.*` values)
+2. `<role>.extraEnvVars` (e.g., `master.extraEnvVars`)
+3. Top-level `extraEnvVars`
+
+When Kubernetes encounters duplicate env var names, **the last one wins**. If the parent chart's `values.yaml` defaults an `extraEnvVars` entry that conflicts with a security helper value, you must either override or clear the array. To diagnose:
+
+```bash
+# Render and count occurrences of a suspicious env var:
+helm template integration charts/camunda-platform-8.X \
+  -f <your-values.yaml> \
+  --show-only charts/elasticsearch/templates/master/statefulset.yaml \
+  | grep -c 'ELASTICSEARCH_ENABLE_REST_TLS'
+# Should be exactly 1. If >1, there's a duplicate.
+```
+
 ## Commit and Branch Conventions
 - Branches: `issueId-description` (example `123-adding-bpel-support`).
 - Commit/PR titles: Conventional Commits.
@@ -182,3 +228,50 @@ make tools.asdf-install
 4. Run single package/test first, then chart-scoped test run.
 5. Update golden files only for intentional rendering changes.
 6. Record discoveries and remaining work in `STATE.md`.
+
+## Adding New Persistence Layers and Scenarios
+
+### New Persistence Layer
+A persistence layer is a values file at `charts/<version>/test/integration/scenarios/chart-full-setup/values/persistence/<name>.yaml`. To add one:
+
+1. Create the YAML file with the values needed for the data backend.
+2. Add the name to `validPersistence` in `scripts/camunda-core/pkg/scenarios/scenarios.go`.
+3. Update help text and shell completions in `scripts/deploy-camunda/cmd/prepare_values.go` and `scripts/deploy-camunda/cmd/root.go`.
+
+### New CI Test Scenario
+Scenarios are defined in `charts/<version>/test/ci-test-config.yaml`. Each entry specifies identity, persistence, platforms, flows, and optional features. Example:
+
+```yaml
+- name: elasticsearch-self-signed-upgrade
+  enabled: false                    # set true when ready for CI
+  identity: keycloak
+  persistence: elasticsearch-self-signed
+  platforms: [gke]
+  flows: [upgrade-minor]
+  features: [migrator]              # includes values/features/migrator.yaml
+  shortname: esss                   # 4-char, used in namespace generation
+```
+
+The `features` array maps to `values/features/<name>.yaml`. The `migrator` feature enables identity and data migration jobs during upgrades — use it for any `upgrade-minor` scenario. Note: the automatic `needsMigrator()` function in `scenarios.go` only activates when `ChartVersion` starts with "13", but the matrix runner does not set `ChartVersion`, so always use `features: [migrator]` explicitly.
+
+### Pre-Install Hooks (Scenario-Specific)
+When a scenario needs prerequisites in the namespace before `helm install` (e.g., TLS secrets), use a pre-install script:
+
+1. Create `charts/<version>/test/integration/scenarios/pre-setup-scripts/pre-install-<scenario-name>.sh`.
+2. The script receives `NAMESPACE`, `RELEASE`, and `KUBE_CONTEXT` as environment variables.
+3. Discovery is automatic: `versionmatrix.HasPreInstallScript(repoRoot, appVersion, scenario)` checks for the file.
+4. The runner calls it after namespace creation but before `helm install`, in both single-step and two-step upgrade flows.
+
+For reusable logic (e.g., cert generation), create a separate script and have the pre-install wrapper `exec` into it:
+```bash
+#!/usr/bin/env bash
+set -euo pipefail
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+exec bash "$SCRIPT_DIR/create-elasticsearch-tls-secrets.sh"
+```
+
+### Pre-Upgrade Scripts
+For cleanup needed between Step 1 (old version) and Step 2 (new version) of an upgrade flow, use `pre-upgrade-minor.sh` in the target version's `pre-setup-scripts/`. Common needs for 8.7→8.8:
+- Delete identity deployment (port naming conflict: 8.7 uses `containerPort: 8080`, 8.8 uses `8084`, both named `http` — Kubernetes strategic merge patch keeps both, causing a duplicate name error).
+- Delete stale 8.7 ingresses that route to non-existent services.
+- Delete PostgreSQL statefulsets (Bitnami version changes).

charts/camunda-platform-8.7/test/integration/scenarios/chart-full-setup/values/persistence/elasticsearch-self-signed.yaml

@@ -0,0 +1,136 @@
+# =============================================================================
+# Elasticsearch Backend with Self-Signed TLS
+# =============================================================================
+# Layer 3: Data backend - Elasticsearch (bundled, self-signed TLS)
+# Used with: --persistence elasticsearch-self-signed
+#
+# This layer enables the bundled Bitnami Elasticsearch subchart with X-Pack
+# security and TLS using pre-provisioned self-signed certificates.
+#
+# Prerequisites (must exist in the namespace before deployment):
+#   - Secret "elasticsearch-tls-certs" with keys:
+#       elasticsearch.keystore.jks, elasticsearch.truststore.jks
+#   - Secret "elasticsearch-tls-passwords" with keys:
+#       keystore-password, truststore-password
+#   - Secret "elasticsearch-jks" with keys:
+#       externaldb.jks (JKS truststore for Camunda components)
+#       truststore-password
+# =============================================================================
+#
+# Auth notes:
+#   The 8.7 templates only inject ES auth env vars when
+#   global.elasticsearch.external=true. Since we use the bundled subchart we
+#   cannot flip that flag (constraints.tpl blocks it), so we:
+#   1. Set a known elasticPassword on the subchart.
+#   2. Inject username + password via component-level env using plain values
+#      (no secretKeyRef / template expressions), which avoids the missing
+#      tpl() call in the operate init-container env rendering.
+#
+# Java 21 defaults keystore.type to pkcs12; our truststore is JKS format so we
+# must set -Djavax.net.ssl.trustStoreType=jks and provide the password.
+# The 8.7 templates prepend the trustStore path to javaOpts when
+# tls.existingSecret is set.
+# =============================================================================
+
+global:
+  elasticsearch:
+    enabled: true
+    url:
+      protocol: https
+      host: "{{ .Release.Name }}-elasticsearch"
+      port: 9200
+    auth:
+      username: elastic
+      existingSecret: "{{ .Release.Name }}-elasticsearch"
+      existingSecretKey: elasticsearch-password
+    tls:
+      enabled: true
+      existingSecret: elasticsearch-jks
+
+operate:
+  javaOpts: "-Djavax.net.ssl.trustStoreType=jks -Djavax.net.ssl.trustStorePassword=changeit"
+  env:
+    - name: CAMUNDA_OPERATE_ELASTICSEARCH_USERNAME
+      value: elastic
+    - name: CAMUNDA_OPERATE_ELASTICSEARCH_PASSWORD
+      value: "camunda-ci-es-password"
+    - name: CAMUNDA_OPERATE_ZEEBE_ELASTICSEARCH_USERNAME
+      value: elastic
+    - name: CAMUNDA_OPERATE_ZEEBE_ELASTICSEARCH_PASSWORD
+      value: "camunda-ci-es-password"
+    - name: CAMUNDA_DATABASE_USERNAME
+      value: elastic
+    - name: CAMUNDA_DATABASE_PASSWORD
+      value: "camunda-ci-es-password"
+
+optimize:
+  javaOpts: "-Djavax.net.ssl.trustStoreType=jks -Djavax.net.ssl.trustStorePassword=changeit"
+  env:
+    - name: CAMUNDA_OPTIMIZE_ELASTICSEARCH_SECURITY_USERNAME
+      value: elastic
+    - name: CAMUNDA_OPTIMIZE_ELASTICSEARCH_SSL_ENABLED
+      value: "true"
+    - name: CAMUNDA_OPTIMIZE_ELASTICSEARCH_SECURITY_PASSWORD
+      value: "camunda-ci-es-password"
+
+tasklist:
+  javaOpts: "-Djavax.net.ssl.trustStoreType=jks -Djavax.net.ssl.trustStorePassword=changeit"
+  env:
+    - name: CAMUNDA_TASKLIST_ELASTICSEARCH_USERNAME
+      value: elastic
+    - name: CAMUNDA_TASKLIST_ELASTICSEARCH_PASSWORD
+      value: "camunda-ci-es-password"
+    - name: CAMUNDA_TASKLIST_ZEEBE_ELASTICSEARCH_USERNAME
+      value: elastic
+    - name: CAMUNDA_TASKLIST_ZEEBE_ELASTICSEARCH_PASSWORD
+      value: "camunda-ci-es-password"
+    - name: CAMUNDA_DATABASE_USERNAME
+      value: elastic
+    - name: CAMUNDA_DATABASE_PASSWORD
+      value: "camunda-ci-es-password"
+
+zeebe:
+  javaOpts: "-Djavax.net.ssl.trustStoreType=jks -Djavax.net.ssl.trustStorePassword=changeit"
+  env:
+    - name: ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_AUTHENTICATION_USERNAME
+      value: elastic
+    - name: ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_AUTHENTICATION_PASSWORD
+      value: "camunda-ci-es-password"
+    - name: CAMUNDA_DATABASE_USERNAME
+      value: elastic
+    - name: CAMUNDA_DATABASE_PASSWORD
+      value: "camunda-ci-es-password"
+
+zeebeGateway:
+  env:
+    - name: CAMUNDA_DATABASE_USERNAME
+      value: elastic
+    - name: CAMUNDA_DATABASE_PASSWORD
+      value: "camunda-ci-es-password"
+
+connectors:
+  env:
+    - name: JAVA_TOOL_OPTIONS
+      value: "-Djavax.net.ssl.trustStore=/usr/local/connectors/certificates/externaldb.jks -Djavax.net.ssl.trustStoreType=jks -Djavax.net.ssl.trustStorePassword=changeit"
+
+elasticsearch:
+  enabled: true
+  security:
+    enabled: true
+    elasticPassword: "camunda-ci-es-password"
+    tls:
+      restEncryption: true
+      autoGenerated: false
+      verificationMode: certificate
+      master:
+        existingSecret: elasticsearch-tls-certs
+      passwordsSecret: elasticsearch-tls-passwords
+      keystorePassword: ""
+      truststorePassword: ""
+  # The parent chart's values.yaml defaults master.extraEnvVars to
+  # [{name: ELASTICSEARCH_ENABLE_REST_TLS, value: "false"}], which would
+  # override the security helper's "true" (from restEncryption: true above).
+  # Setting extraEnvVars to an empty array clears that default so only the
+  # security helper's correct value remains — no duplicate env var.
+  master:
+    extraEnvVars: []