[TASK] Implement Camunda 8.8 unified config with fallback mechanism

[aabouzaid]

Related to

Epic: https://github.com/camunda/team-distribution/issues/433
Related to: #2477

Overview

Camunda 8.8 will have a unified config for the core app (Zeebe, Zeebe Gateway, Operate, Identity, and Tasklist).

By today (07.05.2025), we used the old apps config for Core 8.8, and when the unified config is implemented in the app, we should reflect that in the Camunda 8.8 Helm chart (v13.x.x).

We should also implement a fallback mechanism to support 8.7 chart values in the 8.8 chart.

The refactored config has 3 groups:

1. global: No action is needed, as we will not change.
2. component level and defined in one component only (e.g., zeebe.clusterSize): it will be mapped directly to the new config (so it will be orchestration.clusterSize, and the old syntax will have higher priority if defined).
3. component level and defined in multiple components only (e.g., zeebe.retention, operate.retention, and tasklist.retention): we will review it case by case to see if we should use one of the values or mark it as a breaking change (e.g., the Operate retention probably will not be needed in 8.8 due to the arch change).

Most configurations fall under points 1 and 2, so the change should be easy for most Helm values.

Actions

• Implement the app's unified config as an extra ConfigMap - refactor: initial version of unified config #3853
• Implement the compatibility layer via Helm named functions, which will use the Helm values syntax from 8.7 if defined; if not, use the Helm values 8.8 syntax - feat: introduce orchestration cluster compatibility layer between 8.7 and 8.8 charts #3886
• Rename core to orchestration - refactor: rename core to orchestration #3935
• Get the final decision about the retention and history retention and update the OC ConfigMap - refactor: set orchestration retention and history retention #3970
• The old and new values.yaml should be documented in the official docs - docs(self-managed): helm upgrade for orchestration cluster consolidation camunda-docs#6543
• Avoid any breaking changes as much as possible in the 8.7 to 8.8 upgrade - [TASK] Set back orchestration statefulset matching to zeebe #4026
• Run Camunda Orchestrator cluster with the unified config and ensure the config is updated with the latest changes from the code side - refactor: run orchestration cluster from unified config #4138
• Fix OpenSearch auth issue - [TASK] Unified Config - Port Operate OpenSearch AWS config #4188
• Use main config for importer - refactor: use the main config in migrations #4235

[aabouzaid]

Yesterday, the Distro team had an insightful meeting with @buccarel about the new config changes in the 8.8 release.
As the dev team plans to work on the change in iterations (it's not a working version yet), the Distro team can also do the same. We'll start by mapping the changes we have in the chart, and once the app release is ready, we can add the fallback mechanism (in Helm).