[ENHANCEMENT] Support topologySpreadConstraints

[ManuelDittmar]

Epic: https://github.com/camunda/team-distribution/issues/433

Describe the use case:

In our deployment of Zeebe, it is crucial to ensure high availability and fault tolerance. Our infrastructure spans multiple Availability Zones (AZs) to minimize the risk of downtime due to a single point of failure. By having Zeebe pods spread across multiple AZs, we can ensure that the failure of an entire AZ will not affect the overall availability and reliability of our Zeebe deployment. This setup is essential for maintaining continuous business operations and achieving our service level agreements (SLAs).

Describe the enhancement/feature:

We propose an enhancement to the Zeebe Helm charts that allows the configuration of topologySpreadConstraints. This feature will enable us to specify constraints ensuring that Zeebe pods are evenly distributed across multiple AZs.

Desired outcome and acceptance tests:

Zeebe pods are evenly distributed across all available AZs, ensuring that no single AZ failure can cause a complete service outage.

[aabouzaid]

@ManuelDittmar, just to ensure that this request is for Multi-Availability Zones and is not related to Multi-region, right?

[ManuelDittmar]

@aabouzaid yes exactly :)

[leiicamundi]

We identified a similar / related issue : https://github.com/camunda/team-infrastructure-experience/issues/408 ( Clarify Multi-AZ Support and Improve Pod Distribution in Helm Chart)

[aabouzaid]

@theburi we can easily implement topologySpreadConstraints in all Camunda components and give the customers the option to use it (it's not enabled by default).

[hisImminence]

Should be resolved once https://github.com/camunda/product-hub/issues/2697 is implemented.

[jessesimpson36]

in https://github.com/camunda/product-hub/issues/2697 , it was proposed that we document how to use kustomize or post-renderers to add arbitrary configurations that aren't directly supported by the helm chart. So I don't think it's fair to say that topologySpreadConstraints will be solved by that anymore, because this issue is requesting for something directly supported in our helm charts for topologySpreadConstraints.