feat(eks): Deploy ECK operator for EKS dual-region

[marcel-dias]

feat(eks): Deploy ECK operator for EKS dual-region

references: https://github.com/camunda/team-infrastructure-experience/issues/1026

This pull request refactors the dual-region AWS EKS test and deployment workflow to use the ECK (Elasticsearch Operator) for managing Elasticsearch clusters, replacing the previous Helm-based approach. The changes affect CI workflow triggers, environment variables, deployment/cleanup steps, Helm values, procedure scripts, and test helpers to align with the new ECK-based Elasticsearch management. This improves modularity, cluster lifecycle management, and backup/restore reliability.

ECK Operator integration and workflow updates:

• The .github/workflows/aws_kubernetes_eks_dual_region_tests.yml workflow now triggers on changes to ECK operator-related files and directories, and adds deployment and cleanup steps for Elasticsearch via ECK in both London and Paris clusters. It also sets new environment variables for namespaces and release names, and updates job configurations to use these consistently. [1] [2] [3] [4] [5] [6] [7]

Helm values and exporter configuration changes:

• The Helm values file camunda-values.yml removes the embedded Elasticsearch configuration, adds documentation for ECK-managed Elasticsearch, and updates Zeebe exporter URLs to point to ECK-managed services. This ensures Camunda components connect to the correct Elasticsearch endpoints. [1] [2] [3] [4]

Procedure and secret management improvements:

• The create_elasticsearch_secrets.sh script is reworked to create ECK-compatible secrets for S3 snapshot/backup credentials, aligning environment variable usage and secret structure with ECK requirements. [1] [2]

Test and helper updates for ECK:

• Test fixtures and helpers are refactored to remove Helm-based Elasticsearch tolerations/affinities, and update pod/service names to ECK conventions. Helper functions now use environment-configurable pod/service names for exec operations and cross-region URLs, improving test flexibility. [1] [2] [3] [4] [5]

Script and exporter URL generation alignment:

• The generate_zeebe_helm_values.sh script is updated to generate Zeebe exporter URLs using ECK-managed service names, ensuring consistency with the new deployment model. [1] [2]

[github-actions]

🔀 Skip Workflow Checklist

Check the boxes below to skip specific workflows for this PR.

Global options

• skip_all - Skip all workflows

Individual workflows

• skip_aws_cognito_daily_cleanup
• skip_aws_common_procedure_s3_bucket
• skip_aws_compute_ec2_single_region_daily_cleanup
• skip_aws_compute_ec2_single_region_golden
• skip_aws_compute_ec2_single_region_tests
• skip_aws_ecs_single_region_fargate_daily_cleanup
• skip_aws_ecs_single_region_fargate_golden
• skip_aws_ecs_single_region_fargate_tests
• skip_aws_eks_single_region_daily_cleanup
• skip_aws_kubernetes_eks_dual_region_daily_cleanup
• skip_aws_kubernetes_eks_dual_region_golden
• skip_aws_kubernetes_eks_dual_region_teleport_tests
• skip_aws_kubernetes_eks_dual_region_tests
• skip_aws_kubernetes_eks_single_region_golden
• skip_aws_kubernetes_eks_single_region_tests
• skip_aws_modules_eks_rds_os_create_destruct_tests
• skip_aws_modules_eks_rds_os_daily_cleanup
• skip_aws_modules_eks_rds_os_tests
• skip_aws_openshift_rosa_hcp_dual_region_golden
• skip_aws_openshift_rosa_hcp_dual_region_tests
• skip_aws_openshift_rosa_hcp_single_region_golden
• skip_aws_openshift_rosa_hcp_single_region_tests
• skip_aws_rosa_hcp_dual_region_daily_cleanup
• skip_aws_rosa_hcp_single_region_daily_cleanup
• skip_azure_aks_single_region_daily_cleanup
• skip_azure_common_procedure_storageaccount_test
• skip_azure_kubernetes_aks_single_region_golden
• skip_azure_kubernetes_aks_single_region_tests
• skip_generic_kubernetes_operator_based_test
• skip_internal_global_pr_todo_checker
• skip_local_kubernetes_kind_single_region_tests

---

This checklist is automatically managed. Checked items will skip the corresponding workflow.