[EPIC] QA / Distro Baseline Established

[hisImminence]

After the 8.8 release, we aim to strengthen our testing and feedback loop for application SNAPSHOTs and QA integration tests as part of the alwaysGreen initiative.

This issue tracks the baseline setup to ensure a baseline reliability and shared ownership between QA and Distribution.

Success Criteria

Alerting

• Automated alert in #team-distribution-alerts when any Helm Chart–related CI action fails in QA testing.
• Automated alert in the same channel when Distro nightly tests fail.
• Bonus: POC/ Spike to get visibility/ data into the CIs for tacking of stability (see Ci Analytics of infra team, infra GHAs)

CI Health & Ownership

• Helm Chart CI pipelines/GHAs are cleaned up and known refactoring/ improvements implemented.
  • @eamonnmoloney, please scope this tasks for us
  • https://github.com/camunda/team-distribution/issues/602
  • https://github.com/camunda/team-distribution/issues/616
  • [ISSUE] issues with the introduction of 8.7 e2e tests for every PR run #3809
  • [ENHANCEMENT] Add OIDC (Entra ID) Support to Helm Chart Workflows for 8.7− and 8.8+ Reusable Workflows in QA Repo #3845
  • https://github.com/camunda/team-distribution/issues/640
  • https://github.com/camunda/team-distribution/issues/613
  • [ISSUE] Strategically use [skip ci] on add-and-commit workflow tasks that you don't want CI to run #4536
  • https://github.com/camunda/c8-cross-component-e2e-tests/pull/1059 - this is the optimization PR for 8.7. It will be the base of work needed to optimize 8.8 and 8.9
  • https://github.com/camunda/team-distribution/issues/641 - this is a prerequisite for other tests. Working with the CI is slow so cutting off 2 mins per run as this will, is a big win.
• Adjust cluster scaling to support more load on the cluster for e2e tests
• Every Distro team member has signed off (understands setup, can debug independently).
• [ISSUE] issues with the introduction of 8.7 e2e tests for every PR run #3809 is resolved
• Evaluate (and if possible implement) solutions for [ENHANCEMENT] Enable Dynamic Cluster Configuration Updates for QA Automated Tests #2770

QA Sign-off on Core Scenarios

• Tasklist v1 API (ticket)
• Entra Flow (see thread)
• Migration flows (functional testing included)
• Document handling flow running and stable

[eamonnmoloney]

Can we move this into another Epic?

Evaluate (and if possible implement) solutions for #2770

I don't see this as relevant to this work

[eamonnmoloney]

Status:
I've completed https://github.com/camunda/c8-cross-component-e2e-tests/pull/1059. Waiting for QA signoff to merge
Working on externalising infra now..

This Epic is the prerequisite for the AlwaysGreen project. This ticket gets the CI into a state where it is stable, robust and maintainable. Here is the sequence of change:

Roadmap for this epic:

• Get the infra externalised with https://github.com/camunda/team-distribution/issues/641
• Setup tests for TaskList v1/v2 now that the CI is faster
• Refactor scenarios so they are layer (like Hamzas original design) https://github.com/camunda/team-distribution/issues/602
• Fix up the CI so it is easy to run locally.. its hard to debug and maintain without this https://github.com/camunda/team-distribution/issues/616
• Adjust optimize startup so it doesn't kill itself with exp backoff https://github.com/camunda/team-distribution/issues/640