[ENHANCEMENT] Enable Dynamic Cluster Configuration Updates for QA Automated Tests

[slolatte]

Describe the use case:

In QA, we often need to test specific features by enabling or disabling them dynamically during the execution of automated Playwright test runs. Examples of such features include:

• Resource-Based Authorization (RBA)
• Multi-Tenancy
• User Access Restrictions
• Updating License Keys

Currently, the only way to achieve this is by regenerating the Helm chart, which is inefficient and time-consuming. A more streamlined approach is required to support these configurations.

Describe the enhancement/feature:

We propose adding the ability to update the CI cluster dynamically via a script or API that can be triggered directly from the Playwright test suite. This script should:

1. Allow feature toggles or updates (e.g., enable/disable RBA, change tenancy settings, update license keys).
2. Apply the changes without requiring a full cluster recreation.

This would enable more efficient and flexible testing of various features and configurations in automated scenarios.

Desired outcome and acceptance tests:

Desired Outcome:

• QA engineers can dynamically update the cluster's configuration during test runs without regenerating the Helm chart.

This feature will empower the QA team to test more complex scenarios effectively and save time during test execution.

[hisImminence]

We concluded that this feature request is currently not possible for us to implement as it would require changes in the app teams to dynamically changes configurations. We will keep it in the backlog and we will come to this once we caught up on QA topics (do a more profound breakdown and think about iterations in this direction).

[hisImminence]

in the context of tasklist v1 testing, @slolatte pointed out ounce more that the scalability of the current testing approach is not sufficient - basically we would need to have 1 HC spawn up per configuration what is not scaling well - specially also thinking on enabling PRs on the monorepo side.

One idea to have one Helm Chart deployed to test several test runs might also be to setup a configuration with a mix of configurations enabled, so QA can test several single test suites against the same cluster. For example, having a helm chart setup configured with:

Taskilist V1 test

• Tasklist v1
• multi-tenancy
• rba

Taskilist V2 test

• Tasklist v2
• multi-tenancy
• rba

Internal Keycloak general

• Auth: internal keycloak
• multi-tenancy
• rba

Entra general

• Auth: entra
• multi-tenancy
• rba

...

Would that be a possible first iteration?

We could then align together between QA/ Distro what the configurations should look like/ we need.

[hisImminence]

From a quick discussion with @eamonnmoloney:

We currently have some limitations on the app side:

• MT and RBA cannot be enabled at the same time.
• Tasklist v1/v2 requires a Helm upgrade (so it’s faster to test in a separate namespace).
• Auth cannot be mixed, so separate namespaces are needed as well.

Given that, we think our focus should shift toward scalability of the cluster:

• Elasticsearch (ES) and Keycloak (KC) external:
• Use a custom realm per deployment for KC.
• Use custom prefixes for ES indices.

This setup would reduce startup time by roughly 2 minutes (≈30–40%), make better use of resources (since ES is CPU-intensive), and also avoid the current PVC/node-AZ attachment issue.
Similarly, making Postgres external would solve the same PVC problem there as well.

With the decision to start integration only for the super basic scenario and keep the nightly QA tests as they are, I’d suggest de-prioritizing this for now.

• We can continue scaling test coverage by adding new scenarios to the nightly tests.
• For now, the PR integration can just run smoketests + basic health checks, which should be sufficient.