ci: adding nightly integration tests

[hamza-m-masood]

Which problem does the PR fix?

closes: https://github.com/camunda/team-distribution/issues/390

What's in this PR?

Adding nightly integration tests so the multitenancy scenario can run.

This workflow does the following steps:

1. Generate a matrix file for the scenarios mentioned in ci-test-config.yaml. This file is included in each Helm chart folder
2. This matrix is used in test-chart-version-nightly.yaml. This workflow calls test-chart-version-nightly-template.yaml for each version.
3. The workflow in test-chart-version-nightly.yaml loops through all the scenarios of a specific version and calls test-integration-template.yaml to run all enabled scenarios for each version.

The values.yaml files for each scenario is in the same folder as the default values.yaml used for our integration testing.

This is an example of a successful run:
https://github.com/camunda/camunda-platform-helm/actions/runs/14607701741

Checklist

Please make sure to follow our Contributing Guide.

Before opening the PR:

• In the repo's root dir, run make go.update-golden-only.
• There is no other open pull request for the same update/change.
• Tests for charts are added (if needed).
• In-repo documentation are updated (if needed).

After opening the PR:

• Did you sign our CLA (Contributor License Agreement)? It will show once you open the PR.
• Did all checks/tests pass in the PR?

[hamza-m-masood]

I am trying to think of a good solution to run multiple nighly integration tests where a specific scenario is supported by a differing range of versions. For example, multitenancy is supported for version 8.3+, and oidc is supported for version 8.5+

This creates a "version vs. scenario" compatibility puzzle.
I see two routes:

1. I could create separate matrix files. Eac file would represent a scenario:

{
  "versions": [
    "8.3",
    "8.4",
    "8.5",
    "8.6",
    "8.7",
    "8.8"
  ],
  "scenario-file":  "values-integration-test-multitenancy.yaml"
}

{
  "versions": [
    "8.5",
    "8.6",
    "8.7",
    "8.8"
  ],
  "scenario-file": "values-integration-test-oidc.yaml"
}

The downside to this is that we have a lot of matrix files and we have to hardcode the versions, since they are different for each scenario. The benefit is that it makes running these matrix files very easy.

2. I could have one big matrix file and add a bunch of if conditions in the workflow file to check for unsupported versions.
   Here is what the matrix file would look like:

{
  "versions": [
    "8.2",
    "8.3",
    "8.4",
    "8.5",
    "8.6",
    "8.7",
    "8.8"
  ],
  "multitenancy-values-file":  "values-integration-test-multitenancy.yaml",
  "oidc-values-file": "values-integration-test-oidc.yaml"
}

Here is what the if conditions in the workflow would look like:

on:
...

jobs:
...

    steps:
      - name: Check compatibility
        run: |
          if [[ "${{ matrix.scenario.name }}" == "multitenancy" && "${{ matrix.version }}" < "8.3" ]]; then
            echo "Skipping multitenancy for version ${{ matrix.version }}."
            exit 0
          fi
          if [[ "${{ matrix.scenario.name }}" == "oidc" && "${{ matrix.version }}" < "8.5" ]]; then
            echo "Skipping oidc for version ${{ matrix.version }}."
            exit 0
          fi

      - name: Run Integration Test
        ...

The downside is that the if conditions are going to get very messy with more scenarios. The upside is that the versions won't be hardcoded, and there is only one matrix file.

---

Overall, I am in favor of the first option. Even though there are going to be more matrix files, at least the workflow file will be easier to maintain.

[eamonnmoloney]

Generally looks good.. A question I'm left with is, these look very similar to the standard test workflows. Is there some refactoring work that could/should be done in another ticket to reuse some of the code?

[aabouzaid]

I've left a couple of comments.
Also, please use the team convention for comments and style.

I will make another review once the comments are addressed so it's easier to review.

[aabouzaid]

No need to make anything for versions that reached the end of the standard support (8.2, 8.3).

[hamza-m-masood]

Ah okay, Nice!

So that means I would need an if statement in my generate matrix workflow to not pick up versions 8.3 and below 🤔

[hamza-m-masood]

Or maybe modify the ls command to list out versions after 8.3

[hamza-m-masood]

this is working but it's not that pretty:

ls -d camunda-platform-8.* | sort -V | awk -F'-' '$3 > 8.3'

I guess I can include it and write a comment above explaining what it does

[hamza-m-masood]

done

[aabouzaid]

Nice way to make it, but let's make it as an env var at the workflow top-level (later we can make it a file), just to avoid forgetting it in the future.

[hamza-m-masood]

I created the env var and and added the min supported version as an input so it is mentioned at the top level workflow.

[aabouzaid]

Nice work @hamza-m-masood 👏
I left some tiny comments, but I approve to merge after fixing them 🙌

[hamza-m-masood]

I rebased

[hamza-m-masood]

locally squashed all commits