[TT-17706] Add reusable action for distro tests and new OS

[konrad-sol]

Jira Ticket

Add RHEL7 and Ubuntu 26.04 to distro matrix; add upgrade-tests reusable workflow

Adds ubuntu:26.04 to the deb distro list. Adds a new rpm_amd64 output to the distro-matrix action for images that only support linux/amd64 (currently UBI7/RHEL7), keeping them separate from the main rpm list so existing pipelines are unaffected.

Introduces .github/workflows/upgrade-tests.yml, a reusable workflow that consolidates the upgrade smoke test logic (upgrade-deb, upgrade-rpm, upgrade-rpm-amd64) in one place. Future distro changes only require a PR here rather than a gromit template change.
TT-17706

Description

Type of Change

• Bug fix
• New feature / action
• Refactor / improvement
• Documentation update
• CI/CD / workflow change
• Other (please describe):

Changes Made

Testing

• Manually triggered the affected workflow(s) and verified expected behaviour
• Checked that existing workflows are not broken

Checklist

• My changes follow the existing conventions in this repo
• I have updated relevant documentation (e.g. README.md, action description fields)
• For changed shell scripts (if applicable): I ran shellcheck, used an appropriate shebang and error handling, and preserved required executable permissions
• For changed actions/scripts (if applicable): I added or updated validation, tests, or clear manual verification steps
• For changed JavaScript files (if applicable): I ran the relevant tests and linting/formatting checks
• For changed Python files (if applicable): I ran the relevant tests and linting/formatting checks
• For Dockerfile changes (if applicable): I reviewed the base image, build context, image size, and runtime security
• For changed actions (if applicable): I updated the action.yml interface, defaults, outputs, and examples as needed
• I reviewed security implications, including least-privilege permissions and safe handling of inputs, secrets, and tokens (if applicable)
• For workflow changes (if applicable): I reviewed triggers, permissions, concurrency, and fork safety
• I have assigned a reviewer

[probelabs]

This PR expands the CI testing matrix to include Ubuntu 26.04 and RHEL 7, and introduces a new reusable workflow to standardize package upgrade testing.

Files Changed Analysis

• .github/actions/tests/distro-matrix/action.yaml: Modified to add ubuntu:26.04 to the Debian-based distribution list and introduces a new rpm_amd64 output for amd64-only images, which now includes registry.access.redhat.com/ubi7/ubi (RHEL 7). This expands the test matrix for workflows using this action.
• .github/workflows/upgrade-tests.yml: A new, reusable workflow is added to automate package upgrade testing. This workflow builds Docker images for each specified distribution, installs a previous package version from Packagecloud, and then upgrades it using a newly built package artifact. It supports both Debian and RPM packages across amd64 and arm64 architectures, with a dedicated job for amd64-only RPM distributions.

Architecture & Impact Assessment

• What this PR accomplishes: It enhances CI validation by testing against a future Ubuntu LTS release and a legacy RHEL 7 environment. More significantly, it introduces a dedicated, reusable workflow (upgrade-tests.yml) to ensure that package upgrades are tested systematically and consistently, reducing code duplication in CI pipelines.
• Key technical changes introduced:
  • The distro-matrix action now has a new rpm_amd64 output to separate amd64-only RPM distributions.
  • A new, parameterized workflow (upgrade-tests.yml) encapsulates the logic for upgrade testing within dynamically generated Docker containers, promoting reusability.
• Affected system components: Any CI workflow that calls the .github/actions/tests/distro-matrix action will be affected. The new upgrade-tests.yml workflow is now available for integration into package-building CI pipelines.

graph TD
    subgraph sg1 [Calling Workflow]
        A[Build Package Artifacts] --> B("Call upgrade-tests.yml");
    end

    subgraph sg2 ["upgrade-tests.yml Workflow"]
        direction LR
        B -- inputs --> C{Matrix Jobs};
        C -- "distro, arch" --> D[Generate Dockerfile];
        D --> E[Build & Test in Docker];
    end

    subgraph sg3 ["Docker Build & Test Steps"]
        direction TB
        F("FROM base_distro_image") --> G[Install old package version];
        G --> H[Upgrade with new package artifact];
        H --> I[Run API tests];
    end

    A -.-> H

Loading

Scope Discovery & Context Expansion

• The primary contribution of this PR is the new upgrade-tests.yml workflow, which refactors CI testing logic. The title focuses only on the expansion of the distribution matrix.
• The new workflow is not yet called by any existing workflow in this repository. Its introduction is a foundational step, and subsequent PRs will be required to integrate it into the relevant package-building pipelines.
• To understand the full impact of the distro-matrix change, a search for usages of .github/actions/tests/distro-matrix across the organization's repositories would be necessary to identify all affected CI pipelines.

Metadata

• Review Effort: 3 / 5
• Primary Label: chore

---

Powered by Visor from Probelabs

Last updated: 2026-07-07T17:43:32.523Z | Triggered by: pr_updated | Commit: 0c22142

💡 TIP: You can chat with Visor using /visor ask <your question>

[probelabs]

Security Issues (3)

Severity	Location	Issue
🔴 Critical	.github/workflows/upgrade-tests.yml:113-114	Workflow inputs such as `inputs.package_name` and `needs.setup.outputs.from_ver` are directly substituted into shell commands within a dynamically generated Dockerfile. An attacker who can control these inputs could inject arbitrary shell commands, leading to code execution within the Docker build environment on the CI runner. For example, an `inputs.package_name` of `foo; rm -rf /` would result in the execution of the `rm -rf /` command. 💡 Suggestion Sanitize all inputs that are used to construct shell commands. Add a validation step at the beginning of each job to check for and reject inputs containing shell metacharacters (e.g., `;`, `&`, `|`, `(`, `)`, `` ` ``). This will prevent the injection of unintended commands.
🟠 Error	.github/workflows/upgrade-tests.yml:112	The workflow executes a remote script by piping the output of `curl` directly into `bash`. This is a significant security risk, as it allows for arbitrary code execution if the remote server is compromised or if a man-in-the-middle attack occurs. The comment `# SECURITY: accepted risk` indicates awareness, but this pattern should be avoided as it creates a supply chain vulnerability. 💡 Suggestion The remote script should be downloaded to a file, and its integrity should be verified before execution. This can be done by checking its SHA256 checksum against a known-good, pre-defined value. ```yaml # Example remediation RUN curl -fsSL https://.../script.deb.sh -o setup.sh && \ echo "EXPECTED_CHECKSUM setup.sh" | sha256sum -c - && \ bash ./setup.sh ```
🟡 Warning	.github/workflows/upgrade-tests.yml:181	The `rpm -Uvh --force` command is used to install the package. The `--force` flag can be dangerous as it instructs RPM to install the package even if it means overwriting files from other packages or installing a package with an older version number. This can mask underlying packaging issues. 💡 Suggestion Remove the `--force` flag unless it is absolutely necessary. If it is required, add a comment explaining why it's needed for this specific test scenario.

Architecture Issues (1)

Severity	Location	Issue
🟠 Error	.github/workflows/upgrade-tests.yml:203-249	The `upgrade-rpm-amd64` job (lines 203-249) is almost a complete duplicate of the `upgrade-rpm` job (lines 140-201). This introduces significant code duplication, violating the DRY (Don't Repeat Yourself) principle. Maintaining two nearly identical blocks of code for multi-arch and amd64-only RPM builds increases maintenance overhead and the risk of introducing inconsistencies when updating the test logic. 💡 Suggestion Refactor the common steps into a reusable composite action. Both `upgrade-rpm` and `upgrade-rpm-amd64` jobs can then call this action, passing parameters to handle the minor differences (like the target architecture and the `--platform` flag in the Dockerfile's `FROM` instruction). This will centralize the logic, improve maintainability, and make the workflow more robust.

Performance Issues (2)

Severity	Location	Issue
🟡 Warning	.github/workflows/upgrade-tests.yml:105-112	The Dockerfile is generated in an order that prevents effective layer caching. The `COPY` command, which introduces the frequently changing package artifact, is placed before `RUN` commands that install stable dependencies. This invalidates the cache for these subsequent layers on every workflow run, causing dependencies to be re-downloaded and re-installed in every job. This significantly increases build time and network I/O across all matrix jobs. 💡 Suggestion To leverage Docker layer caching more effectively, reorder the Dockerfile generation steps to place the `COPY` command as late as possible. Steps that install system dependencies, configure repositories, and install the previous version of the package should come before copying the new package artifact. This issue also applies to the `upgrade-rpm` job at line 172 and the `upgrade-rpm-amd64` job at line 236.
🟡 Warning	.github/workflows/upgrade-tests.yml:136-138	The workflow uses a fixed `sleep 2` to wait for the gateway container to become ready before running API tests. This approach is inefficient and unreliable. If the service starts faster, the workflow waits unnecessarily. If the service takes longer to start (e.g., on a loaded runner), the tests will fail. The same pattern is repeated in the `upgrade-rpm` job at line 202. 💡 Suggestion Replace the fixed `sleep` with a polling mechanism that repeatedly checks a health endpoint until the service is confirmed to be ready. This would make the tests faster and more robust. For example, use a loop with `curl` to check the gateway's health or status endpoint before proceeding with the tests.

Quality Issues (5)

Severity	Location	Issue
🟠 Error	.github/workflows/upgrade-tests.yml:182	The workflow attempts to use `matrix.ro`, which is not defined in the strategy matrix. This will cause the step to fail. The intended variable is likely `matrix.distro`. 💡 Suggestion Replace `matrix.ro` with `matrix.distro` to correctly reference the distribution from the matrix. 🔧 Suggested Fix docker run -d -p8080:8080 --name=test --platform linux/${{ matrix.arch }} --network ${{ job.container.network }} --rm test-${{ matrix.distro }}-${{ matrix.arch }}
🟡 Warning	.github/workflows/upgrade-tests.yml:120	The use of fixed `sleep 2` intervals to wait for services to start can lead to flaky tests. This pattern is repeated on lines 122, 183, and 185. The service may not be ready within 2 seconds on a loaded runner, causing the test to fail intermittently. 💡 Suggestion Replace `sleep` with a more robust mechanism, such as a loop that polls a health check endpoint until the service reports it is ready.
🟡 Warning	.github/workflows/upgrade-tests.yml:147-220	The `generate dockerfile` step is duplicated between the `upgrade-rpm` (lines 147-158) and `upgrade-rpm-amd64` (lines 209-220) jobs. This duplication increases maintenance overhead, as any changes to the Dockerfile generation logic must be applied in multiple places. 💡 Suggestion To reduce duplication, extract the Dockerfile generation logic into a reusable composite action or a script that can be called from both jobs.
🟡 Warning	.github/workflows/upgrade-tests.yml:158	The command `rpm -Uvh --force` is used to install the package (also on line 220). The `--force` flag can mask underlying installation problems by overwriting files and ignoring dependencies. This can lead to a seemingly successful installation that is actually broken, making test results unreliable. 💡 Suggestion Remove the `--force` flag. If it is required to work around a specific, known issue, add a comment explaining why it is necessary.
🟡 Warning	.github/workflows/upgrade-tests.yml:96	The workflow executes a script downloaded via `curl` directly with `bash` (also on lines 154 and 216). This is a security risk, as a compromise of the script's source could lead to arbitrary code execution in the CI environment. While this risk is acknowledged with a TODO comment, it remains a significant vulnerability. 💡 Suggestion Instead of piping `curl` to `bash`, download the script to a file, verify its checksum (e.g., SHA256) against a known value, and only then execute it. This ensures the integrity of the script.

---

Powered by Visor from Probelabs

Last updated: 2026-07-07T17:43:16.446Z | Triggered by: pr_updated | Commit: 0c22142

💡 TIP: You can chat with Visor using /visor ask <your question>