Discuss the environment combinations handling

[psss]

As agreed on the hacking session, let's dedicate some limited time (e.g. one or two hours) to a discussion about the proposed and brainstormed options for handling various environment combinations. Let's put together all approaches suggested so far and identify essential advantages and disadvantages for all of them.

The Problem

Users need to run the same tests across multiple environment combinations (distros, architectures, configurations) but tmt has no first-class mechanism to express this.

What is possible

• Parameterize Tests using fmf inheritance and manual --environment iterations
• Anchors and Aliases help to prevent metadata duplication across the tree branches

Why it matters

• Usability ... Environment combination testing is a fundamental workflow. Not supporting it natively means every user reinvents the wheel.
• Scalability ... Manual plan duplication breaks down quickly. Testing across N distros × M arches × K configs means N×M×K nearly-identical plans.
• Nested execution ... Issue Support for running tmt in tmt #3298 describes the related need for running tmt within tmt — e.g., provisioning a bare-metal hypervisor, then spawning VMs as guests — with aggregated results from child runs. The combination problem and the nesting problem share a need for orchestrating multiple environments and collecting unified results.

Alternatives

Here is the list of the alternatives brainstormed so far.

Matrix Plan Shaper

Implementation draft already proposed in the pull request below. Similar to how --max-tests-per-plan or importing multiple remote plans is working.

• Add matrix plan shaper for expanding plans across variable combinations #5049

Jinja Templates

Using templates and iteration across the desired values together with pre-commit. A tangible example linked below.

• https://github.com/thrix/nix-toolbox/blob/main/tmt/build.fmf.j2

Dedicated Command

A separate tmt go command which would generate the combinations, execute tmt run for each environment combinations and aggregate the results from individual runs at the end to perform a nice overview of the whole execution.

• Support for running tmt in tmt #3298

External Tools

Use external tools like newa to take care of the environment combination handling.

• Users have to learn and set up a new tool, not possible to just enter the git repo and tmt run

Related

• Parallelize container install jobs #5076
• Document possible ways to parameterize tests #3873
• Introduce a new test attribute hardware #765
• TT-311 ... As a tester I want to generate tmt plans for relevant combinations of environment dimensions

[jstodola]

We could use such a feature in the future for some of our installation test cases, for example running a test case on multiple architectures, each time with a different RAID chunk size or RAID level.

[happz]

Another related issue: #765 - if implemented, plans would/could split into smaller subsets of tests that share the same HW requirements.

[psss]

Some food for thought from AI:

Environment Combinations — Approaches Comparison

1. Matrix Plan Shaper (PR #5049)

A native matrix: key in plan metadata. tmt expands the cartesian product at runtime using the existing plan shaper plugin mechanism (plan.derive_plan()), injecting TMT_MATRIX_* environment variables into each derived plan.

Advantages:

• Native tmt feature — users write .fmf, run tmt run, no extra tools or steps
• Builds on the proven plan shaper infrastructure (same pattern as --repeat and --max-tests-per-plan)
• Metadata stays in .fmf files, visible to tmt lint, tmt plans show, etc.
• Low learning curve — the matrix: syntax is familiar from GitHub Actions
• Community-contributed implementation already exists with tests and docs

Disadvantages:

• Limited expressiveness — cartesian product only; include/exclude overrides (like GitHub Actions) add complexity fast (comment)
• Variable substitution pitfalls — ensuring $TMT_MATRIX_* is expanded in all relevant fields (provision image, environment, context) is non-trivial and has historically been a source of bugs
• Dynamically generated plan IDs are hard to track in external systems like Testing Farm (comment)
• Only one shaper can apply per plan — a matrix shaper cannot compose with --max-tests-per-plan or --repeat on the same plan
• Secrets handling is undefined — matrix values end up in plan names and environment variables, which may leak sensitive data

2. Jinja Templates

Write .fmf.j2 template files with Jinja loops/conditionals. A pre-commit hook or external tool (e.g., fmf-jinja) renders them into plain .fmf files before tmt sees them. Tangible example: nix-toolbox/build.fmf.j2.

Advantages:

• Full power of a real templating language — loops, conditionals, filters, includes, inheritance — no feature gaps to fill on tmt's side
• Generated .fmf files are plain metadata — all tmt tooling (lint, show, run) works unchanged
• No new tmt concepts or code to maintain
• Already demonstrated in practice
• Can express arbitrary structures, not just cartesian products

Disadvantages:

• Adds a generation step — metadata must be rendered before tmt run, breaking the "just clone and run" workflow
• Source of truth is .j2 files, but tmt operates on generated .fmf files — a disconnect that can cause confusion (stale generated files, forgetting to re-render)
• Users must learn Jinja syntax on top of fmf/YAML
• Generated files either get committed (noise in the repo, merge conflicts) or .gitignored (invisible to reviewers, CI must run the render step)
• No native tmt integration — tmt plans show on the template is meaningless; linting requires rendering first

3. Dedicated tmt go Command (#3298)

A new top-level command that generates environment combinations from a declared matrix, spawns a separate tmt run for each, and aggregates the results.

Advantages:

• Clean separation — combination logic is decoupled from plan metadata and the run pipeline
• Each tmt run is a fully independent, self-contained execution — easy to parallelize, retry individually, or distribute across machines
• Natural fit for nested execution (running tmt-in-tmt on a hypervisor, then aggregating child results)
• Plan IDs remain stable — each run uses a concrete, static plan, not a dynamically generated one
• Could integrate with external schedulers (Testing Farm, CI systems) by emitting individual run specifications

Disadvantages:

• Significant new command and concept — design surface is large (how to declare dimensions, how to aggregate results, how to handle partial failures)
• Combination metadata lives outside the plan, so a plan alone does not describe what environments it targets
• Result aggregation from independent runs is non-trivial — different run directories, separate logs, no shared state
• Doesn't compose with existing tmt run options transparently — users must learn when to use tmt run vs. tmt go
• No implementation exists yet — this is the highest-effort option

4. External Tools (e.g., newa)

Use a wrapper tool that handles combination generation, scheduling, and result aggregation outside of tmt.

Advantages:

• Already exists and works today — no tmt changes needed
• Handles concerns beyond combination (scheduling, reporting, CI integration) that tmt intentionally does not cover
• Can evolve independently of tmt's release cycle
• Appropriate for complex orchestration scenarios (Testing Farm integration, Errata workflows)

Disadvantages:

• Breaks the "enter repo, tmt run" experience — users must install, learn, and configure a separate tool
• Combination logic is invisible to tmt — tmt plans show and tmt lint cannot validate or display the intended matrix
• Ties users to a specific ecosystem (newa targets Testing Farm; other environments need different wrappers)
• Duplication of concerns — the external tool must understand tmt's plan structure to parameterize it correctly
• Not viable for users who run tmt locally or in generic CI without Testing Farm