konflux: integrate Testing Farm for multi-arch kola testing by Roshan-R · Pull Request #4030 · coreos/fedora-coreos-config

Roshan-R · 2026-02-26T10:40:03Z

Enable execution of kola tests in Konflux through Testing Farm using the upstream tmt-via-testing-farm Tekton pipeline.

This introduces two IntegrationTestScenario resources for x86_64 and aarch64, enabling Testing Farm to provision infrastructure and run kola via tmt. Other architectures are not yet enabled due to current Testing Farm provisioning limitations

gemini-code-assist

Code Review

This pull request integrates Testing Farm for multi-arch kola testing by adding new Tekton and TMT configurations. The review has identified several important issues that need to be addressed. There are critical copy-paste errors in two shell scripts (tmt/tests/kola.sh and tmt/tests/test.sh) where diff headers have been included, which will cause them to fail. Additionally, there are high-severity concerns regarding the use of a privileged container, which is a security risk, and the hardcoding of a personal GitHub fork in the test scenarios, which impacts maintainability. Other medium-severity suggestions include verifying a potentially incorrect Fedora compose version and improving shell script robustness. Please address these points to ensure the changes are secure, correct, and maintainable.

tmt/tests/kola.sh

tmt/tests/test.sh

gemini-code-assist · 2026-02-26T10:41:20Z

.tekton/testing-devel/testing-farm.yaml

+  - name: GIT_URL
+    value: https://github.com/Roshan-R/fedora-coreos-config
+  - name: GIT_REF
+    value: kola-testing-farm


The GIT_URL is set to a personal fork and GIT_REF to a development branch. For long-term maintainability and stability, this should point to the official upstream repository and a stable reference (e.g., a tag or main branch) rather than a personal fork, especially before merging.

gemini-code-assist · 2026-02-26T10:41:20Z

.tekton/testing-devel/testing-farm.yaml

+  - name: GIT_URL
+    value: https://github.com/Roshan-R/fedora-coreos-config
+  - name: GIT_REF
+    value: kola-testing-farm


The GIT_URL is set to a personal fork and GIT_REF to a development branch. For long-term maintainability and stability, this should point to the official upstream repository and a stable reference (e.g., a tag or main branch) rather than a personal fork, especially before merging.

tmt/tests/kola.sh

.tekton/testing-devel/testing-farm.yaml

tmt/tests/test.sh

Roshan-R · 2026-02-26T10:43:46Z

/retest

Enable execution of kola tests in Konflux through Testing Farm using the upstream tmt-via-testing-farm Tekton pipeline. This introduces two IntegrationTestScenario resources for x86_64 and aarch64, enabling Testing Farm to provision infrastructure and run kola via tmt. Other architectures are not yet enabled due to current Testing Farm provisioning limitations

jbtrystram

Initial review

jbtrystram · 2026-03-02T10:35:15Z

tmt/tests/kola.sh

+cosa buildextend-qemu
+cosa kola run --rerun --allow-rerun-success=tags=needs-internet --build=latest --on-warn-failure-exit-77 --arch=$(arch) '--tag=!reprovision' --parallel=5
+mv $COSA_DIR/tmp/kola $TMT_TEST_DATA


we probably want to also test the ISO and build all the artifacts

jbtrystram · 2026-03-02T10:43:50Z

.tekton/testing-devel/testing-farm.yaml

+  - name: IMAGE_TAG
+    value: v3.16
+  - name: GIT_URL
+    value: https://github.com/Roshan-R/fedora-coreos-config


Update to coreos org

jbtrystram · 2026-03-02T10:44:14Z

.tekton/testing-devel/testing-farm.yaml

+  - name: IMAGE_TAG
+    value: v3.16
+  - name: GIT_URL
+    value: https://github.com/Roshan-R/fedora-coreos-config


update to coreOS org

jbtrystram · 2026-03-02T10:45:57Z

tmt/plans/kola.fmf

+/qemu:
+  summary: QEMU tests
+  tag: qemu
+  environment+:
+    PLATFORM: qemu


Let's split that in /build-qemu and /test-qemu at least

jbtrystram · 2026-03-02T10:46:31Z

tmt/tests/kola.sh

+    env | grep --color=auto COREOS_ASSEMBLER;
+    local -r COREOS_ASSEMBLER_CONTAINER_LATEST="quay.io/coreos-assembler/coreos-assembler:latest";
+    set -x;
+    podman run --rm -ti --security-opt=label=disable --privileged --uidmap=1000:0:1 --uidmap=0:1:1000 --uidmap=1001:1001:64536 -v=\${COSA_DIR}:/srv/ --device=/dev/kvm --device=/dev/fuse --tmpfs=/tmp -v=/var/tmp:/var/tmp --name=cosa \${COREOS_ASSEMBLER_CONFIG_GIT:+-v=\$COREOS_ASSEMBLER_CONFIG_GIT:/srv/src/config/:ro} \${COREOS_ASSEMBLER_GIT:+-v=\$COREOS_ASSEMBLER_GIT/src/:/usr/lib/coreos-assembler/:ro} \${COREOS_ASSEMBLER_ADD_CERTS:+-v=/etc/pki/ca-trust:/etc/pki/ca-trust:ro} \${COREOS_ASSEMBLER_CONTAINER_RUNTIME_ARGS} \${COREOS_ASSEMBLER_CONTAINER:-\$COREOS_ASSEMBLER_CONTAINER_LATEST} "\$@";


we can drop the customization env vars here

jbtrystram · 2026-03-02T10:47:00Z

tmt/tests/kola.sh

+pushd ${COSA_DIR}/src/config
+git checkout $CONFIG_COMMIT
+popd
+cosa buildextend-qemu


Suggested change

cosa buildextend-qemu

cosa osbuild qemu

Roshan-R · 2026-03-05T07:14:56Z

/retest

Roshan-R · 2026-03-09T09:18:06Z

/retest

Roshan-R · 2026-03-12T08:01:56Z

/retest

Roshan-R · 2026-03-12T11:10:52Z

/retest

HuijingHei · 2026-03-23T13:12:55Z

tmt/tests/kola.fmf

+  environment:
+    TEST_CASE: test-qemu
+
+/build-iso:


Maybe we can put build-iso and test-iso in another VM to make the running time shorter?

gemini-code-assist bot reviewed Feb 26, 2026

View reviewed changes

Roshan-R force-pushed the testing-farm-kola branch from 6e776e7 to 52b58ca Compare March 2, 2026 08:19

Roshan-R force-pushed the testing-farm-kola branch from 52b58ca to 61f312b Compare March 2, 2026 09:10

jbtrystram requested changes Mar 2, 2026

View reviewed changes

tmt: split qemu build and kola test into two tests

48e20b5

Roshan-R force-pushed the testing-farm-kola branch from f9b2765 to 42b66b7 Compare March 4, 2026 13:02

tmt: fix git ref

900f4a4

Roshan-R force-pushed the testing-farm-kola branch 6 times, most recently from fca7f79 to f4d9f74 Compare March 4, 2026 20:34

tmt: fix cosa command

051c986

Roshan-R force-pushed the testing-farm-kola branch from f4d9f74 to 051c986 Compare March 4, 2026 21:13

Roshan-R force-pushed the testing-farm-kola branch 3 times, most recently from 06c8300 to da3f958 Compare March 5, 2026 14:53

tmt: use GIT_URL and GIT_COMMIT to get which repo to init

dd6a865

Roshan-R force-pushed the testing-farm-kola branch from da3f958 to dd6a865 Compare March 6, 2026 10:44

tmt: add testiso support

56f002b

Roshan-R force-pushed the testing-farm-kola branch from e3ac542 to 56f002b Compare March 6, 2026 12:54

Roshan-R added 3 commits March 9, 2026 16:46

cosa: remove unwanted params to podman

5148221

temp-commit: remove set -eo pipefail to testiso.sh to anaylse error

68fd398

tmt: split testiso.sh to build-iso.sh and test-iso.sh

382a84b

Roshan-R force-pushed the testing-farm-kola branch from edd4824 to cdb149d Compare March 9, 2026 13:27

tests: add --rm to podman command

25be9f8

Roshan-R force-pushed the testing-farm-kola branch 2 times, most recently from 57b9b52 to 55ad4fa Compare March 10, 2026 09:00

tmt: add init test

0ff4256

Roshan-R force-pushed the testing-farm-kola branch from 55ad4fa to 0ff4256 Compare March 10, 2026 10:04

tmt: mark git safe directory

9326d84

Roshan-R force-pushed the testing-farm-kola branch from 10fe25f to 9326d84 Compare March 10, 2026 12:58

tmt(test-iso): add --inst-insecure to skip signature verification

83befdd

Roshan-R added 9 commits March 14, 2026 06:15

tmt: add trap to tests to upload kola artifacts if tests fail

1f26d3a

tmt: add run_kola bash func for better kola integration

5d2647d

tmt(fix): add cd to run_kola

0e72aac

tmt(fix): fix kola command EXTRA_ARGS

610beb7

tmt: exit on first test fail

9568370

tmt: cosa command add --userns=keep-id:uid=1000,gid=1000

3efe01d

tmt: remove exit-first temporarily

fa71ba6

tmt: add cosa shell -- to mktemp

19c15dc

tmt: mkdir -p TMT_TEST_DATA

5bebef6

HuijingHei reviewed Mar 23, 2026

View reviewed changes

Conversation

Roshan-R commented Feb 26, 2026

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

gemini-code-assist bot Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Roshan-R commented Feb 26, 2026

Uh oh!

jbtrystram left a comment

Choose a reason for hiding this comment

Uh oh!

jbtrystram Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

jbtrystram Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

jbtrystram Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

jbtrystram Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

jbtrystram Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

jbtrystram Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

Roshan-R commented Mar 5, 2026

Uh oh!

Roshan-R commented Mar 9, 2026

Uh oh!

Roshan-R commented Mar 12, 2026

Uh oh!

Roshan-R commented Mar 12, 2026

Uh oh!

HuijingHei Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants