[testing] Migrate CI to GH Workflow

[arif-ali]

• CentOS and Fedora as docker containers.
• Fedora for stagetwo depends on py3.14 enablement for avocado.
• CentOS foreman tests to be added later, docker container and foreman
  may not behave as well.
• Tests for systemd and/or journals have been skipped for docker
  containers for the time being.
• Debian will be added later.
• 26.04 enablement is same as Fedora, plus waiting for the runners to
  be available in GH

Related: #4288

---

Please place an 'X' inside each '[]' to confirm you adhere to our Contributor Guidelines

• Is the commit message split over multiple lines and hard-wrapped at 72 characters?
• Is the subject and message clear and concise?
• Does the subject start with [plugin_name] if submitting a plugin patch or a [section_name] if part of the core sosreport code?
• Does the commit contain a Signed-off-by: First Lastname email@example.com?
• Are any related Issues or existing PRs properly referenced via a Closes (Issue) or Resolved (PR) line?
• Are all passwords or private data gathered by this PR obfuscated?

[packit-as-a-service]

Congratulations! One of the builds has completed. 🍾

You can install the built RPMs by following these steps:

• sudo dnf install -y 'dnf*-command(copr)'
• dnf copr enable packit/sosreport-sos-4310
• And now you can install the packages.

Please note that the RPMs should be used only in a testing environment.

[packit-as-a-service]

Congratulations! One of the builds has completed. 🍾

You can install the built RPMs by following these steps:

• sudo dnf install -y 'dnf*-command(copr)'
• dnf copr enable packit/sosreport-sos-4310
• And now you can install the packages.

Please note that the RPMs should be used only in a testing environment.

[pmoravec]

Looks great (as far as I my limited knowledge of GH workflows reach..)!

[pmoravec]

The failed:

(240/307) /__w/sos/sos/sos/tests/report_tests/plugin_tests/logs.py:LogsPluginTest.test_journalctl_collections:  FAIL: /var/tmp/avocado_koakdqmzsos_tests.py.LogsPluginTest/sosreport-LogsPluginTest/sosreport-52a3bbea0876-2026-05-15-ocwynko/sos_commands/logs/journalctl_--no-pager_--boot does not exist (0.60 s)

is expected on systems with empty / no journals - see https://github.com/sosreport/sos/blob/main/sos/report/plugins/logs.py#L81-L85 . The https://github.com/sosreport/sos/blob/main/tests/report_tests/plugin_tests/logs.py#L27 must be called conditionally (or never).

[arif-ali]

I am trying to work through this, as fedora and centos is container based, so figuring things out, hence still WiP. I'm conscious that we are closing towards EoL of Cirrus

[arif-ali]

if anyone knows how to get systemd working in a docker container for fedora/centos, then that would be awesome, that's the last bit to get the stage one tests working under GH workflow/actions. I have tried many things first in my own Canonical repo, and some today. I'll keep researching, and see if I can find anything, but if you already have tackled it, then that could be useful

/cc @jcastill @pmoravec @TurboTurtle

[jcastill]

First of all, thank you so much @arif-ali for doing this!
Now, images - from what I know, for centos-stream, an alternative could be to use the bootc images, that should have systemd enabled by default. The two images for centos stream should be:

quay.io/centos-bootc/centos-bootc:stream9
quay.io/centos-bootc/centos-bootc:stream10

Other than that, I think the other way would be to use a modified dockerfile and I'm not sure how feasible is that. I wonder if we could actually create sos-specific images in quay.io that are lightweight and also have systemd enabled, just for testing, but I'm not an expert in containers so maybe anybody else knows if it can be done and it's worth it?

Same for Fedora, these are the ones I've found:

quay.io/fedora/fedora-bootc:43
quay.io/fedora/fedora-bootc:44

Does that help?

[pmoravec]

No much knowledge on this topic. I can try to learn & dig in, if necessary - if so, please describe the issue more specifically.

Checking one particular error:

110-tests/report_tests/command_priority_tests.py:CommandPriorityTest.test_logs_full_journal_correct_priority: FAIL

It is because journalctl --no-pager --reverse wasnt collected, because systemd-journald service was not present (and since /var/log/journal was there, cf https://github.com/sosreport/sos/blob/main/sos/report/plugins/logs.py#L81-L85)?

AFAIK the service isnt usually present in containers. So if we want to run these journal tests inside containers, we must modify the service-enabling condition (to is_executable('journalctl')? that is weird..). Or skip the test if the service isnt present (sounds better approach to me).

[arif-ali]

Is it worth to run a quick test using the systemd enabled containers I mentioned before?

Based on the current test, the bootc image doesn't work in stageone tests, but the normal one does 🤷🏽, and this is with all the skips for systemd/journal. Something about krb5-server not being able to install

[jcastill]

What about the option of creating sos-specific centos-stream docker images in quay.io? I imagine that in that scenario we could control the dockerfile and have a combination of the base image + the bootc image (and give us more flexibility in the future if we want to add more things)

[arif-ali]

What about the option of creating sos-specific centos-stream docker images in quay.io? I imagine that in that scenario we could control the dockerfile and have a combination of the base image + the bootc image (and give us more flexibility in the future if we want to add more things)

That's an option, I think we did similar in Cirrus world way before I joined the project here.

Seems like you have to pay to get images on there, unless you have some insider thing that allows you to do this. I would be open to that, as long as other folks here would also be happy with that too.

We are so close in closing this off and getting it to work, I'll see if I can do something similar to these last failures with skipIf decorators or variables to skip these for now, and we can work on the new containers with less pressure on getting this done for the end of the month

[jcastill]

Seems like you have to pay to get images on there, unless you have some insider thing that allows you to do this. I would be open to that, as long as other folks here would also be happy with that too.

OK, let me investigate what options do we have and if there's no free tier, if we can get sponsored.

We are so close in closing this off and getting it to work, I'll see if I can do something similar to these last failures with skipIf decorators or variables to skip these for now, and we can work on the new containers with less pressure on getting this done for the end of the month

Agreed, lets finish this up before the deadline in 10 days and then we can work on making it "perfect". Once this PR is merged, I'll open a new one where we can experiment further.

Thank you again @arif-ali !

[arif-ali]

Right, this should be it now, there are issues with snapstore at the moment, but I know the ubuntu side will pass once the snapstore infra is good.

So, reviews from the team would be awesome

[pmoravec]

Comparing tests and builds triggered before and after:

• fedora RPM builds just for x86_64 arch. IMHO sufficient
• Ubuntu 22.04 and 24.04 is used instead of 26.04 (or 25.10). More tests on older version, is it pros or cons? (no strong opinion)
• foreman tested only on ubuntu deb, not on debian or snap. it would be great to have 2 distros, ideally something from Red Hat family, but I am ok with this
• what replaced "Breakage test"s?
• no stage one or two tests on debian..?

Thanks again for all the huge work, @arif-ali !

[arif-ali]

Comparing tests and builds triggered before and after:

• fedora RPM builds just for x86_64 arch. IMHO sufficient

That's what we were doing before?

• Ubuntu 22.04 and 24.04 is used instead of 26.04 (or 25.10). More tests on older version, is it pros or cons? (no strong opinion)

These are LTS releases, and GH native runners are available. 26.04 is still pending to be added by GH, so that's a waiting game.

For intermediate releases GH doesn't do runners, so we may need to do containers for those. So, some extra work is required there. It would be good to do it, but thought as they are only supported for 9 months, and the latest 25.10 is only supported for another month didn't make sense.

• foreman tested only on ubuntu deb, not on debian or snap. it would be great to have 2 distros, ideally something from Red Hat family, but I am ok with this

Totally understand, the RH and Debian both need adding here, but as these will be done in docker, this may take some time to get done, but it's in my plan to do it.

I'm away some of next week and today, and have other work on my radar, so wasn't going to have time to complete all of it

• what replaced "Breakage test"s?

The py tests are the breakage tests, which tests python 3.9 and latest 3.x

• no stage one or two tests on debian..?

Again, docker related, so need to spend extra time to fine tune it all to get these going, it's on my radar to do

Thanks again for all the huge work, @arif-ali !

No worries at all

[pmoravec]

Ack from me, thanks for all the work done.

Not sure if we should wait for @TurboTurtle review as well, to have full consensus here. As the change is quite substantial..

[pmoravec]

Comparing tests and builds triggered before and after:

• fedora RPM builds just for x86_64 arch. IMHO sufficient

That's what we were doing before?

Per some other PR and its tests, yes :) but as I wrote, I see no much benefit in those.

• Ubuntu 22.04 and 24.04 is used instead of 26.04 (or 25.10). More tests on older version, is it pros or cons? (no strong opinion)

These are LTS releases, and GH native runners are available. 26.04 is still pending to be added by GH, so that's a waiting game.

LGTM.

• foreman tested only on ubuntu deb, not on debian or snap. it would be great to have 2 distros, ideally something from Red Hat family, but I am ok with this

Totally understand, the RH and Debian both need adding here, but as these will be done in docker, this may take some time to get done, but it's in my plan to do it.

+1, plus I should have this on my radar for the foremanctl stuff as well (and feel free to point to my this promise there :) )

• what replaced "Breakage test"s?

The py tests are the breakage tests, which tests python 3.9 and latest 3.x

Ah, I was blind, thx.

[TurboTurtle]

Ack. Thanks for taking this on! This was a real curveball from Cirrus, and this is a very strong pivot.

[TurboTurtle]

Holding merge for ~24 hours for final comments given the size and scope of this.