ci: rework testing farm to use cloud takeover, not libvirt

[cgwalters]

Today our packit/tmt/testing-farm code ends up using the bootc provisioner which uses libvirt. In some cases this is convenient.

However, I think we should try to actually switch over to a replace workflow by default. This would have more impact on the tmt side I think, but it should dramatically improve the reliability and flexibility of our CI because we'll drop out bare metal (or at least nested virt) requirements.

(The nested virt is a whole separate thing to try to get into testing farm)

[cgwalters]

@henrywang can you summarize the issues seen with trying to do tmt with this flow?

[cgwalters]

OK wow I just found out that Github Actions has supported /dev/kvm in the default runners for literally years

[henrywang]

I'm trying this workflow: request image mode TF runner with CentOS-Stream-9-image-mode compose. And build integration test image inside this runner, then bootc switch to this new image, then run integration test.

In this case, our test does not need nested virtualization TF runner any more.

[henrywang]

Here's my bootc install on TMT package mode VM result: henrywang/tmt-bootc-install-switch#1

And bootc switch works in TMT. I mean deploy image mode VM with TMT -> build image -> bootc switch.

[henrywang]

Okey, commit henrywang/tmt-bootc-install-switch@6ced2bd fixed issue.

I'll run this kind of test on QE CI for stability.

[henrywang]

Now running on Testing Farm works. TF artifacts: https://artifacts.osci.redhat.com/testing-farm/d88be6ff-2281-4f49-9cdf-e8dc7a62fde9/

[cgwalters]

Now running on Testing Farm works. TF artifacts: https://artifacts.osci.redhat.com/testing-farm/d88be6ff-2281-4f49-9cdf-e8dc7a62fde9/

Awesome!

So as an incremental first step, WDYT about changing the tmt tests in this repo to do this?

[henrywang]

I group bootc install, upgrade, switch tests together as base line test with this new method. Things looks great! Test runs much faster than before. https://artifacts.osci.redhat.com/testing-farm/c5ec3337-af4d-46fa-93b3-d689623c933f/.

Yeah, I sent a MR https://gitlab.com/fedora/bootc/tests/bootc-workflow-test/-/merge_requests/601 to add this method in QE CI to try its stability. Let's run it this week. I'll update this repo next week if everything works well.

It's near GA, so I don't want to break things due to this change.

[cgwalters]

At least as far as this upstream git repo, I think it's a fine time to make changes; in practice we'll probably need to cherry-pick changes to any frozen downstreams anyways.