Experiment with running blackbox tests in VMs

[ajeddeloh]

Feature Request

Background

Currently the blackbox tests are somewhat terrifying, albeit fast. They run directly on the host, as root, with no isolation except that we use loopback devices. This means working on the test runner itself is particularly fragile and makes it easy to leak disk images all over /var/tmp. It also means running the tests in a container is very hard.

Proposal to experiment with

Run each test in its own (very minimal) vm via qemu (not libvirt, more on that later). The vm image would be three parts:

1. kernel: something minimal that boots to the initramfs very fast. There are a lot of tests, so speed savings here actually matter quite a bit
2. initramfs: probably generated by dracut but with custom dracut modules only. We actually don't want to switch root, we just want something to build a minimal filesystem. We don't want any other cruft that might interfere with the tests either. We basically want udev rules, systemd, something for networking (more on that later), and ignition + the tools it needs.
3. input disk images: This should be generated by the test runner and have a bare skeleton of a linux filesystem (just enough to run things like useradd with the --root flag and other things Ignition does), plus whatever input files are needed for the tests. Additionally we should have a secondary disk to attach for testing things like partitioning.

Generating the initramfs:

• This should be done in a container. This ensures we have a known set of tools (i.e. dracut + the tools it pulls in for Ignition) used to create the environment under test.
• We should have a custom set of dracut modules for Ignition for the blackbox tests that are not intended for use by distros. This ensures we don't have any distro-isms slip in and lets us drop extra complexity from things like the first boot flag file.

Running the tests

• There will need to be some test runner that generates tests, sets up misc test infra (e.g. http servers to serve test artifacts), runs the tests with qemu, and aggregates the results. Managing libvirt for this is probably more cumbersome than just using qemu by itself.
• This should be runnable from a container (assuming we have /dev/kvm mounted in). Probably the same container we use to build the initramfs

Networking

• We probably want to keep this as minimal as possible
• If we can use qemu user mode networking, that's great since it makes it easier to run in a container
• We don't need:
  • DNS (test runner can just use IPs by themselves)
  • DHCP (test runner can assign static and keep track of IPs)
  • ICMP (Ignition doesn't do anything with this)
  • Routes to the internet (tests shouldn't require internet access)
• We can probably find/write a dracut module that just reads kernel command line args and runs ip commands to set up the bare minimal networking we need. Again, I want to emphasize we should have the bare minimum to reliably reach the test infra's http/tftp servers and nothing more
• Ideally we have the network used by the test runner entirely isolated from the host

cc @arithx since we chatted about this, let me know if I missed something

[arithx]

It's worth noting that we should also refactor the existing tests to have a generalized input disk so that we can ideally use a single image created by the test runner. Ideally I think the runner should have the functionality to detect the input disks and create an image for each unique input disk so that if we need a custom input set in the future it's possible, but for an initial refactor generating a single image will likely be a sizable time saving.