CI: KubeVirt provider e2e on GitHub Actions (k3s single-node + KVM)

[tjjh89017]

Motivation

The scaffolded e2e suite (test/e2e/) only verifies that the manager deploys onto a kind cluster. The KubeVirt provider's actual data path — SPDY exec into the virt-launcher compute container → virsh qemu-agent-command → QGA guest-exec of the rendered vbash script — has zero automated coverage. Today it is validated manually against Harvester HCI and Proxmox VE before releases, so provider regressions (QGA command changes, exec plumbing, readiness/status parsing) can only be caught by hand.

Goal: a real KubeVirt e2e job in GitHub Actions that boots an actual VM, round-trips QGA, and exercises the full CheckReady → ExecScript → GetExecStatus path. KubeVirt provider only; Proxmox/daemon providers are out of scope.

Feasibility findings

KVM is available on free ubuntu-latest runners. Since April 2024 GitHub exposes /dev/kvm on standard Linux runners (2-vCPU and up); public repos get 4 vCPU / 16 GB RAM standard runners at no cost. Access requires a one-line udev rule (the same snippet the Android emulator ecosystem uses). Caveat: officially it is only documented for the Android emulator use case — generic libvirt/KubeVirt workloads have no SLA (community discussion #8305), so the workflow must detect /dev/kvm and fall back to useEmulation: true (TCG) rather than hard-depend on it.

k3s single-node instead of kind. Most known KubeVirt-in-CI failures are kind-specific: the node is a container, so virt-handler fights an extra layer for /dev/kvm device ownership (kubevirt#8949), hits cgroup-nesting issues (kubevirt#4861), and has a history of CrashLoopBackOff in kind (kubevirt#7463). k3s runs as a systemd service on the runner host, so the node is the real machine: /dev/kvm is exposed directly by the kubelet device plugin and cgroups are managed normally. k3s itself runs its e2e on GitHub Actions, and Harvester is RKE2 + KubeVirt, so the Rancher-family + KubeVirt combination is production-proven. RKE2 would give exact Harvester parity but costs ~2–4 extra minutes of startup and ~1 GB more memory per job with no meaningful behavioral difference for what this suite tests; k3s is the better CI tradeoff.

Guest image: real VyOS userland via DozenOS — a community 1:1 rebuild of VyOS rolling with freely downloadable nightly releases (generic/KVM ISO + KVM qcow2, minisign-signed, rebuilt on upstream changes with a weekly heartbeat). The provider needs three things from the guest (internal/provider/kubevirt/provider.go, internal/provider/qga/):

1. qemu-guest-agent responding (guest-ping),
2. systemctl show -p SubState --value vyos-router.service reporting the unit ran to completion,
3. /bin/vbash to execute the script written to /tmp/vrouter-apply.sh (script re-execs via sg vyattacfg).

A real VyOS-family image satisfies all three natively and makes the commit/save semantics of the rendered vbash script real instead of stubbed — the suite then covers the full contract, not just the operator-side pipeline. Plan:

• Disk delivery, two options: (a) wrap the release qcow2 into a containerDisk in-workflow (FROM scratch + ADD dozenos-<version>-kvm-amd64.qcow2 /disk/, import into the node containerd) — no extra cluster components; or (b) install CDI and use a DataVolume with source.http.url pointing straight at the release qcow2 URL — no image build at all, automatic qcow2 conversion, and a PVC-backed VM closer to how Harvester runs VMs in production, at the cost of installing CDI and its images. containerDisk is the default plan; switch to CDI if the build/import step turns out slower than the CDI install.
• Guest OS flavor detection (prerequisite operator change): DozenOS ships the router unit as dozenos-router.service (rebranded from vyos-router.service), but CheckReady hardcodes qga.VyOSService. Design: read /etc/os-release via QGA and key off the ID field — VyOS reports ID=vyos, DozenOS reports ID=dozenos. Map the ID through a small OS-flavor interface that answers flavor-specific questions (today just the router service unit name: vyos → vyos-router.service, dozenos → dozenos-router.service; future forks or flavor-specific behaviors slot in as new implementations). A wrong unit name fails loudly (CheckReady reports not ready (substate=dead)), so no defensive plumbing is needed beyond the interface itself. This detection lands as a small operator change before the e2e suite work.
• Always test against the latest release: each run resolves https://github.com/dozenos/dozenos-nightly-build/releases/latest/download/version.json (also at repo root on main), which lists the current version plus per-flavor artifact URLs, sha256, and minisig URLs; download the KVM qcow2 it points at and verify against its recorded sha256. Tracking latest is deliberate — it doubles as early warning that upstream rolling changes broke the operator's guest contract.
• Confirm qemu-guest-agent is active on first boot in the KVM image (production KubeVirt setups already rely on QGA with VyOS; if it needs enabling, do it via cloud-init/config on day 0).
• Fallback if image size/boot time ever becomes a CI bottleneck: a generic cloud containerdisk (e.g. quay.io/containerdisks/fedora) + cloud-init stubs (vyattacfg group, /bin/vbash → /bin/bash symlink, stub vyos-router.service) still exercises the operator↔QGA pipeline end to end. The stub unit must be Type=oneshot + RemainAfterExit=yes + ExecStart=/bin/true: CheckReady requires SubState to be exactly exited (internal/provider/kubevirt/provider.go:125 — vyos-router is a oneshot whose readiness means "ran to completion"). Verified on systemd: oneshot with RemainAfterExit=yes reports exited; without it the unit reports dead, and a long-running ExecStart (e.g. sleep infinity) reports running — both fail CheckReady forever.

Upstream VyOS images themselves are not CI-usable (1.4+ LTS requires a paid subscription; official rolling nightlies are an unpinned moving target), which is exactly the gap DozenOS fills. Full-fidelity testing on Harvester/Proxmox (a self-hosted runner via ARC + kubevirt-actions-runner) remains a possible future upgrade path.

Proposed workflow shape

e2e-kubevirt:
  runs-on: ubuntu-latest
  timeout-minutes: 30
  steps:
    - name: Enable KVM access
      run: |
        echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' \
          | sudo tee /etc/udev/rules.d/99-kvm4all.rules
        sudo udevadm control --reload-rules
        sudo udevadm trigger --name-match=kvm
    - name: Raise inotify limits            # known kind/k3s + KubeVirt failure mode
      run: |
        sudo sysctl fs.inotify.max_user_watches=1048576
        sudo sysctl fs.inotify.max_user_instances=8192
    - name: Install k3s (pinned)
      run: |
        curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.33.x+k3s1 \
          sh -s - --write-kubeconfig-mode 644     # default kubeconfig is root-only
        echo "KUBECONFIG=/etc/rancher/k3s/k3s.yaml" >> "$GITHUB_ENV"
    - name: Install KubeVirt (pinned)
      run: |
        kubectl apply -f .../v1.x.y/kubevirt-operator.yaml
        kubectl apply -f .../v1.x.y/kubevirt-cr.yaml
        # if /dev/kvm is missing, patch useEmulation=true before waiting
        kubectl -n kubevirt wait kubevirt/kubevirt --for=condition=Available --timeout=10m
    # build operator image, `docker save | sudo k3s ctr images import -`,
    # install cert-manager + CRDs, deploy manager, run the ginkgo e2e suite

Notes:

• Pin the k3s and KubeVirt versions; bump deliberately.
• KVM detection + useEmulation fallback keeps the same workflow runnable anywhere (including if GitHub ever withdraws /dev/kvm); TCG boots take 5–6 min, which the 30 min budget absorbs.
• Start as a separate non-required job (or nightly / label-triggered) until flake rate is known, then promote to PR-blocking.

Implementation plan

1. Probe workflow first (one commit, throwaway or kept as a canary): udev rule → k3s → KubeVirt → boot the DozenOS qcow2 as a VMI → kubectl wait vmi --for=condition=AgentConnected, then dump /etc/os-release and systemctl show -p SubState dozenos-router.service via QGA. One run settles every environmental unknown: KVM on the runner, qemu-guest-agent active on first boot, and the exact ID/unit-name values the flavor interface will encode.
2. Adapt the e2e scaffold: Makefile setup-test-e2e variant for k3s; replace utils.LoadImageToKindClusterWithName with a containerd import for k3s; keep cert-manager handling as is.
3. KubeVirt provider e2e spec: build the DozenOS containerDisk (latest via version.json), create a VirtualMachine from it + a VRouter CR referencing it; assert the Applied condition goes true and status.observedGeneration advances; verify the applied config is actually present in the guest (e.g. show configuration via QGA) — with a real VyOS userland this asserts commit semantics, not just script exit codes. Also cover the failure path on the same VM: apply a config that fails commit and assert phase=Failed with the Applied condition False (and no auto-retry, per SPEC §7.2) — exit-code/status parsing is half the provider surface and the part manual testing skips most.
4. Promote to required check once stable across a week or two of runs.

Risks / mitigations

Risk	Mitigation
/dev/kvm on hosted runners is undocumented beyond Android	Detect at runtime, fall back to useEmulation: true (slower but functional)
TCG mode is slow (5–6 min boot) and has quirks (e.g. reboot kubevirt#6885)	Only a fallback; suite avoids VM reboot scenarios; generous timeouts
inotify exhaustion kills pods	sysctl bump step (documented kind/minikube+KubeVirt failure mode)
Runner disk pressure from KubeVirt images + the DozenOS qcow2	Public-repo runners have headroom; download per run (GitHub Releases is fast from GHA — caching by version is pointless when tracking a nightly latest); if disk pressure hits, prune /opt/hostedtoolcache first
Tracking latest rolling — guest changes can break CI independently of code changes	Accepted by design (early-warning signal); sha256 from version.json guards download integrity; when triaging, re-run against the previous release to separate "guest changed" from "operator regressed"
DozenOS availability/continuity (community project)	Fallback path documented: generic cloud containerdisk + cloud-init stubs keeps the pipeline covered if releases stop

References

• GitHub changelog: hardware-accelerated virtualization on 2-vCPU Linux runners
• KubeVirt: software emulation fallback
• KubeVirt containerdisk images / quay.io/containerdisks
• k3s e2e on GitHub Actions
• KubeVirt on k3s single node walkthrough

[tjjh89017]

Updated the guest-image plan: use DozenOS (community 1:1 rebuild of VyOS rolling, freely downloadable nightly KVM qcow2, minisign-signed) as the CI guest instead of a stubbed cloud image — the suite then exercises real vbash/commit semantics, not just the operator-side pipeline. Pin a release tag and bump deliberately; the cloud-init stub image remains documented as a fallback.