Skip to content

Commit 37d1e91

Browse files
nmantheynmanthey
committed
test: add kvm-unit-tests as a functional virtualization test
Run the upstream kvm-unit-tests suite against the guest KVM on a nested-virtualization-enabled instance, as a functional check of the kernel's virtualization stack. The test runs in two stages: run-01 installs the target kernel from the pipeline's shared kernel-rpms area and reboots into it (reusing the flow from unixbench-kernel-regression), and run-02 confirms the kernel switched, then builds QEMU and runs the suite against that kernel's KVM. A hardware gate skips unsupported configurations early, QEMU is built from source because the base image ships no system emulator, and a small stable subset runs by default. Adds the vm-tests/kvm-unit-tests test directory and a c8i.4xlarge VM entry to the example config. The test artifacts are pinned so a daily pipeline run reflects kernel changes only, not test-suite drift: kvm-unit-tests is checked out at a fixed upstream revision (KVMUT_REF, default the commit validated on c8i), and the QEMU source tarball is verified against a pinned sha256 before building. Setting KVMUT_REF empty tracks the upstream default branch instead. Where these artifacts should ultimately come from -- a pinned upstream revision as here, versus a mirrored or vendored copy under our control -- is worth discussing for the merged pipeline. Signed-off-by: Norbert Manthey <nmanthey@amazon.de>
1 parent 2d3abf3 commit 37d1e91

9 files changed

Lines changed: 787 additions & 0 deletions

File tree

‎.gitignore‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,3 +60,11 @@ analysis/data/
6060
# Distribution archives
6161
share/
6262
test-nested.json
63+
64+
# kvm-unit-tests run artifacts (SSM logs + fetched sources; not source)
65+
vm-tests/*/ssm-output
66+
vm-tests/*/ssm-output-*/
67+
vm-tests/kvm-unit-tests/kvm-unit-tests/
68+
vm-tests/kvm-unit-tests/kernel_version_before.txt
69+
vm-tests/kvm-unit-tests/kvm-unit-tests.log
70+
vm-tests/kvm-unit-tests/results_kvm-unit-tests.*

‎examples/aws/config.json‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -236,6 +236,16 @@
236236
"unixbench-kernel-regression"
237237
],
238238
"min_count": 1
239+
},
240+
{
241+
"ami_id": "resolve:ssm:/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64",
242+
"instance_type": "c8i.4xlarge",
243+
"root_volume_size": 40,
244+
"max_runtime": 5400,
245+
"test": [
246+
"kvm-unit-tests"
247+
],
248+
"min_count": 1
239249
}
240250
]
241251
}

‎vm-tests/kvm-unit-tests/README.md‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
# kvm-unit-tests
2+
3+
Runs the upstream [kvm-unit-tests](https://gitlab.com/kvm-unit-tests/kvm-unit-tests)
4+
suite against the guest's KVM hypervisor. Intended to run on an EC2 instance
5+
with **nested virtualization enabled** (c8i/c7i and related Intel families,
6+
handled automatically by `_supports_nested_virtualization()` in `launch_vm.py`).
7+
8+
## What it does
9+
10+
The test runs in two stages (two `run-*.sh` scripts, with a reboot between them
11+
handled by the pipeline client):
12+
13+
**run-01 -- install the target kernel.** Records the running kernel, then
14+
installs the kernel under test from the pipeline's shared `kernel-rpms` area
15+
(reusing the flow from `unixbench-kernel-regression`) and makes it the default
16+
boot target; the client then reboots into it. If no kernel RPMs are provided,
17+
the current kernel is kept.
18+
19+
**run-02 -- run kvm-unit-tests.** After the reboot:
20+
21+
1. Confirms the running kernel is the one run-01 installed (when applicable).
22+
2. Installs the build toolchain (`git`, `gcc`, `make`, `binutils`) and builds
23+
**QEMU from source** into `/opt/qemu` (`x86_64-softmmu`, `--enable-kvm`) via
24+
`build-qemu.sh`. Amazon Linux 2023 ships no qemu system emulator in its core
25+
repos (only `qemu-img`), and its EPEL9-derived SPAL repo does not carry it
26+
either, so a source build is required. `install_qemu()` exports `QEMU` so
27+
`run_tests.sh` uses the built binary.
28+
3. Verifies `/dev/kvm` is present in the guest -- a built-in regression check on
29+
the EC2 nested-virtualization enablement itself.
30+
4. Clones and builds kvm-unit-tests (pinned to a fixed revision by default via
31+
`KVMUT_REF`; set it empty to track upstream tip).
32+
5. Runs a **curated stable subset** by default (see `KVMUT_TESTS`) with
33+
`ACCEL=kvm ./run_tests.sh -v`. Set `KVMUT_TESTS=""` for the full suite.
34+
6. Writes `results_kvm-unit-tests.txt` (human summary) and
35+
`results_kvm-unit-tests.csv` (`test_name,status,kernel_version,arch`) for
36+
KCIDB ingestion. The parser strips ANSI colour codes from run_tests.sh output
37+
before counting. Both use the `results_` prefix so `test-vm-client.sh`
38+
uploads them; the pass/fail verdict itself is conveyed by the script's exit
39+
code, which the client records as `SUCCESS`/`FAILED` in `result.txt`.
40+
41+
This is a **functional** test (pass/fail per sub-test), not a benchmark, so it
42+
produces no `benchmark-*.csv` and is not part of the performance-regression
43+
analysis.
44+
45+
## Why
46+
47+
kvm-unit-tests is the standard upstream suite for exercising a running kernel's
48+
KVM implementation. Running it continuously in nested-virtualization cloud
49+
guests gives a per-kernel regression signal for the virtualization stack, and
50+
additional test groups can be layered in later via `KVMUT_GROUP`.
51+
52+
## Validated on c8i (2026-07-10)
53+
54+
Run on a live `c8i.4xlarge` with nested virt + source-built QEMU 9.2.0:
55+
56+
- **Full suite:** 54 PASS / 10 FAIL / 23 SKIP. The 10 failures are a coherent
57+
nested-virt-sensitive class: timeouts (`xapic`, `access`, `vmx_apicv_test`,
58+
`vmx_posted_intr_test`, `vmx_pf_exception_test`), MSR/PMU emulation gaps
59+
(`msr`, `msr64`, `pmu`), and nested-VMX instability (`vmx` SIGSEGV, `la57`).
60+
- **Curated subset (default):** 6 PASS / 0 FAIL / 0 SKIP -- clean and stable,
61+
suitable as a hard gate.
62+
63+
## Two-tier model
64+
65+
- **Default = stable gate.** `KVMUT_TESTS` runs the proven-stable subset and
66+
`KVMUT_OBSERVE_ONLY=false` gates on it. A regression that breaks one of these
67+
tests fails the pipeline.
68+
- **Full suite = tracking.** Set `KVMUT_TESTS=""`. `KVMUT_IGNORE_FAILURES` is
69+
pre-seeded with the 10 nested-virt-sensitive tests above so a full run still
70+
gates on the stable core; set `KVMUT_OBSERVE_ONLY=true` to only report.
71+
72+
## Configuration (environment overrides)
73+
74+
| Variable | Default | Purpose |
75+
|---|---|---|
76+
| `KVMUT_TESTS` | `debug intel_iommu lam vmx_init_signal_test vmx_sipi_signal_test hyperv_clock` | Space-separated test names to run. Empty = full suite. |
77+
| `KVMUT_OBSERVE_ONLY` | `false` | Report counts but never fail. Set `true` for full-suite tracking runs. |
78+
| `KVMUT_IGNORE_FAILURES` | *(10 nested-virt-sensitive tests)* | Test names excluded from the pass/fail decision. |
79+
| `KVMUT_REF` | *(pinned commit `1da1819e`)* | kvm-unit-tests revision for reproducibility. Empty = upstream tip. |
80+
| `KVMUT_GROUP` | *(empty)* | Run only a specific `run_tests.sh -g` group. |
81+
| `QEMU_VERSION` | `9.2.0` | QEMU source version built by `build-qemu.sh`. |
82+
83+
## Requirements
84+
85+
- x86_64 instance from a nested-virt-capable family (e.g. `c8i.4xlarge`),
86+
32 GB+ RAM and ~40 GB disk (QEMU builds in `/tmp`, needs ~3 GB free).
87+
- Kernel RPMs for the kernel under test, supplied by the pipeline
88+
(`external_requirements.json` sets `kernel-rpms/binary: true`). Without them
89+
run-01 keeps the current kernel.

‎vm-tests/kvm-unit-tests/build-qemu.sh‎

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
#!/bin/bash
2+
3+
# Authors: Norbert Manthey <nmanthey@amazon.de>
4+
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
5+
# SPDX-License-Identifier: Apache-2.0
6+
7+
# Build QEMU from source on AL2023 (no QEMU in default repos).
8+
# Builds a minimal x86_64-softmmu target — enough for nested KVM testing.
9+
set -e
10+
exec 2>&1
11+
12+
QEMU_VERSION="${QEMU_VERSION:-9.2.0}"
13+
# Expected sha256 of qemu-${QEMU_VERSION}.tar.xz. download.qemu.org publishes
14+
# only GPG signatures, so the digest is pinned here. This value matches the
15+
# default 9.2.0 tarball; override QEMU_SHA256 when bumping QEMU_VERSION, or set
16+
# it to empty to skip the integrity check.
17+
QEMU_SHA256="${QEMU_SHA256:-f859f0bc65e1f533d040bbe8c92bcfecee5af2c921a6687c652fb44d089bd894}"
18+
QEMU_DIR="/opt/qemu"
19+
NPROC=$(nproc)
20+
21+
# Pre-flight: disk space check
22+
AVAIL_MB=$(df --output=avail -m /tmp | tail -1 | tr -d ' ')
23+
if [ "$AVAIL_MB" -lt 3072 ]; then
24+
echo "ERROR: Need ≥3GB free in /tmp for QEMU build (have ${AVAIL_MB}MB)"
25+
exit 1
26+
fi
27+
28+
if [ -x "$QEMU_DIR/bin/qemu-system-x86_64" ]; then
29+
echo "QEMU already built: $($QEMU_DIR/bin/qemu-system-x86_64 --version | head -1)"
30+
exit 0
31+
fi
32+
33+
echo "=== Installing build dependencies ==="
34+
dnf install -y gcc gcc-c++ make ninja-build python3 python3-pip \
35+
glib2-devel pixman-devel zlib-devel libfdt-devel \
36+
flex bison diffutils findutils tar gzip xz wget bzip2
37+
# meson needs a TOML parser on Python <3.11 (AL2023 ships 3.9); tomllib is only
38+
# in the 3.11+ stdlib. Prefer the distro package python3-tomli; fall back to pip
39+
# only if it is not available.
40+
dnf install -y python3-tomli 2>/dev/null || pip3 install tomli 2>/dev/null || true
41+
42+
# libslirp-devel may not be available — build from source if needed
43+
if ! dnf install -y libslirp-devel 2>/dev/null; then
44+
echo "libslirp-devel not available, building from source..."
45+
dnf install -y meson git 2>/dev/null || pip3 install meson
46+
cd /tmp
47+
wget -q "https://gitlab.freedesktop.org/slirp/libslirp/-/archive/v4.7.0/libslirp-v4.7.0.tar.gz"
48+
tar xf libslirp-v4.7.0.tar.gz
49+
cd libslirp-v4.7.0
50+
meson setup build --prefix=/usr --default-library=both
51+
ninja -C build install
52+
ldconfig
53+
cd /tmp
54+
rm -rf libslirp-v4.7.0*
55+
fi
56+
57+
echo "=== Downloading QEMU $QEMU_VERSION ==="
58+
cd /tmp
59+
wget -q "https://download.qemu.org/qemu-${QEMU_VERSION}.tar.xz"
60+
if [ -n "$QEMU_SHA256" ]; then
61+
echo "Verifying sha256 of qemu-${QEMU_VERSION}.tar.xz ..."
62+
echo "${QEMU_SHA256} qemu-${QEMU_VERSION}.tar.xz" | sha256sum -c - || {
63+
echo "ERROR: qemu-${QEMU_VERSION}.tar.xz sha256 mismatch (expected ${QEMU_SHA256})"
64+
exit 1
65+
}
66+
else
67+
echo "WARNING: QEMU_SHA256 is empty -- skipping archive integrity check"
68+
fi
69+
tar xf "qemu-${QEMU_VERSION}.tar.xz"
70+
cd "qemu-${QEMU_VERSION}"
71+
72+
echo "=== Configuring (x86_64-softmmu only) ==="
73+
./configure \
74+
--prefix="$QEMU_DIR" \
75+
--target-list=x86_64-softmmu \
76+
--enable-kvm \
77+
--enable-slirp \
78+
--disable-docs \
79+
--disable-user \
80+
--disable-gtk \
81+
--disable-sdl \
82+
--disable-opengl \
83+
--disable-virglrenderer \
84+
--disable-xen \
85+
--disable-spice \
86+
--disable-vnc \
87+
--disable-curses
88+
89+
echo "=== Building (${NPROC} jobs) ==="
90+
make -j"$NPROC"
91+
make install
92+
93+
echo "=== Cleanup ==="
94+
cd /
95+
rm -rf /tmp/qemu-${QEMU_VERSION}*
96+
97+
echo "QEMU installed: $($QEMU_DIR/bin/qemu-system-x86_64 --version | head -1)"

0 commit comments

Comments
 (0)