Skip to content

README.md

Latest commit

 

History

History
288 lines (246 loc) · 17 KB

README.md

File metadata and controls

288 lines (246 loc) · 17 KB
Raw

Live cross-architecture validation (tests/vm)

tests/vm/run boots a real, publicly-fetchable kernel under qemu-system, captures the kernel's ground-truth text base, runs the cross-built kasld against the running kernel, and checks the invariant the tool rests on:

truth ∈ [virt_image_base_min, virt_image_base_max]

It is the cross-architecture, end-to-end counterpart of the single-host check extra/collect + extra/validate-bundle, and of the offline tests/replay corpus. Where tests/replay proves kasld parses and runs over captured fixtures, this proves the inferred window contains the real base on a live kernel, across architectures and attacker profiles.

Table of Contents

Prerequisites

  • make cross already run, with the same cross toolchains on PATH (see the project docs). The static per-arch build/<triple>/kasld binaries and the matching <triple>-gcc (used here to build the tiny init) are both required.
  • qemu-system-<arch> on PATH. On Debian/Ubuntu: apt install qemu-system-x86 qemu-system-arm qemu-system-misc.
  • curl, cpio, gzip.

No root is required, and nothing on the host is modified — each arch runs in a throwaway VM.

Usage

Cells are named <arch>-<distro>-<kernel maj.minor> (e.g. aarch64-alpine-6.12, aarch64-mainline-7.0); run with no cell lists them.

tests/vm/run                          # every supported cell, default profile
tests/vm/run aarch64-alpine-6.12      # one cell
tests/vm/run aarch64-alpine-6.12 hardened
tests/vm/run all hardened             # every cell in one profile
tests/vm/run table                    # results matrix + speculative-narrowing table
tests/vm/run chart                    # results matrix as an SVG range chart (stdout)
tests/vm/run spec-table               # only the speculative-narrowing table
tests/vm/run spec-table --with-timing # ...also listing timing/side-channel rows
tests/vm/run aarch64-alpine-6.12 capture # build a truth-bearing fixture from a live boot
tests/vm/run fetch-modules            # stage a distro cell's own modules (see below)

Every module-region component reads an empty /proc/modules on a cell with no module loaded, which is indistinguishable from a component that does not work, so each cell carries two real in-tree modules (dummy, veth) that init loads before the run. How they are staged depends on the kernel's origin:

  • distro cells (Alpine, Debian): tests/vm/run fetch-modules [cell ...] takes the modules from the same published build the kernel came from — the netboot modloop or the linux-lts apk — so vermagic matches by construction. It reads the release from the artefact and refuses when it disagrees with the staged kernel, since the mirror moves independently of the cache.
  • mainline cells (tests/vm/build-kernel): the build forces MODULES/DUMMY/ VETH modular and stages them beside the kernel, so any rebuilt cell comes back with modules. tests/vm/stage-modules <cell> rebuilds them against an existing .config without re-staging the kernel — usable only when the build tree still matches the staged image (it checks, and refuses otherwise).

The capture mode is a maintainer workflow, not a validation profile: it boots the kernel as root with kptr_restrict=0, frames the real /proc + /sys + /boot fact-set (kallsyms landmarks + iomem = ground truth) back over the serial console, and reconstructs a fixture under tests/fixtures/<arch>/<host>/ — host identity (CPU brand, build tag, device-tree serial/MAC) scrubbed. Those fixtures are then checked offline by make test-fixtures (see docs/testing.md).

Each run prints a per-arch verdict and a summary; the exit status is non-zero if any arch produced an unsound or incomplete result. After running the scenarios, tests/vm/run table reads the boot logs and emits a markdown matrix (arch × scenario → KASLR / virt residual / phys residual; soundness is a gate, not a column), then a speculative-narrowing table showing any cells where the likely best-guess window beats the guaranteed one and what signal drove it — the published tables in docs/reproducibility.md are generated this way.

tests/vm/run chart renders the same rows as an SVG range chart, one row per architecture, written to stdout — the committed copy is docs/diagrams/entropy-by-arch.svg. It reads cmd_table's output rather than the boot logs, so the chart and the matrix cannot disagree.

The speculative-narrowing table excludes microarchitectural side-channel narrowings (method:timing — cache/speculation oracles such as prefetch and entrybleed) by default: their success depends on the host CPU and varies from run to run, so a table listing them would not be reproducible. This is not a claim that they fail — they work, and can recover more than the guaranteed matrix shows (e.g. pinning a hardened base to exact where no file-derived leak can); they are withheld only for reproducibility. --with-timing (or SPEC_INCLUDE_TIMING=1) lists them. The guaranteed results matrix is resolved only at the sound floor and never depends on timing regardless.

aarch64     PASS  truth=0xffff800080000000 ∈ [0x...,0x...] recovered=Y
x86_64      PASS  truth=0xffffffff88000000 ∈ [0x...81000000,0x...bd000000] recovered=N

recovered=Y means the window collapsed to the exact base; recovered=N means it is wider but still contains the truth — the correct outcome under restriction. N/A means the boot produced no comparable window/truth (not a failure).

Profiles

The analysis always runs unprivileged (uid 1000, with all supplementary groups dropped) — the threat model is an unprivileged local attacker — so the modes differ only in the sysctl vector applied, never in the reader's identity. The one privileged step is the per-boot ground-truth capture the check compares against.

default is the kernel's own compile-time sysctl posture, read back at boot and left as booted (kptr_restrict=0 and perf_event_paranoid=2 upstream; dmesg_restrict whatever the .config sets — 0 on the mainline builds, 1 on Alpine). Each other mode moves exactly one axis off that baseline. A key consequence: kptr_restrict=0 alone does not expose symbol values — at the upstream perf_event_paranoid=2, /proc/kallsyms is zeroed for an unprivileged reader (kallsyms_show_value() needs perf<=1 or CAP_SYSLOG), so on a stock kernel the base comes from inference, not the symbol table.

mode sysctls (at uid 1000) what it exercises
default booted compile-time defaults (kptr=0, perf=2; dmesg per .config) a stock kernel — kallsyms zeroed by perf=2, so inference alone
kptr-hidden default + kptr_restrict=2 pointers hidden; isolates kptr's effect (perf already gates kallsyms)
perf-open default + perf_event_paranoid=0 perf relaxed — unlocks /proc/kallsyms and the perf_event_open text-poke leak (exact)
dmesg-open default + dmesg_restrict=0 world-readable dmesg (differs from default only where the kernel ships dmesg_restrict=1)
bpf-open default + unprivileged_bpf_disabled=0 unprivileged bpf() permitted (the BPF verifier-log leaks). Every cell boots with it disabled — distro cells ship CONFIG_BPF_UNPRIV_DEFAULT_OFF, mainline cells are built with it forced on — so this is a real posture change everywhere. Shown on every cell like the other sysctl profiles, including where it reads the same as default — that is a result, not an absence. On the mainline cells it yields nothing because the verifier-log offset table is uname-keyed and holds no locally built kernel, not because bpf() was already permitted
hardened kptr=2 + dmesg_restrict=1 + perf=3 + unprivileged_bpf_disabled=2 the realistic unprivileged floor (file-derived facts only)
nokaslr nokaslr on the cmdline the KASLR-disabled pin
no5lvl no5lvl on the cmdline (x86_64, riscv64) the kernel's fallback paging level when 5-level is disabled
no4lvl no4lvl on the cmdline (riscv64) a third paging level from one image — a distinct PAGE_OFFSET
la57 -cpu max under -accel tcg (x86_64) 5-level paging emulated where the host lacks la57; the only mode that varies QEMU, not the cmdline — slow (TCG)

no5lvl/no4lvl/la57 resolve a different PAGE_OFFSET from the same kernel image by changing the paging level, so one build exercises several linear-map bases. la57 forces TCG because KVM cannot expose a CPU feature the host lacks.

Architectures

Alpine kernels cover the arches it ports (below); arches it doesn't are built from kernel.org by tests/vm/build-kernel (see "Gap architectures"). Kernels come from a netboot image where one exists, otherwise the linux-lts apk (flavor=apk). An arch is skipped (not failed) when its qemu-system-* or its static build/<prefix>-*/kasld is unavailable.

arch kernel source qemu
x86_64 netboot virt qemu-system-x86_64
i686 netboot lts qemu-system-i386
aarch64 netboot virt qemu-system-aarch64
armv7 netboot lts qemu-system-arm
riscv64 linux-lts apk qemu-system-riscv64
loongarch64 linux-lts apk qemu-system-loongarch64 + UEFI firmware
ppc64le netboot lts qemu-system-ppc64
s390x netboot lts qemu-system-s390x

All eight have been verified PASS. ppc64le/s390x need qemu-system-misc (or any qemu with those targets on PATH). loongarch64 needs an edk2 firmware image: the recipe auto-discovers one next to the qemu binary (pc-bios/) or in the usual share dirs, or set LOONGARCH_BIOS explicitly — and the firmware must match the qemu that loads it.

Gap architectures (built from kernel.org)

Alpine has no port for some arches; their kernel is built from source by tests/vm/build-kernel — a pinned kernel.org tarball + a stock upstream defconfig + fixed config overlays (endianness where the byte order differs from the base defconfig, devtmpfs for an init console, and RANDOMIZE_BASE for riscv64 so its text KASLR is compiled in). The result is staged into the cache and booted by tests/vm/run with the same init.c as the Alpine flavors. Reproducible but slow; run it once per arch, manually. The arch-gated rule logic is covered per-push by tests/test-cross.

tests/vm/build-kernel mipsel-mainline-7.0  # download source + cross-build -> cache (slow)
tests/vm/run mipsel-mainline-7.0           # boot it, verdict
arch kernel-ARCH / defconfig qemu
mips mips / malta_defconfig + BE qemu-system-mips -M malta
mipsel mips / malta_defconfig (LE) qemu-system-mipsel -M malta
mips64el mips / malta_defconfig + 64-bit (LE) qemu-system-mips64el -M malta
riscv32 riscv / defconfig + 32-bit.config qemu-system-riscv32 -M virt
ppc32 powerpc / mpc85xx_defconfig (BE) qemu-system-ppc -M ppce500
powerpc64 powerpc / ppc64_defconfig (BE) qemu-system-ppc64 -M pseries
armeb arm / multi_v7_defconfig + BE qemu-system-arm -M virt

Validation status of the gap arches (built fresh from kernel.org, booted here):

  • mips, mipsel, mips64el, riscv32 — verified end-to-end, boots PASS. malta_defconfig is little-endian, so mips exercises the big-endian overlay (and mipsel boots the native byte order); mips64el promotes the same board to a 64-bit CPU (MIPS64R2-generic); riscv32 is staged as the flat Image (the virt board rejects the raw vmlinux ELF) and needs the 32-bit OpenSBI firmware (auto-discovered, see below).
  • ppc32 — the Freescale e500v2 target on qemu's generic -M ppce500 board (qemu-system-ppc from the qemu-system-misc/-ppc package, console ttyS0 on an 8250 UART). Built from mpc85xx_defconfig with text KASLR enabled (see "Gap architectures"), so it boots PASS and randomizes for real. pmac/g3beige (book3s32) is not used: its kernel has no text KASLR, so it could only report the disabled-base pin.
  • powerpc64 — boots PASS on -M pseries (power9, console hvc0), but the board delivers no KASLR seed, so the kernel boots unrandomized; the base is pinned via the disabled-base path, not a KASLR defeat.
  • armeb — verified end-to-end on -M virt, boots PASS. It needs -mbe8 on both payloads. A big-endian arm kernel runs its userspace BE8 (byte-invariant) from ARMv6 on, while make cross's only big-endian arm toolchain (armeb-linux-musleabi) emits BE32 (word-invariant) unless asked: a BE32 binary faults on its first instruction there, and since the faulting binary is init, the kernel panics killing pid 1 and the run reads as a boot failure rather than a build one. -mbe8 produces BE8 instead; the Makefile passes it for the armeb-* triple when building kasld, and run passes it when building the init it stages alongside. Under qemu-user a BE32 binary runs either way, which is why only a full VM boot surfaces the difference.

These rows skip cleanly in tests/vm/run until build-kernel populates the cache, so the Alpine arches are unaffected. Stock upstream defconfigs are used throughout; fall back to a Buildroot qemu_* defconfig if a vanilla one won't boot.

The pinned source is current mainline (7.0): it builds clean with a C23-default compiler (gcc 15), unlike the 6.12 LTS tree whose pre-6.13 MIPS vdso clashed with true/false/bool becoming keywords. riscv32 has no standalone rv32_defconfig on this tree (removed after 6.15); it is configured from defconfig plus the 32-bit.config fragment. Override the version with LINUX_VERSION if needed.

Two kernel versions per arch (the -mainline cells)

Each arch Alpine ships gets a second cell, <arch>-mainline-<version>, that boots a mainline (kernel.org) build of the same arch instead of the Alpine distro kernel — so the matrix carries an older distro kernel and current mainline per arch, the axis that surfaced most historical soundness bugs (VA-layout floors, module-region size, TEXT_OFFSET). The Alpine cell (<arch>-alpine-<version>) and the mainline cell (<arch>-mainline-<version>) render under the same arch in run table, distinguished by the source and release columns. Build and boot a mainline cell like any gap arch:

tests/vm/build-kernel aarch64-mainline-7.0  # mainline build -> cache (slow)
tests/vm/run aarch64-mainline-7.0           # boot it, verdict

Wired for x86_64, aarch64, armv7, i686, s390x, riscv64, ppc64le. ppc64le has no ppc64le_defconfig on this tree, so it builds ppc64_defconfig with the little-endian overlay (CPU_LITTLE_ENDIAN, which selects the ELF v2 ABI). The mainline cells are built on request, never part of the default gap set.

Notes and limitations

  • Soundness, not tightness: the check is that the truth is inside the window, not how small the window is. A wider-but-sound window under hardened is the expected result.
  • One stock kernel config per arch. Config-gated paths (VA-bits, endianness, VMSPLIT, CONFIG_* toggles) need purpose-built kernels and are out of scope here.
  • loongarch64 boots via UEFI; the firmware is auto-discovered next to the qemu binary or set via LOONGARCH_BIOS.
  • riscv text KASLR needs both a kernel built with RANDOMIZE_BASE and a boot-supplied seed, and qemu -M virt provides neither by default (its DTB carries only rng-seed, not kaslr-seed, and the cells run without -cpu max so there is no Zkr self-seed). For the cells actually built with it — the mainline riscv64 6.6/7.0 kernels — run dumps qemu's generated DTB, splices a fresh per-boot /chosen/kaslr-seed, and boots from the patched blob (dtc required), so the base is randomized like the other 64-bit arches. This is gated on the staged .config having CONFIG_RANDOMIZE_BASE=y, so the Alpine riscv64 kernel (not built with it) and 5.15 (predates riscv KASLR, added in 5.18) stay seedless — KASLR off, the disabled-base pin, still a soundness point. Other seedless arches (ppc64le on -M pseries, etc.) likewise land on the disabled-base pin.
  • riscv32 needs 32-bit OpenSBI, which most qemu builds do not bundle (only the riscv64 image). It is auto-discovered next to the qemu binary, in the system share dir, or in the distro cross package (/usr/lib/riscv32-linux-gnu/opensbi/generic/fw_dynamic.bin), or set via RISCV32_BIOS.
  • Useful overrides: QEMU_DIR (qemu not on PATH), ALPINE_VER, BUILD_DIR, TIMEOUT, LOONGARCH_BIOS, RISCV32_BIOS, and LINUX_VERSION (for build-kernel).