Repository navigation
Description
ext.ignition.blackbox is a single kola test that runs all 1,641 subtests in
one VM. Two problems follow from that shape: the suite dies as a unit when it
runs out of memory, and every PR pays for all of it regardless of the diff.
Memory
The VM gets kola's default 1 GiB, and LUKS2's default argon2id KDF asks
cryptsetup for 1048576 kB. One luksFormat trips the global OOM killer:
oom-kill:constraint=CONSTRAINT_NONE,...,global_oom,task_memcg=/system.slice/kola-runext.service,task=cryptsetup
Out of memory: Killed process 35581 (cryptsetup) total-vm:494992kB, anon-rss:441968kB
The killed process lives inside kola-runext.service, so the Go suite dies at
once and dozens of subtests fail at 0.00s with failed: "losetup": context canceled, ending on Unhandled systemd unit state:deactivating. None of those
names mention LUKS, so it reads as device contention rather than memory
exhaustion.
It is not a parallelism problem: the guest has 1 vCPU, so -test.parallel
defaults to 1 and the subtests run serially.
#2364 raises the VM to 4 GiB (minMemory: 4096), which matches the reprovision
tests and leaves room for the ~827 MiB the OS already uses.
Still open on the memory side:
- A 4 GiB request queues on the memory-limited runner. Expect
kola: Waiting on memory to run ... need 4096 MiBbefore the test starts.
That looks like a hang and is not one. - Nothing bounds per-subtest memory, so a future test that asks for more than
the guest has reproduces the same cascade, with the same misleading names. - A crash, OOM or timeout still fails all 1,641 subtests and surfaces as a
singleext.ignition.blackboxline. The real failure has to be dug out of
the artifact tarball.
Runtime
On #2359's build the test took 4,904s, and the blackbox branch was the critical
path at 2h44m of a 3h20m build. A docs-only change pays the same cost.
#2365 adds diff-driven selection:
ci/blackbox_manifest.shrecords each registered subtest with its test
directory and config version. Names come fromregister.Testsrather than
from source, becauseRegister()fans 189 definitions across spec versions
and partitioner backends and some names are built withfmt.Sprintf.ci/blackbox_select.pymaps changed Go packages to test directories and
writes a-test.runpattern toselection.args, adding-test.shortwhen
the diff cannot affect config parsing.tests/kola/blackbox/test.shpassesselection.argsto the test binary when
the file is present. Branch builds and the nightly run have none, so they run
everything.
Measured on two real commits: a files-stage change drops from 473 to 62
subtests, disks/LUKS from 647 to 84, both roughly 3 to 4 minutes instead of 82.
Docs, dracut and _test.go-only diffs skip the branch entirely. Config,
registry and harness changes still run all 1,641.
Two decisions are deliberate. Selection fails closed: any path it cannot
classify runs the full suite, as does an unresolvable base ref. And there is no
per-version selection, because config.Parse calls the latest spec's parser,
which upgrades every config through every older translate package, so a change
under config/v3_N reaches all versions, not just N. -test.short is only
used for internal/* diffs, where the engine always operates on the latest
types.
Still open on the runtime side:
- The nightly run in
.cci.jenkinsfileneeds a real scheduled trigger to prove
it fires onmainonly. That is the only thing covering what selection
skips, since ignition has no post-merge kola job. -test.shortcurrently means "minimum-version variant only", but Skip blackbox test in short mode #1123
proposes making it skip the blackbox suite entirely. If that lands as
written, a selected run would execute zero tests and pass. We should agree on
a dedicated flag before either change lands.- The package-to-directory map is the only hand-maintained part. A
coverage-derived map would remove it: build the ignition binary with
-cover, run the suite serially with a per-subtestGOCOVERDIR, and record
which packages each subtest actually hits. - Selection granularity is the test directory, so the floor is the bucket size,
not minutes.
Related
#2364 (memory fix), #2365 (selection), #1123 (short-mode request), #728
(running blackbox tests in VMs).
Activity
prestist commentedon Nov 20, 2026
#2364 fixed the OOM cascade and the IGNCONFIG udev race: #2364
prestist commentedon Nov 20, 2026
#2365 adds the diff-driven selection: #2365