Skip to content

tests: blackbox is one 82-minute test in one 1 GiB VM #2366

Open

Description

@prestist

ext.ignition.blackbox is a single kola test that runs all 1,641 subtests in
one VM. Two problems follow from that shape: the suite dies as a unit when it
runs out of memory, and every PR pays for all of it regardless of the diff.

Memory

The VM gets kola's default 1 GiB, and LUKS2's default argon2id KDF asks
cryptsetup for 1048576 kB. One luksFormat trips the global OOM killer:

oom-kill:constraint=CONSTRAINT_NONE,...,global_oom,task_memcg=/system.slice/kola-runext.service,task=cryptsetup
Out of memory: Killed process 35581 (cryptsetup) total-vm:494992kB, anon-rss:441968kB

The killed process lives inside kola-runext.service, so the Go suite dies at
once and dozens of subtests fail at 0.00s with failed: "losetup": context canceled, ending on Unhandled systemd unit state:deactivating. None of those
names mention LUKS, so it reads as device contention rather than memory
exhaustion.

It is not a parallelism problem: the guest has 1 vCPU, so -test.parallel
defaults to 1 and the subtests run serially.

#2364 raises the VM to 4 GiB (minMemory: 4096), which matches the reprovision
tests and leaves room for the ~827 MiB the OS already uses.

Still open on the memory side:

  • A 4 GiB request queues on the memory-limited runner. Expect
    kola: Waiting on memory to run ... need 4096 MiB before the test starts.
    That looks like a hang and is not one.
  • Nothing bounds per-subtest memory, so a future test that asks for more than
    the guest has reproduces the same cascade, with the same misleading names.
  • A crash, OOM or timeout still fails all 1,641 subtests and surfaces as a
    single ext.ignition.blackbox line. The real failure has to be dug out of
    the artifact tarball.

Runtime

On #2359's build the test took 4,904s, and the blackbox branch was the critical
path at 2h44m of a 3h20m build. A docs-only change pays the same cost.

#2365 adds diff-driven selection:

  • ci/blackbox_manifest.sh records each registered subtest with its test
    directory and config version. Names come from register.Tests rather than
    from source, because Register() fans 189 definitions across spec versions
    and partitioner backends and some names are built with fmt.Sprintf.
  • ci/blackbox_select.py maps changed Go packages to test directories and
    writes a -test.run pattern to selection.args, adding -test.short when
    the diff cannot affect config parsing.
  • tests/kola/blackbox/test.sh passes selection.args to the test binary when
    the file is present. Branch builds and the nightly run have none, so they run
    everything.

Measured on two real commits: a files-stage change drops from 473 to 62
subtests, disks/LUKS from 647 to 84, both roughly 3 to 4 minutes instead of 82.
Docs, dracut and _test.go-only diffs skip the branch entirely. Config,
registry and harness changes still run all 1,641.

Two decisions are deliberate. Selection fails closed: any path it cannot
classify runs the full suite, as does an unresolvable base ref. And there is no
per-version selection, because config.Parse calls the latest spec's parser,
which upgrades every config through every older translate package, so a change
under config/v3_N reaches all versions, not just N. -test.short is only
used for internal/* diffs, where the engine always operates on the latest
types.

Still open on the runtime side:

  • The nightly run in .cci.jenkinsfile needs a real scheduled trigger to prove
    it fires on main only. That is the only thing covering what selection
    skips, since ignition has no post-merge kola job.
  • -test.short currently means "minimum-version variant only", but Skip blackbox test in short mode #1123
    proposes making it skip the blackbox suite entirely. If that lands as
    written, a selected run would execute zero tests and pass. We should agree on
    a dedicated flag before either change lands.
  • The package-to-directory map is the only hand-maintained part. A
    coverage-derived map would remove it: build the ignition binary with
    -cover, run the suite serially with a per-subtest GOCOVERDIR, and record
    which packages each subtest actually hits.
  • Selection granularity is the test directory, so the floor is the bucket size,
    not minutes.

Related

#2364 (memory fix), #2365 (selection), #1123 (short-mode request), #728
(running blackbox tests in VMs).

Activity

  1. prestist commented on Nov 20, 2026

    @prestist
    CollaboratorAuthor

    #2364 fixed the OOM cascade and the IGNCONFIG udev race: #2364

  2. prestist commented on Nov 20, 2026

    @prestist
    CollaboratorAuthor

    #2365 adds the diff-driven selection: #2365

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    jirafor syncing to jira

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions