Skip to content

Require completed checker evidence for Jepsen qualification - #15

Open
jeregrine wants to merge 4 commits into
fix-qualification-cachefrom
fix-jepsen-qualification-status
Open

Require completed checker evidence for Jepsen qualification#15
jeregrine wants to merge 4 commits into
fix-qualification-cachefrom
fix-jepsen-qualification-status

Conversation

@jeregrine

@jeregrine jeregrine commented Sep 11, 2026

Copy link
Copy Markdown
Member

Problem

Live checker qualification treated every nonzero negative-run exit as a successful rejection. Docker startup, JVM bootstrap, and timeout failures could therefore certify a checker which never completed. An acknowledged corruption request alone also does not prove its injection or intended invariant check actually ran.

Fix

Use one Elixir script for qualification orchestration: allocate fresh per-run artifacts, run the mutation baseline, launch the healthy and corrupted histories, stream their logs to disk, and validate both the process exit and the checker record. qualify.sh is only a compatibility launcher. The separate shell result helper and manual shell regression are removed; executable ExUnit regressions exercise the real runner with isolated external commands as part of the normal gate.

Internal corruption runs must provide target-specific injection completion and the exact stable invariant identifier on the same node. Arming the cursor-marker file does not count as injection: the snapshot must actually insert the marker and reject it. Terminal unavailability must retire the required target and report that target missing.

Supporting information

Stacked on #16 (fix-qualification-cache): artifact-directory ownership moves from its Bash bootstrap into the Elixir runner, and its standalone-launcher regression is retained.

Direct invocation is elixir test/jepsen/qualify.exs; existing CI and soak commands can continue using qualify.sh. The existing GNU timeout process-tree deadline and TERM/KILL grace period are preserved. The live workload still requires its existing Java, Leiningen, and Docker infrastructure.

The checker artifact is separate from human logs and scoped to one run. Missing or malformed evidence, unrelated invariant failures, failed injection, unexpected acceptance, infrastructure exit codes, and timeout exits fail qualification. Human-readable diagnostics remain available, but qualification does not match their text.

@jeregrine
jeregrine changed the base branch from main to fix-qualification-cache September 11, 2026 21:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant