Require completed checker evidence for Jepsen qualification - #15
Open
jeregrine wants to merge 4 commits into
Open
Require completed checker evidence for Jepsen qualification#15jeregrine wants to merge 4 commits into
jeregrine wants to merge 4 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Live checker qualification treated every nonzero negative-run exit as a successful rejection. Docker startup, JVM bootstrap, and timeout failures could therefore certify a checker which never completed. An acknowledged corruption request alone also does not prove its injection or intended invariant check actually ran.
Fix
Use one Elixir script for qualification orchestration: allocate fresh per-run artifacts, run the mutation baseline, launch the healthy and corrupted histories, stream their logs to disk, and validate both the process exit and the checker record.
qualify.shis only a compatibility launcher. The separate shell result helper and manual shell regression are removed; executable ExUnit regressions exercise the real runner with isolated external commands as part of the normal gate.Internal corruption runs must provide target-specific injection completion and the exact stable invariant identifier on the same node. Arming the cursor-marker file does not count as injection: the snapshot must actually insert the marker and reject it. Terminal unavailability must retire the required target and report that target missing.
Supporting information
Stacked on #16 (
fix-qualification-cache): artifact-directory ownership moves from its Bash bootstrap into the Elixir runner, and its standalone-launcher regression is retained.Direct invocation is
elixir test/jepsen/qualify.exs; existing CI and soak commands can continue usingqualify.sh. The existing GNUtimeoutprocess-tree deadline and TERM/KILL grace period are preserved. The live workload still requires its existing Java, Leiningen, and Docker infrastructure.The checker artifact is separate from human logs and scoped to one run. Missing or malformed evidence, unrelated invariant failures, failed injection, unexpected acceptance, infrastructure exit codes, and timeout exits fail qualification. Human-readable diagnostics remain available, but qualification does not match their text.