fix(vitest-runner): never report a mutant run that executed zero tests as survived - #6146
Conversation
|
Follow-up: I've now pinned the exact mechanism with probes inside vitest's main chunk, and it changes the framing slightly — the lost runs are bail's own cancellation racing the result flush. Stryker passes Three confirmations:
Given the root cause, you may prefer a different fix than the retry — e.g. not enabling vitest's bail and implementing bail runner-side (cancel after reading results), or detecting the interrupted run via the reporter lifecycle. The retry guard stands as defense-in-depth either way (an empty filtered run should never be scored survived regardless of the cause). Happy to rework in whichever direction you prefer. |
Fixes #6073
Under load, a filtered mutant run can complete with its test tasks collected but never executed.
ctx.statethen holds result-less tasks,.filter((test) => test.result)empties the result list, andtoMutantRunResultreports survived withhitCount 0— even though no test ever ran against the mutant. This is the source of the non-deterministicSurvived↔Killed/Timeoutverdict flips in #6073 (full instrumented evidence in this comment): every unstable verdict is a zero-test run, and verdicts with at least one executed test are stable.The run loss itself is timing-dependent (clustered after bail-aborted runs). I could not reproduce it outside real Stryker orchestration (~1,400 instrumented pure-vitest runs mimicking the runner exactly are clean), but the runner should not trust an empty run regardless of the trigger: a run that executed none of its selected tests proves nothing about the mutant.
Change
In
mutantRun, whentestFilteris non-empty and the run completed with zero executed tests:Unfiltered runs (no
testFilter) are unaffected.Verification
Against stryker-pertest-flip-repro (vitest 4.1.9 and 4.1.10):
>>> FLIP DETECTEDon every 5-run batch; survivors swing 5–11;47 Killed / 5 Survived / 5 Timeout); 16 lost runs absorbed by retries (max 3 for one mutant); genuine survivors unchanged.Open question: if you'd rather surface exhaustion as a different status than an error result (e.g. treat it like a timeout so core retries in a fresh process), happy to adjust.