Skip to content

fix(vitest-runner): never report a mutant run that executed zero tests as survived - #6146

Open
scolladon wants to merge 1 commit into
stryker-mutator:masterfrom
scolladon:fix/vitest-runner-empty-run-survived
Open

fix(vitest-runner): never report a mutant run that executed zero tests as survived#6146
scolladon wants to merge 1 commit into
stryker-mutator:masterfrom
scolladon:fix/vitest-runner-empty-run-survived

Conversation

@scolladon

Copy link
Copy Markdown

Fixes #6073

Under load, a filtered mutant run can complete with its test tasks collected but never executed. ctx.state then holds result-less tasks, .filter((test) => test.result) empties the result list, and toMutantRunResult reports survived with hitCount 0 — even though no test ever ran against the mutant. This is the source of the non-deterministic SurvivedKilled/Timeout verdict flips in #6073 (full instrumented evidence in this comment): every unstable verdict is a zero-test run, and verdicts with at least one executed test are stable.

The run loss itself is timing-dependent (clustered after bail-aborted runs). I could not reproduce it outside real Stryker orchestration (~1,400 instrumented pure-vitest runs mimicking the runner exactly are clean), but the runner should not trust an empty run regardless of the trigger: a run that executed none of its selected tests proves nothing about the mutant.

Change

In mutantRun, when testFilter is non-empty and the run completed with zero executed tests:

  1. retry the run (bounded, 3 attempts) — re-dispatching succeeds in practice;
  2. if it still comes back empty, return an error run result stating the invariant, so core reschedules/reports it instead of recording a phantom survivor.

Unfiltered runs (no testFilter) are unaffected.

Verification

Against stryker-pertest-flip-repro (vitest 4.1.9 and 4.1.10):

  • unpatched: >>> FLIP DETECTED on every 5-run batch; survivors swing 5–11;
  • patched: 5/5 byte-identical reports (47 Killed / 5 Survived / 5 Timeout); 16 lost runs absorbed by retries (max 3 for one mutant); genuine survivors unchanged.

Open question: if you'd rather surface exhaustion as a different status than an error result (e.g. treat it like a timeout so core retries in a fresh process), happy to adjust.

@scolladon

Copy link
Copy Markdown
Author

Follow-up: I've now pinned the exact mechanism with probes inside vitest's main chunk, and it changes the framing slightly — the lost runs are bail's own cancellation racing the result flush. Stryker passes bail: 1; on the first failing test the worker fires rpc().onCancel("test-failure") immediately, while that failing test's result is still in the throttled task-update batch. The main process handles the cancel → pool.cancel() force-stops the worker before the batch is applied, and state.cancelFiles registers bare result-less tasks for the remaining files — so the run resolves "complete" with zero results, and the runner scores it survived. The flipping mutants were actually being killed — bail destroyed the evidence of its own trigger.

Three confirmations:

  • probes at executeTests / cancelCurrentRun / the worker→main onCancel RPC: 13/13 zero-test verdicts show the in-window onCancel("test-failure") → silent-cancel trace (files parked via state.cancelFiles, run resolves cleanly);
  • this PR's retry guard eliminates the flip (5/5 byte-identical reports on the repro);
  • disableBail: true also eliminates it (5/5) — so users hitting this today have a config-level workaround, at the cost of running all covering tests per mutant.

Given the root cause, you may prefer a different fix than the retry — e.g. not enabling vitest's bail and implementing bail runner-side (cancel after reading results), or detecting the interrupted run via the reporter lifecycle. The retry guard stands as defense-in-depth either way (an empty filtered run should never be scored survived regardless of the cause). Happy to rework in whichever direction you prefer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

vitest-runner: non-deterministic mutant verdicts under coverageAnalysis=perTest (9.6.1, vitest 4.1.9)

1 participant