Skip to content

fix(vitest-runner): match Vitest 5 test name separator when filtering tests - #6214

Open
scolladon wants to merge 1 commit into
stryker-mutator:masterfrom
scolladon:fix/vitest5-test-name-separator
Open

fix(vitest-runner): match Vitest 5 test name separator when filtering tests#6214
scolladon wants to merge 1 commit into
stryker-mutator:masterfrom
scolladon:fix/vitest5-test-name-separator

Conversation

@scolladon

Copy link
Copy Markdown

Fixes #6210

On Vitest 5 the per-test name filter matches nothing, so every mutant with per-test coverage runs zero tests and is reported Survived.

Vitest 5 matches testNamePattern against the full chain joined with ' > '. The runner builds both the test id and the filter regex from collectTestName, which joins with a single space, so the pattern never matches and the mutant run selects no tests.

Change

collectTestName / toRawTestId take the separator as a parameter, defaulting to the current ' ' so nothing changes below Vitest 5. The runner derives it once from the running Vitest version — reusing the semver.satisfies(vitestWrapper.version, …) approach already used for isGreaterThanVitest4Point1 — and both ends use it:

  • provided to the sandbox as testNameSeparator, so the ids recorded in mutantCoverage are built with it;
  • passed to convertTestToTestResult, so the ids reported back match those keys.

Both ends must agree, or the coverage keys and the filter would disagree again.

Verification

Reproduced against a real project (~1,700 unit tests, Vitest 5.0.0, runner 10.0.0), scoped to one source file:

statuses testsCompleted
before { Ignored: 1, Survived: 23 } 0
after { Ignored: 1, Killed: 23 } 1

coveredBy was correctly populated (2-37 tests per mutant) in both cases — coverage analysis was never the problem, only the filter. The patched run is also faster (25s vs 41s), because it runs the selected test instead of the whole file's worth of nothing.

Isolated with a small probe driving createVitest the way the runner does, on Vitest 5.0.0:

testNamePattern built from tests run
leaf test name only 1
suite chain joined with ' ' (current behaviour) 0
suite chain joined with ' > ' 1

Setting project.config.testNamePattern after createVitest, and passing cwd-relative paths to ctx.start(), both still work on Vitest 5 — only the separator is wrong.

Tests

  • test-helpers.spec.ts: toRawTestId joins the suite chain with the given separator.
  • vitest-runner.spec.ts: the runner provides ' ' for Vitest <5 and ' > ' for >=5. Verified discriminating — moving the threshold to >=6.0.0 fails exactly the >=5.0.0 case.

npm run test:unit (35) and npm run test:integration (33) pass. Note the repo's dev dependency is Vitest 4.1.11, where this change is a deliberate no-op, so CI here exercises only the backward-compatible path — the Vitest 5 behaviour is covered by the version-gate unit tests above rather than by a live Vitest 5 run.

Related

#6146 is complementary: it stops a zero-test mutant run being reported as survived at all. With that in place this bug would have surfaced as a loud failure rather than a silently collapsed score.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

vitest-runner: on Vitest 5 the per-test name filter matches nothing, so every covered mutant survives (testNamePattern now joins the chain with ' > ')

1 participant