CoverageBasedTest misses recently added tests in coveredBy, marking killable mutants as Survived

[im47cn-bot]

Stryker.NET version: 4.16.0
Coverage analysis: perTest (CoverageBasedTest)

Summary

A mutant was reported Survived although two tests that kill it were present in the run. The mutant's coveredBy list did not include those tests — they appear to have been missed by the per-test coverage mapping, so the mutant never ran against them.

Reproduction outline

1. A Block removal mutant inside a private helper (SkipValue, a JSON-ish scanner) survived a full run (388–390 tests, all green).
2. mutation-report.json for that mutant listed only 6 covering tests — all older tests; two recently added tests that explicitly exercise the mutated branch ("a":"a\"}" style inputs where string-aware skipping is the whole point) were absent from coveredBy.
3. Manually applying the identical mutation (removing the block body) and running only those two tests: both fail → the mutant is killable, not equivalent.

So the effective score was understated: Survived: 1 in the report, ground truth Survived: 0.

Impact

• False survivors mislead equivalent-mutant triage: reviewers spend effort constructing equivalence proofs for mutants that are actually killed by tests the mapper missed.
• Silent: nothing in the report suggests the coverage map is incomplete.

Environment

• .NET 8.0, xUnit 2.9.2 (Theory-heavy suite; the missed tests were [Fact]s added in the same session, file rebuilt before the run — Number of tests found included them).
• The killed-by-omission tests cover exactly the mutated lines (verified via coverlet line data).

Suggestion

When a mutant is marked Survived, consider a diagnostic mode that re-runs it against tests covering the mutant's line range per the raw coverage data (or at minimum flags survivors whose coveredBy set looks suspiciously small relative to line coverage).

[dupdob]

thanks for opening this issue. Alas, there is not enough information to work upon. Hence, I have detail requests for you.

1. Please run Stryker with coverage analysis disabled (using "coverage-analysis": 'off' in the config file). If there the mutant still survives, I suggest you explicitly inject the mutation and run the test base again. (directly, not Stryker). I would expect the mutant to still survive, which will demonstrate there is an error in the premise that the mutant should be killed. If the mutant is killed via this method, see below.

2. If the mutant is properly killed with test coverage disabled, it relates to a coverage analysis issue. The most frequent cause for this is that some static data/singleton/class constructor is involved. Static data are known to be the cause of problems for test suite, because they create invisible coupling between tests. Stryker uses several detection mechanisms to detect what we call 'static mutations' and test them against all tests, but this may fail.Access to the code base and a pointer to the problematic mutation is required to help you with this analysis

3. ultimately, this may be linked to xUnit integration with VsTest, which exhibits some shortcomings. One of the most problematic one (for Stryker) is that xUnit does not properly report theories to VsTest which may lead to Stryker missing some test results (which could explain a false survivor).

Details: VsTest expects the test framework to specify how many results are awaited for a given test. When VsTest has received the appropriate number of results, it reports the test as complete. Note that supplemental results are still reported and can be integrated. xUnit does not report the number of test results for some theories. If such a test theory happens to be run last, or to 'outlive' any other tests, the test session will be reported as finished and Stryker cannot receive any more test results.

While I have seen this problem firsthand, this is quite rare in the wild, so I do not think this is the problem here