Context
Every integration scenario is validated the same way, through CheckReportMutants in integrationtest/Validation/ValidationProject/ValidateStrykerResults.cs: the mutants of every file in the report are summed per status and compared against seven numbers.
CheckReportMutants(report, total: 673, ignored: 275, survived: 2, killed: 3, timeout: 2, nocoverage: 357);
Eleven validation methods share that oracle.
Problem
Aggregate counts are a weak oracle, in three distinct ways.
They don't say what changed. A failure reads actualNoCoverage should be 357 but was 358. It names no file, no mutant and no reason. Whoever hits it cannot tell a real regression from a legitimate consequence of their own change, on a report holding hundreds of mutants.
They invite being updated rather than investigated. Because the message carries no diagnosis, the cheapest way past a one-mutant delta is to write the new number down. A count that gets adjusted on every unexplained failure stops guarding anything.
They are blind to compensating shifts. One file losing three covered mutants while another gains three leaves every total intact. That is not hypothetical for the MTP runner, where coverage is the union over several test projects: which assembly loses coverage depends on flush ordering, so a redistribution with an unchanged total is exactly the shape a regression can take.
Evidence
Measured while working on #3769, on the MTPSolution scenario, with and without the fix that PR carries:
| Run |
total |
nocoverage |
killed mutants of Lesson.cs |
| Bug present |
673 |
358 |
0 |
| Bug fixed |
673 |
357 |
1 |
An assembly lost all of its coverage, and the whole signal at the aggregate level is one mutant out of 673 moving between two statuses.
Proposal
Assert which mutants are expected to be covered, and with which status, rather than how many there are of each. Per source file at least, per mutant where it is worth it, so that a failure names the file and the mutant instead of a delta.
This only pays off applied to every scenario: a single scenario with precise expectations next to ten with counts leaves the same blind spots everywhere else.
Worth considering as part of the work:
- Where expectations live. Inline arguments get unwieldy at this granularity; a committed expectation file per scenario, diffable against the report, reads better and makes a change visible in review.
- How they are produced. Hand-writing per-mutant expectations for eleven scenarios is not realistic — a way to regenerate them from a known-good run, reviewed as a diff, keeps the cost down.
- What stays aggregate. Totals still catch mutants appearing or vanishing wholesale, so the counts are worth keeping alongside rather than replacing.
Prior art in the repo
#3769 adds CheckEveryMutatedProjectIsCovered, which asserts one killed mutant per named source file for a single scenario. It exists because the aggregate count in that scenario moves by one and names nothing. It is a step in this direction, deliberately narrow, and the reviewer's suggestion to generalise it is what this issue is about.
Context
Every integration scenario is validated the same way, through
CheckReportMutantsinintegrationtest/Validation/ValidationProject/ValidateStrykerResults.cs: the mutants of every file in the report are summed per status and compared against seven numbers.Eleven validation methods share that oracle.
Problem
Aggregate counts are a weak oracle, in three distinct ways.
They don't say what changed. A failure reads
actualNoCoverage should be 357 but was 358. It names no file, no mutant and no reason. Whoever hits it cannot tell a real regression from a legitimate consequence of their own change, on a report holding hundreds of mutants.They invite being updated rather than investigated. Because the message carries no diagnosis, the cheapest way past a one-mutant delta is to write the new number down. A count that gets adjusted on every unexplained failure stops guarding anything.
They are blind to compensating shifts. One file losing three covered mutants while another gains three leaves every total intact. That is not hypothetical for the MTP runner, where coverage is the union over several test projects: which assembly loses coverage depends on flush ordering, so a redistribution with an unchanged total is exactly the shape a regression can take.
Evidence
Measured while working on #3769, on the
MTPSolutionscenario, with and without the fix that PR carries:nocoverageLesson.csAn assembly lost all of its coverage, and the whole signal at the aggregate level is one mutant out of 673 moving between two statuses.
Proposal
Assert which mutants are expected to be covered, and with which status, rather than how many there are of each. Per source file at least, per mutant where it is worth it, so that a failure names the file and the mutant instead of a delta.
This only pays off applied to every scenario: a single scenario with precise expectations next to ten with counts leaves the same blind spots everywhere else.
Worth considering as part of the work:
Prior art in the repo
#3769 adds
CheckEveryMutatedProjectIsCovered, which asserts one killed mutant per named source file for a single scenario. It exists because the aggregate count in that scenario moves by one and names nothing. It is a step in this direction, deliberately narrow, and the reviewer's suggestion to generalise it is what this issue is about.