Metrics

The plugin calculates a set of metrics that help determine the effectiveness of your test suite and facilitate finding the areas to concentrate unit test development effort on.

All these measures can be found in the Measures view of your project in the Mutation Analysis domain.

Metrics marked with (beta) are experimental features that have to be enabled in the plugin settings. These features might be subject to change in future releases or do not yet work as expected in all cases. For example the %-valued metrics currently do not work for multi-module projects.

• Alive: Survivors - Number of mutations not killed by a test (see Survivor rule)

• Alive: Not Covered - Number of mutations non covered by any test (see Lurker rule)

• Alive: Total % (beta) - The ratio of non-covered & surviving mutations to the total number of mutations. This gives an indication where your test development efforts should concentrate on. The major difference to the Mutation Coverage metric is, that this is global while the coverage is local. For single-module projects, there shouldn't be a difference, but for multi-module projects there is. For example when a single class has very low coverage, but only 2 mutations, it would still rank very high on the coverage metrics, while on the Alive Total it would rank much lower compared to a class that has higher coverage but more absolute surviving mutants. Also note, the measure always sums up to 100% on project level.

• Killed: Total - The number of all mutations that are either killed by a test,killed by a timeout or killed by a memory error

• Killed: by Tests - Number of mutations killed by tests

• Killed: by Memory Error - Number of mutations detected by memory errors (the test execution failed with a memory related exception)

• Killed: by Timeout - Number of mutations detected by time outs (the test took unusually long)

• Mutation: Coverage - Ratio (%) of how many mutations got killed vs. total mutations (i.e. 67% got killed). In multi-module projects, the values get averaged based on the children, where each children contributes equally. Therefore the computed value might be higher and the Test Kill % might provide a better indication of the overall kill ratio accross the entire project.

• Mutation Density (beta) - Number of Mutations per coverable lines (lines to cover). This measure should give an indication of the density of your code. A value > 100% indicates more than 1 mutation per line (statistically). High values (such as 400%) indicate very dense code. This value also depends on the active mutation operators. When you have dedicated classes for constants, the "inline constants" mutator generates mutations in this class, although there might be little or no coverage line in the same class. On the other hand, a density of < 100% might be an indication of too verbose code, where you need to many lines to express your logic, which could negatively affect the maintainability.

• Mutations: Total - Total number of mutations generated

• Mutations: Total % (beta) - Same as Total Mutations but in % to see the distribution of the mutations across your project. This enables a TreeMap view in SonarQube which allows a better visual representation of the hotspots in your codebase.

• Test: Executions - The number of tests that were executed in order to try to kill the mutant. High number indicate a not well-targeted test (or a surviving mutant), while low numbers indicate, the test is a precise specification of the semantics in the code.

• Test: Kills - Number of Mutations killed by a specific tests, with a per-test drill-down. This might indicate a weakness in your test suite where most of the tests is done by one or a few tests. Note that Pitest aborts a mutation run for a single mutant if one test killed it to improve execution time. This first test is displayed in this measure.

• Test: Kills % (beta) - Same as Test Kills but in % of the total number of kills to better visualize test-kill distribution across your project. This enables a TreeMap view in SonarQube. This not the same as the Mutation Coverage. Test Kills % describes the number of all mutations have been killed while the score is how many mutations have been killed per component. Therefore the drill-down shows different values and the total is averaged aggregation.

• Unknown Status - Number of mutations with unknown status (see Unknown status rule). This metric is usually not visible.