Skip to content

feat(core): add experimental performance report #6136

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
lo1tuma wants to merge 1 commit into
base: master
Choose a base branch
from
Open

feat(core): add experimental performance report #6136

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 12 additions & 7 deletions docs/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -215,13 +215,12 @@ _Note: Disable bail needs to be supported by the test runner plugin in order to

### `disableTypeChecks` [`boolean` | `string`]


<details>

<summary>History</summary>

| Version | Changes |
| ------- | --------------------------- |
| Version | Changes |
| ------- | ------------------------------------------------------------------------------- |
| 7.0 | Default changed from `"{test,src,lib}/**/*.{js,ts,jsx,tsx,html,vue}"` to `true` |

</details>
Expand All @@ -230,12 +229,10 @@ Default: `true`<br />
Command: _none_<br />
Config file: `"disableTypeChecks": false`

Set to `true` to disable type checking, or `false` to enable it. For more control, configure a pattern that matches the files of which type checking has to be disabled, for example: `"{test,src,lib}/**/*.{js,ts,jsx,tsx}"`. Setting it to `true` will disable type checking for all TypeScript-ish files (currently *.ts, *.js, *.tsx, *.jsx, *.mjs, *.mts, *.cts, *.cjs, *.html and *.vue files).
Set to `true` to disable type checking, or `false` to enable it. For more control, configure a pattern that matches the files of which type checking has to be disabled, for example: `"{test,src,lib}/**/*.{js,ts,jsx,tsx}"`. Setting it to `true` will disable type checking for all TypeScript-ish files (currently _.ts, _.js, _.tsx, _.jsx, _.mjs, _.mts, _.cts, _.cjs, _.html and _.vue files).

Disabling type checking is needed because Stryker will create (typescript) type errors when inserting the mutants in your code. Stryker disables type checking by inserting `// @ts-nocheck` atop those files and removing other `// @ts-xxx` directives (so they won't interfere with `@ts-nocheck`).



### `dryRunOnly` [`boolean`]

Default: `false`<br />
Expand All @@ -262,6 +259,14 @@ The options for the event recorder reporter:

- `baseDir`: The relative base dir to write the events to.

### `experimentalPerformanceReport` [`boolean`]

Default: `false`<br />
Command line: `--experimentalPerformanceReport`<br />
Config file: `"experimentalPerformanceReport": true`

Experimental. Write a detailed performance report to `reports/mutation/performance.json`. It contains per-phase timing (setup, initial run, mutation), per-mutant timing, the static-vs-runtime cost split, environment reloads, and machine and configuration info. Useful for diagnosing and comparing the performance of a mutation test run. Off by default; the format is unstable and may change between releases. See [performance report](./performance-report.md) for the format and how to read the numbers (wall time vs summed work).

### `fileLogLevel` [`string`]

Default: `off`<br />
Expand Down Expand Up @@ -326,7 +331,7 @@ Default: `[]`<br />
Command line: _none_<br />
Config file: `"ignorers": ["console.debug"]`<br />

Specify which ignore-plugins to use. With an ignore-plugin, you can skip mutating specific code patterns that you don't want to test or cannot be mutated. For example, you may use an ignore-plugin to exclude all `console.debug()` statements from mutation testing.
Specify which ignore-plugins to use. With an ignore-plugin, you can skip mutating specific code patterns that you don't want to test or cannot be mutated. For example, you may use an ignore-plugin to exclude all `console.debug()` statements from mutation testing.

Here's a list of built-in ignore plugins:

Expand Down
102 changes: 102 additions & 0 deletions docs/performance-report.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
---
title: Performance report (experimental)
custom_edit_url: https://github.com/stryker-mutator/stryker-js/edit/master/docs/performance-report.md
---

The experimental performance report is an opt-in, machine-readable breakdown of where time goes
during a mutation test run. It is meant for diagnosing and comparing performance, not for regular
reporting. Enable it with [`experimentalPerformanceReport`](./configuration.md#experimentalperformancereport-boolean):

```json
{
"experimentalPerformanceReport": true
}
```

When enabled, Stryker writes `reports/mutation/performance.json`. The standard reports
(`mutation.json`, the HTML report) are unchanged.

:::caution
The format is **experimental and unstable**; fields may change or be removed between releases.
:::

## Reading the numbers

All times are in **milliseconds**. Two kinds of duration appear, and mixing them up is the easiest
mistake to make:

- **Wall time** – real elapsed time. Used for the top-level `totalWallMs` and the `phases` that
partition it (`setup`, `initialRun`, `mutation`, `reporting`).
- **Summed work** – the sum of individual durations across everything that ran, including work that
ran **in parallel**. With concurrency > 1 these values can exceed the matching wall time; they are
meant for relative comparison, not as elapsed time. This applies to `phases.check`,
`phases.testRun`, `totals.static.wallMs`, `totals.runtime.wallMs`, and each worker's `busyWallMs`.

## Fields

### `phases` (wall time, exhaustive)

`setup + initialRun + mutation + reporting === totalWallMs`.

| Field | Meaning |
| -------------------- | ----------------------------------------------------------------------------------------------- |
| `setup` | Process start until the initial (dry) run starts: config, instrumentation, sandbox creation. |
| `initialRun` | The dry run. |
| `initialRunNet` | Sum of the individual test times during the dry run. |
| `initialRunOverhead` | `initialRun − initialRunNet` (runner start-up etc.). |
| `mutation` | From the end of the dry run until all mutants have been tested: planning, checking and running. |
| `check` | **Summed** time spent in type checkers (0 when no checker is configured). |
| `testRun` | **Summed** per-mutant run time across all workers. |
| `reporting` | Generating and writing the reports after the last mutant. |

`check` and `testRun` are breakdowns of work done _during_ the `mutation` phase; because they are
summed across workers they do not add up to the `mutation` wall time.

### `totals`

| Field | Meaning |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `byPlanKind` | Counts of `earlyResult` / `noCoverage` / `static` / `runtime` mutants, taken from the **plan** (before checking/running). |
| `byStatus` | Counts per mutant **outcome** status (`Killed`, `CompileError`, `Survived`, …). |
| `static` | For static mutants **that ran**: `count`, `wallMs` (**summed**), `reloads` (count triggered _by_ static mutants), `reloadWallMs` (**summed**). |
| `runtime` | For runtime mutants **that ran**: `count`, `wallMs` (**summed**). |
| `reload` | **All** reloads: `count` and `wallMs` (**summed**). This is `>= static.reloads`, because a reload also fires on the first runtime mutant after a static one (to clear it); that reload is attributed to the runtime mutant, so `static.reloads` alone understates total reload cost. |
| `staticShareOfMutantWallMs` | `static.wallMs / (static.wallMs + runtime.wallMs)` — the share of mutant-execution **work** spent on static mutants. Concurrency-independent. |
| `staticShareOfCount` | Share of run mutants that are static. |
| `retries` | Test-runner restarts after a crash. |
| `oomRestarts` | Test-runner restarts after running out of memory. |

### `workers`

One entry per test-runner process, with `mutantsHandled`, `busyWallMs` (**summed** run time on that
worker) and `idleWallMs` (time the worker waited between mutants). Useful for spotting an
under-utilized parallel tail.

### `mutants`

One entry per mutant that was run (early-result and no-coverage mutants are counted in
`totals.byPlanKind` but not listed here):

| Field | Meaning |
| --------------------- | ---------------------------------------------------------------------------------- |
| `wallMs` | Wall time of this mutant's run, including any environment reload it triggered. |
| `reloadWallMs` | Time spent reloading the environment for this mutant. See the reload caveat below. |
| `static` / `reloaded` | Whether the mutant is static, and whether its run reloaded the environment. |
| `workerId` | The worker that ran it. |
| `selectedTests` | Number of tests selected for this mutant (whole suite for static mutants). |
| `coveredBy` | Number of tests covering this mutant. |
| `testsCompleted` | Number of tests actually executed (can be fewer than `selectedTests` due to bail). |

## Caveats

- **`reloadWallMs` only measures full worker restarts.** Runners that reload in-process (vitest,
jest) fold the reload into the mutant's `wallMs` without breaking it out, so `reloadWallMs` is `0`
for them even when `reloaded` is `true`. Runners that restart the whole process (mocha, jasmine,
karma, tap) report the restart time in `reloadWallMs`.
- **Summed values scale with concurrency.** Compare them against other summed values, not against
wall-clock phases.
- **Planned vs run.** `context.mutants` is the total; `context.mutantsRun` is how many actually
reached the test runner. Mutants that were ignored, had no coverage, or were killed by a checker
(`CompileError`) never run, so they appear in `totals.byPlanKind` / `totals.byStatus` but **not**
in the `static`/`runtime` wall totals or the `mutants` array. On a TypeScript project a large
fraction can be `CompileError`, so `mutantsRun` is often much smaller than `mutants`.
72 changes: 18 additions & 54 deletions packages/api/schema/stryker-core.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,15 +18,7 @@
"logLevel": {
"title": "LogLevel",
"type": "string",
"enum": [
"off",
"fatal",
"error",
"warn",
"info",
"debug",
"trace"
],
"enum": ["off", "fatal", "error", "warn", "info", "debug", "trace"],
"tsEnumNames": [
"Off",
"Fatal",
Expand All @@ -40,23 +32,13 @@
"coverageAnalysis": {
"title": "CoverageAnalysis",
"type": "string",
"enum": [
"off",
"all",
"perTest"
]
"enum": ["off", "all", "perTest"]
},
"reportType": {
"title": "ReportType",
"type": "string",
"enum": [
"full",
"mutationScore"
],
"tsEnumNames": [
"Full",
"MutationScore"
]
"enum": ["full", "mutationScore"],
"tsEnumNames": ["Full", "MutationScore"]
},
"clearTextReporterOptions": {
"title": "ClearTextReporterOptions",
Expand Down Expand Up @@ -211,10 +193,7 @@
{
"type": "array",
"items": {
"type": [
"string",
"array"
]
"type": ["string", "array"]
}
},
{
Expand Down Expand Up @@ -269,11 +248,7 @@
"buildCommand": {
"type": "string",
"description": "Configure a build command to run after mutating the code, but before mutants are tested. This is generally used to transpile your code before testing. Only configure this if your test runner doesn't take care of this already and you're not using just-in-time transpiler like `babel/register` or `tsx`.",
"examples": [
"tsc -b",
"babel src --out-dir lib",
"npm run build"
]
"examples": ["tsc -b", "babel src --out-dir lib", "npm run build"]
},
"checkers": {
"description": "Enable checker plugins here. A checker plugin will be invoked for each mutant before it is run in a test runner. It can check to see of a given mutant is valid, by for example validate that it won't result in a type error",
Expand Down Expand Up @@ -327,7 +302,7 @@
},
"dryRunOnly": {
"description": "Execute the initial test run only without doing actual mutation testing. Dry run only will still mutate your code before doing the dry run without those mutants being active, thus can be used to test that StrykerJS can run your test setup. This can be useful, for example, in CI pipelines.",
"type":"boolean",
"type": "boolean",
"default": false
},
"eventReporter": {
Expand Down Expand Up @@ -365,7 +340,7 @@
},
"fileLogLevel": {
"description": "Set the log level that Stryker uses to write to the \"stryker.log\" file",
"$ref": "#/definitions/logLevel",
"$ref": "#/definitions/logLevel",
"default": "off"
},
"inPlace": {
Expand Down Expand Up @@ -405,11 +380,7 @@
"default": {}
},
"packageManager": {
"enum": [
"npm",
"yarn",
"pnpm"
],
"enum": ["npm", "yarn", "pnpm"],
"description": "The package manager Stryker can use to install missing dependencies."
},
"plugins": {
Expand All @@ -418,9 +389,7 @@
"items": {
"type": "string"
},
"default": [
"@stryker-mutator/*"
]
"default": ["@stryker-mutator/*"]
},
"appendPlugins": {
"description": "A list of additional plugins you want Stryker to load (`require`) without overwriting the (default) `plugins`.",
Expand All @@ -436,11 +405,7 @@
"items": {
"type": "string"
},
"default": [
"clear-text",
"progress",
"html"
]
"default": ["clear-text", "progress", "html"]
},
"htmlReporter": {
"description": "The options for the html reporter",
Expand All @@ -452,6 +417,11 @@
"$ref": "#/definitions/jsonReporterOptions",
"default": {}
},
"experimentalPerformanceReport": {
"description": "Experimental. Write a detailed performance report (per-phase timing, per-mutant timing, static vs runtime cost, environment reloads, machine and configuration info) to `reports/mutation/performance.json`. Off by default; the format is unstable and may change between releases.",
"type": "boolean",
"default": false
},
"disableTypeChecks": {
"description": "Set to 'true' to disable type checking, or 'false' to enable it. For more control, configure a pattern that matches the files of which type checking has to be disabled. This is needed because Stryker will create (typescript) type errors when inserting the mutants in your code. Stryker disables type checking by inserting `// @ts-nocheck` atop those files and removing other `// @ts-xxx` directives (so they won't interfere with `@ts-nocheck`). The default setting allows these directives to be stripped from all JavaScript and friend files in `lib`, `src` and `test` directories.",
"oneOf": [
Expand All @@ -462,9 +432,7 @@
"type": "string"
}
],
"examples": [
"{test,src,lib}/**/*.{js,ts,jsx,tsx,html,vue,cts,mts}"
],
"examples": ["{test,src,lib}/**/*.{js,ts,jsx,tsx,html,vue,cts,mts}"],
"default": true
},
"symlinkNodeModules": {
Expand All @@ -479,11 +447,7 @@
},
"cleanTempDir": {
"description": "Choose whether or not to clean the temp dir (which is \".stryker-tmp\" inside the current working directory by default).\n - false: Never delete the temp dir;\n - true: Delete the tmp dir after a successful run;\n - \"always\": Always delete the temp dir, regardless of whether the run was successful.",
"enum": [
"always",
false,
true
],
"enum": ["always", false, true],
"default": true
},
"testRunner": {
Expand Down
6 changes: 6 additions & 0 deletions packages/core/src/concurrent/concurrency-token-provider.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,12 @@ export class ConcurrencyTokenProvider implements Disposable {
public get testRunnerToken$(): Observable<number> {
return this.testRunnerTokenSubject;
}
public get checkerConcurrency(): number {
return this.concurrencyCheckers;
}
public get testRunnerConcurrency(): number {
return this.concurrencyTestRunners;
}
public readonly checkerToken$: Observable<number>;
public static readonly inject = tokens(
commonTokens.options,
Expand Down
Loading