Slow performances when running tests with `--experimental-test-coverage`

[rozzilla]

Version

v22.9.0

Platform

Darwin N4V4PGFGPT 23.6.0 Darwin Kernel Version 23.6.0: Mon Jul 29 21:16:46 PDT 2024; root:xnu-10063.141.2~1/RELEASE_ARM64_T8112 arm64

Subsystem

No response

What steps will reproduce the bug?

Run tests with --experimental-test-coverage option. Comparing it to other solutions (f.e. running Node tests and getting the coverage with nyc) this is way slower. For instance, on a repo with ~800 tests, collecting coverage with nyc takes 77s, while running with --experimental-test-coverage takes 148s.

How often does it reproduce? Is there a required condition?

I can reproduce it every time I run the Node tests with the native test coverage option.

What is the expected behavior? Why is that the expected behavior?

The performances should be at least equal (ideally better) than with 3rd party dependencies (like nyc).

What do you see instead?

An average time spent per test of 185ms vs 96ms.

Additional information

I tried running with/without additional options (f.e. --test-coverage-exclude, --test-coverage-include, --test-coverage-branches, --test-coverage-functions, --test-coverage-lines) but it didn't help. The performance issue seems only to be related to the algorithm that's triggered by the --experimental-test-coverage option.

[cjihrig]

Did you happen to look into where the time is being spent? For example, do you know if most of the time is being spent collecting the coverage, or in parsing it to generate the report? If the time is being spent collecting the coverage, there likely isn't much we can do. Another thing you can try is collecting coverage data without using the test runner's coverage feature.

[rozzilla]

Did you happen to look into where the time is being spent? For example, do you know if most of the time is being spent collecting the coverage, or parsing it to generate the report?

Looking closely at the test execution, it seems that both things are happening:

• execution time per single test is higher (~2x times) when running with coverage option
• parsing and generating the report takes a lot of time, in fact, the test runner seems stuck for a bunch of seconds before generating the report, and then finally it outputs the result

To summarize, my assumption is that the performance issue seems to be a combination of both factors you mentioned @cjihrig. 👍🏼

[cjihrig]

A few questions:

• Is this codebase public by any chance?
• Are you using source maps?
• What does the execution time look like if you run Node with NODE_V8_COVERAGE, but not the --experimental-test-coverage flag?

[rozzilla]

• Is this codebase public by any chance?

Nope, but to give some more context about it, it's a fastify app exposing endpoints that do internal HTTP calls, query a PG Database, get cached data from Redis, etc. The repository has both unit tests, and integration tests that check everything (apart from mocking HTTP calls, and some other edge cases).

• Are you using source maps?

Nope

• What does the execution time look like if you run Node with NODE_V8_COVERAGE, but not the --experimental-test-coverage flag?

It's slightly slower (~5s)

[FishOrBear]

Can you try v20? I want to know how much the difference is

[rozzilla]

Can you try v20? I want to know how much the difference is

@FishOrBear, it's still way slower compared to nyc (so 77s). It took 124s (so less time than with Node 22), but I had to remove all of the options not available in Node 20, so:

• --test-coverage-exclude
• --test-coverage-include
• --test-coverage-branches
• --test-coverage-functions
• --test-coverage-lines

[cjihrig]

I think I ran into this on a project of my own over the weekend. Using Node v22.9.0, the test runner without coverage was taking ~1 second. With --experimental-test-coverage, the time was ~4 seconds. However, I tried running with NODE_V8_COVERAGE and no --experimental-test-coverage, and the time was still about the same (~4 seconds).

We may want to consider adding a NODE_DEBUG=test_runner log that includes the amount of time the test runner takes to create the coverage report. That would allow us to debug this issue more easily without asking users to profile.