Jest performance is at best 2x slower than Jasmine, in our case 7x slower

[EvHaus]

🐛 Bug Report

We've been using Jest alongside Jasmine for the same test suite for about a year now. We love Jest because it's developer experience is superb, however, on our very large monorepo with ~7000+ test specs, Jest runs about 7 times slower than Jasmine. This problem has been getting worse and worse as the test suite grows and as a result, we always run our test suite via Jasmine and only use Jest for development --watch mode.

We would ♥ to use Jest as our only test runner, but its poor performance is preventing us from doing so. Having to run both Jest and Jasmine runners requires painful CI setup and constant upkeep of the Jasmine environment setup (which is much more complex than Jest's).

I'd like to better understand why the performance difference is so significant and if there's anything that can be done to optimize it.

To Reproduce

I've created a very detailed project to reproduce and profile both Jest and Jasmine on the same test suite in this project: https://github.com/EvHaus/jest-vs-jasmine

The environment is the same. The configurations are very similar. Both use JSDom. Both use the same Babel setup. Additional instructions are contained therein.

Expected behavior

Running tests through Jest should ideally be as fast as running them through Jasmine.

Link to repl or repo (highly encouraged)

https://github.com/EvHaus/jest-vs-jasmine

Run npx envinfo --preset jest

Tested on a few different platform. See https://github.com/EvHaus/jest-vs-jasmine README for more info.

[layershifter]

@EvHaus thanks for the detailed report.

We have similar issues under Semantic-Org/Semantic-UI-React#2971, when Jest's suite is about 5x slower.

[thymikee]

@layershifter https://github.com/Semantic-Org/Semantic-UI-React/pull/2971/files#r202618422

[lukeapage]

Am I right in saying the problem is that jasmine loads all specs into one process and runs it, where as jest creates a new mini-environment per test suite?
We see exactly the same issue and profiling seems to show a significant amount of time resolving files and parsing javascript - unfortunately the multi-core aspect can't make up for this. I have no idea why resolving is so slow. We made significant speed increases by trying to make suites import the least number of files, but we've hit a wall on going further in that direction as we in many cases want to test multiple components running together and not to mock every dependency.
I planned to do some more profiling and it would be great if anyone on the core jest team can point in any directions to things they would like to see.

[SimenB]

I think it's a fair assumption to say it's the module resolution that's taking time. While require('foo'); is an in-memory cache lookup for jasmine (after the first one), every single test file in jest will have to do full resolution, and execution, of foo and all its dependencies. I doubt it's the resolution itself that takes significant time (we should have the fs in memory (after the first run, at least)), but executing the files probably takes up a significant chunk of time.

Another difference is that jest executes your code inside the jsdom vm, while with jasmine you've just copied over all the globals to the node runtime (https://github.com/jsdom/jsdom/wiki/Don't-stuff-jsdom-globals-onto-the-Node-global), which will always be quicker as you skip an entire abstraction layer (https://nodejs.org/api/vm.html).

That said, I agree it's really not ideal (to put it mildly) that Jest is about twice as slow as jasmine. I'm not really sure what we an do, though. We could try to cache the resolution (although we'd still have to run through the entire tree in case there's been any module mocking) which might allow us to not resolve modules by looking around, but again the FS should be in memory, so I doubt it'd have much impact.

@cpojer @mjesun @aaronabramov @rickhanlonii do you think there's anything clever we can do here? Or any awesome ways of profiling what we spend our time on?

---

Also, thank you so much for setting up a great reproduction case @EvHaus!

[gsteacy]

I doubt it's the resolution itself that takes significant time (we should have the fs in memory (after the first run, at least)), but executing the files probably takes up a significant chunk of time.

I did some profiling of the node processes while running Jest on my projects it seemed like requiring was one of the most time consuming tasks. At least that was the case on Windows (not WSL), which I found to be substantially slower than Linux, especially in watch mode. Granted, I'm not particularly confident in my understanding of the node profiler's output, but that's what it looked like. I saw the same thing with this reproduction.

[SimenB]

require is both resolution and execution

[EvHaus]

I think it's a fair assumption to say it's the module resolution that's taking time. While require('foo'); is an in-memory cache lookup for jasmine (after the first one), every single test file in jest will have to do full resolution, and execution, of foo and all its dependencies.

I haven't looked at the code, so I can't be totally sure -- but this sure "feels" like what's happening. When running through Jasmine there's a very long delay before anything is printed to the console (likely Jasmine resolving/executing ALL deps), and then the tests run through super quick. Whereas with Jest, it immediately starts running tests without any initial lag, but each test is significantly slower to run.

[simonbuchan]

Any chance switching from worker processes to the node vm api could help? That permits pre-compiling sources with vm.Script, then running them in isolated, sandboxed contexts. Obviously seems like a lot of work!

[cpojer]

Jest already does that.

[simonbuchan]

I believe you, but then what are these jest-worker processes?

[cpojer]

@rickhanlonii do you have the Jest architecture chart somewhere?

[simonbuchan]

If I had to guess, you use the workers for multi core, but VM as well for isolation, even with --runInBand? With cachedData that should skip most parse time even with multi proc. So the time is literally execution time of the modules, which would be pretty hard to skip I guess.

Maybe a --runVeryInBand that shares a VM?

[simonbuchan]

Looks like cachedData isn't being used in new vm.Script() (I'm looking at jest-runtime's ScriptTransformer - V8 can now cache script code after code execution, which looks like it's exposed by vm.Script#createCachedData() from node 10.6, so if there's somewhere sensible to do that, (just before any mock/test code execution somehow?) it might help even multi-proc?

[cpojer]

I tried to use cachedData for an experiment about two years back. There is even a PR (sorry on mobile so can’t find the link). There was no difference in perf that I observed. Cached code is much larger and I assume reading and validating that is equal to the parse time overhead that is saved. I’d be curious to see results of somebody re-running that experiment. Changing the script transformer and running some perf tests should give us some data.

[simonbuchan]

That sounds likely, it could well be that the delayed createCachedData() could work better as more useful code would be codegened and thus cacheable (if I'm reading that blog post right).

There's also v8::SnapshotCreator that node started using recently for it's own startup which persists the full execution state, not just parse/codegen output, but it sounds like it's quite fiddly to get working even when you control all the code executed (e.g. it requires all handles to be closed).