Rethink testing strategy

[ph-fritsche]

Problem description

1. We run linting and tests on multiple node versions that we support according to
   

   user-event/package.json

   Lines 13 to 14 in ea50231

   	"engines": {
   	"node": ">=12",

   
   This has no merits.
   Our source code has no dependency on any NodeJS API, so that the result for the source itself will always be the same.
   The tested code is not the distributed code, so that if the different NodeJS versions yield different results, it only flags problems with our testing environment, but don't indicate anything about the build we would distribute from that commit.

2. We don't test our build. Misconfigurations or bugs in our build tools result in broken builds being published and any potential fix is verified only by manually testing the build.

3. We only test on Jsdom. We try to be platform agnostic, but out automation doesn't help us to develop software that works as intended in different environments, and the differences between Jsdom and e.g. Chrome are too numerous and wide for relying on manual testing only.

Suggested solution

• Remove source tests on older node versions.
• Refactor the tests so that they don't rely on Jest and could be run in different environments.
• Run the tests in a headless browser during CI.
• Split up our tests in
  a) those that rely on mocks or test internals and
  b) those that only interact with the DOM and can be observed using only the main export.
• Add at least a smoke test on different node versions using the build.

Additional context

No response

[ph-fritsche]

/cc @nickmccurdy @timdeschryver @MatanBobi

[nickserv]

I was thinking of getting some Testing Library projects running on test frameworks other than Jest. Vitest would probably be a good one to focus on, as it's fast, easy to configure, mostly compatible with Jest, and supports enabling/disabling global injection.

[ph-fritsche]

Can you leverage Vitest to run tests in a headless browser?

[MatanBobi]

@ph-fritsche great initiative, I highly agree with all of the points.
Small thing about 3, I agree that the differences between JSDOM and Chrome are enormous (don't forget about happy-dom too), but our code is based on the standards and not on a specific engine implementation. If we'll also run tests using happy-dom for examples, some of our tests won't work and that's because happy-dom has some missing implementation.
So what's our suggestion here? Should we run all our tests on a headless browser too? Running our tests on JSDOM is nice because a big part of our users are using JSDOM to run our library.

[ph-fritsche]

I don't know if we should run (some?) tests in happy-dom too, but I think we need to run our tests in a headless browser.
In the end the browser has been our point of reference all the time.

Some things are hard to test in a browser and also don't need to be tested twice - like the tests on correctly wiring the APIs through .setup().

There might be exceptions due to limited implementation in the environment, but in general a test using user-event should at least work both in Jsdom and headless Chrome - so our tests should run in those too.

[ph-fritsche]

A little update here: I'm trying to make this work without rewriting too much of the tests.

Current approach is Karma+Jasmine with Jest's expect, spyOn and fn so that any tests without mocking could stay the same, as this will make it more likely that the test environment might be reused in other repos.
jasmine types are a mess though and I'm not fond of maintaining those types myself by hand even if they're unlikely to change. So maybe this isn't practical.

Providing the transpiled files is already resolved. Letting Karma manage the files and adding a preprocessor was slow and inflexible. I wrote a little tool using chokidar and rollup (with @swc/core as transpiler) that allows to easily provide the necessary transpiled modules. It allows to use modules with ts paths and modules with node dependencies (replacing those with stubs). A Karma plugin can then update the fileList when files changed and provide them from memory.
As that's too much code out of scope of this library, I started a repo for this at https://github.com/ph-fritsche/toolbox

[nickserv]

That sounds like an interesting approach. Is your goal to have a more minimal test environment, or specifically ensure Karma and Jasmine are supported? If the former, I think it could be easier to use Vitest, optionally with globals disabled.

[ph-fritsche]

The goal is to run tests in at least one major browser (preferably Chrome) and in Jsdom.
These tests should be run from and reported to the CLI so that you can just use e.g. the embedded console in Vscode.

Inspecting any test breakpoint in the browser console or walk through it per debugger would be nice but isn't strictly necessary.
I'd like to be able to provide the tests in a manner that someone could just build them, add an adapter and run them against their environment to check for differences in the DOM implementation, but this also isn't strictly necessary.

Neither Karma nor Jasmine is a requirement. The more code I read, the more I start thinking that by the point I'll have understood the necessary configurations and plugin hooks, I could also have written the runner myself.

[nickserv]

I've had success with using Karma for that sort of thing in the past, but maybe a more modern alternative like Cypress or Playwright would be easier to set up now.