Performance issue with the byRole query causing timeout errors

[MatheusPoliCamilo]

• @testing-library/jest-dom version: 5.11.0
• node version: 12.14.1
• yarn version: 1.21.1

What you did:

I converted several tests of the project I work on, which were using manual queries with container and querySelector, to use screen prioritizing the query byRole as recommended.

What happened:

There are some timeout errors when I run some tests after the conversion to use screen and the byRole selector.

Reproduction:

I made a small example to demonstrate the difference in ms in a small case, like a table with 8 cells, when using the query byRole compared to byText.

Problem description:

At the moment there is a certain performance problem with the byRole selector, which even causes a timeout problem in tests, something that does not happen with other selectors such as byText.

Below is the time it takes to run a test on the project I work on when using the byRole query (in which there is a timeout error in some moments for taking too long).

Then, the time it takes to perform the same test, but using the byText selector. I've run the test several times this way and with this selector there is no timeout, in addition to being much faster.

Suggested solution:

I believe that for now the solution would be not to suggest the query byRole as a priority, at least until there are some more performance improvements to get closer to the other selectors and avoid timeouts.

I apologize because I did not get to look at the code in depth to try to understand a little if there is a way to bring about an improvement in byRole. I am also aware of the benefits that this selector brings and I am willing to waste more time to run the tests to use it, however the time difference is still large and causing timeout errors. Thank you in advance!!!

[sousajunior]

I have the same problem.

[eps1lon]

If you're confident that you can manually verify the checks ByRole provides then I suggest switching to other queries. ByRole considers the a11y tree, styles and the accname spec. These checks are expensive but have a higher confidence that what you're testing is actually what a screen reader would "see".

It's like any other tool: If you don't need the features then you should switch to a tool with smaller feature set. If test performance is a bigger issue for you than test confidence then ByRole is not the right tool.

One thing: A real-world repro would help us a lot improving performance. We've had success in the past with good repros since they help us identify bottlenecks. Screenshots of code don't help us very much. Small examples that don't reproduce the issue are also not helpful.

[gnapse]

Shouldn't this issue be moved to @testing-library/dom?

[eps1lon]

What's also very important is the environment e.g. which browser or if you're running from node (via e.g. jest) what version of jsdom (yarn why jsdom or npm ls jsdom).

[MatheusPoliCamilo]

Thank you very much for responding! I will try to reproduce the code we have to make it clearer.

jsdom version: 16.2.2
Running jest version 26.1.0 on node.

[jihchi]

I've made a contrived repo here: https://github.com/jihchi/698_byrole_performance_issue

there is a 3rd-party datetime picker and it would take ~1 second when you opened the calendar and pick a specific date in the calendar by using getByRole('button', { name }).

[idanen]

Tried a PR adding a test like:

test('reasonable performance', () => {
  const {getByRole, getByLabelText} = render(
    `<label><input type="checkbox" name="checker" /> check</label>`,
  )
  let start = Date.now()
  getByLabelText(/check/i)
  let end = Date.now()
  const byLabelTextTime = end - start
  start = end
  getByRole('checkbox', {name: /check/i})
  end = Date.now()
  const byRoleTime = end - start

  expect(byRoleTime).toBeLessThan(5 * byLabelTextTime)
})

I think this gives a roughly good start.
I saw we're using getComputedStyle pretty heavily, which might be a bottleneck.
Tried to manage some caching mechanism like we do with isSubtreeInaccessible, which showed small improvement, but not sure it's a good direction.

I'd be happy to continue if anyone has better idea of where might be a bottleneck, or how can I profile it.

(does anyone see any value in a PR with just the failing test?)

[jihchi]

According to my investigation, @eps1lon has summarized the causes pretty clear and I agree with him:

If you're confident that you can manually verify the checks ByRole provides then I suggest switching to other queries. ByRole considers the a11y tree, styles and the accname spec. These checks are expensive but have a higher confidence that what you're testing is actually what a screen reader would "see".

It's like any other tool: If you don't need the features then you should switch to a tool with smaller feature set. If test performance is a bigger issue for you than test confidence then ByRole is not the right tool.

Simply say, you may encounter slowness of ByRole API and/or eventually a timeout occurs when you have a large amount of DOM elements.

[idanen]

I totally agree with @eps1lon , but if there is something to do to improve performance we should give it a try, shouldn't we?
The difference between getByRole and getByLabelText is pretty big (notice in the test I compared to 5 times the duration and it's still longer than that), so I think we should at least make some effort to reduce it

[jihchi]

@idanen For sure, I would definitely love to see any improvements of ByRole API!

[ryuuji3]

I believe that the accessibility checks are nice, but in some cases you don't need them- take for example a UI framework. That's one of the times where you absolutely do want these checks.
But then say you have an application that uses the framework- if the only thing that needs to verified is application behavior, and speed is absolutely important for development.

I think it very much depends on your needs. Not to mention some people have separate pipelines for these checks (eg. lighthouse)