Question: how to make re-usable server override without killing the running time?

[SimonGodefroid]

We are using helpers to stop using jest.mocks and replace this by re-usable server.use overrides.

e.g.

export const renderForUserTypeA = () => {
  server.use(
    rest.get<any, User>('/users/:userId', (req, res, ctx) => {
      const args = [ctx.status(200), ctx.json(userTypeAFixture)];
      return res(...args);
    })
  );
  return render(<App userType="a" />);
};

So in the tests we'd have

describe('my feature',()=>{
  describe('for user type A', async ()=>{
    it('does this behavior', ()=>{
    await renderForUserTypeA()
    ...
    })
  })
})

This works but since doing this we noticed that locally the tests are taking over twice as much time to run (because now we no longer high-jack the API calls with jest mocks). On the pipeline this causes the tests to take way longer to execute and eventually testing library selectors end up failing due to timeout.

The problem is that the right way to test is using MSW + testing library, but its impacting resources in such a way that this makes the whole pipeline very flakey.

Likewise at the moment we're removing user related mocks but if we were to override another endpoint on a regular base how can we achieve this in a scalable way?

We tried a different approach, in which we still have global handlers to manage API calls in a generic way but when trying to make re-usable local overrides we faced the same problem.

Testing feature x for user type a.
Testing feature x for user type b with condition 1.
Testing feature x for user type c with condition 2.

const myResourceHandler = ({data}: {data: Fixture})=>{
  server.use(
    rest.get('myResource/:resourceId, (req, res, ctx) => {
      const args = [ctx.status(200), ctx.json(data)];
      return res(...args);
    })
}
...


describe('Testing feature x',()=>{
  describe('for user type a',()=>{
    it('does this', ()=>{
      renderForUserTypeA();
      myResourceHandler({ data });
     expect(screen.get...); 
    })
  }) 
  describe('for user type b',()=>{
    it('does the other thing in condition 1',()=>{
      renderForUserTypeB();
       myResourceHandler({ data: dataCondition1 });
       expect(screen.get...); 
    });
    it('does the other thing in condition 2',()=>{
       renderForUserTypeB();
       myResourceHandler({ data: dataCondition2 });
       expect(screen.get...); 
    })
  }) 
  describe('for user type c',()=>{
      renderForUserTypeC();
...
  }) 

What's the correct way to handle these cases without having a dramatic impact on resource consumption and running time?

[kettanaito]

Hey, @SimonGodefroid.

Using runtime handler overrides does not have any impact on runtime performance. It's literally a list.unshift() call to add a new list of handlers to the global overrides.

What often happens when migrating from jest.mock() is that people forget to await HTTP calls correctly. Since mocking a module likely results in immediate responses, that's no longer the case once you actually allow your requests to happen. This would be my first recommendation: ensure that you're awaiting asynchronous actions in your tests. This may mean using .findBy*, or waitFor(), or any other means that ensure stable expectations towards async code.

Overall, do you have some metrics/logs to share? Can you profile your test runs and see what starts consuming more resources? It'd be great to have concrete criteria to have a productive discussion.

[kettanaito]

Also, make sure you apply overrides first and then render your component. It seems you have it vice versa in your example:

renderForUserTypeA();
// If this applies "server.use()", move it above the render function!
myResourceHandler({ data });
expect(screen.get...); 

[SimonGodefroid]

Yup my bad, typing the snippet by memory here so made this mistake. It's in the correct order in our tests (this would throw if it was as is I guess) cheers for highlighting.

[SimonGodefroid]

Hey @kettanaito so I made sure indeed the things are properly awaited for and globally respect RTL guidelines. I've measured as follows:
It's not very relatable for your because I don't share code but it's from my job so I can't.
Running the test suites on master which is the version of the code that relies on mocks vs
Running the test suites on a branch where the jest.mock have been removed
NB: both master and the branch use the renderForUserTypeA method, it's just that in the branch I've removed the mocking of the methods (e.g. fetchUser and the module jest.mock('./modules/fetchUser');). (I guess someone didn't notice that the code wasn't going through the msw's server.use override because of the mocks).

So running both branches locally the one using the renderer method with the msw handler takes over twice as much.

[SimonGodefroid]

Pulling the data by hand, for this test file, by running the test command and pasting the outcome to a code editor,

I manage to pull this data table which is each test's running time.

Which I can turn into this graph.

Branch	Master
64.965 s, 759 MB heap size	27.244 s, 468 MB heap size

[kettanaito]

Thank you for a profound analytics on this!

Did you notice anything growing exponentially in the memory heap snapshot? It'd help us see if there are any memory issues (like #1194, which is still open). Which version of Node.js are you using?

[kettanaito]

Also, it's worth mentioning that tests may start to take a little more time. This is related to the point of actually running the request modules as opposed to jest.mock where they are mocked. But that should be a rather small difference, I would still treat serious performance degradations as issues.

[SimonGodefroid]

No problem thanks for your answer. I had taken part of that discussion on jest's issue here :).

Well now I think about it, it's natural that tests are taking way more time...
In a jest.mock world:

1. render
2. expect(await screen.findByRole('button', {name:'save'});

In a msw world:

1. render
2. await waitForElementToBeRemoved(()=>screen.getByText('Loading...'));
3. expect(await screen.findByRole('button', {name:'save'});

In jest world you always have the answer straightaway from your mock (hence no call to e.g. useQuery) and you end up being in a loading state for way less time than in a world with an actual API call being made and intercepted.

I'm trying to play with CI settings to see whether it either mitigates the flakyness and/or increase test speed.

[kettanaito]

Adding that waitFor steps is how we should test our software. Responses take time in the real world, and the more our tests resemble that real world, the more useful they are.

Missing the awaiting step is a sure recipe for flakiness.