High load in browser on 7.10.x with many channels/1:1 conversations

[andreaswolf]

Description:

After upgrading from 7.5.x to 7.10.3, we see a high load (100% CPU) in the browser for both Chrome/Chromium and Firefox; for Firefox on Linux, the process name in top is "Isolated Web Container", for Chromium, it is "chromium"

Steps to reproduce:

1. Open RocketChat
2. Click around a bit
3. The chat becomes unresponsive (only on Firefox), to the point where also the developer tools don't respond anymore
4. The high load drops as soon as I close the tab

Expected behavior:

Definitely not 100% CPU load sustained for multiple minutes

Actual behavior:

CPU load stays at ~100%

Server Setup Information:

• Version of Rocket.Chat Server: 7.10.3
• License Type: Enterprise
• Number of Users: ~5000
• Operating System: Linux (Debian) on the server
• Deployment Method: tar
• Number of Running Instances: 1
• DB Replicaset Oplog:
• NodeJS Version: 22.16.0
• MongoDB Version: 6.0

Client Setup Information

• Desktop App or Browser Version: Firefox/Chromium most recent versions
• Operating System: Linux/Windows

Additional context

Relevant logs:

[andreaswolf]

I have two hypotheses why this happens: either it's only relevant for admin users, or the reason is a larger number of joined channels/1:1 conversations. I tend to the latter, since normal users with only one or two dozen communications don't seem to be affected, and our admins have a really long roster.

[reetp]

I'd imagine this is your issue:

Number of Running Instances: 1

You probably need multiple instances to load balance.

https://docs.rocket.chat/docs/scaling-rocketchat

You can do that with EE (you cannot run multiple instances with CE)

[andreaswolf]

@reetp how can that potentially help me with our client side performance issues? The server is idling at a load of < 1.

[reetp]

Please don't @ people. it isn't necessary, and I don't work here.

Regarding your server setup I'm not trying to be difficult. It's just elimination. I do suggest you check the recommended deployment methods and modify yours. It is recommended, especially with the new NATS stuff. Do the logs show anything untoward?

The next is your reverse proxy setup? Just to check. Again, elimination. Anything in the logs?

After that I know have seen general performance issues on FF with the new "AI" stuff in the browser - may we worth disabling it all to test. I have also seen users with performance complaints under recent versions of Chromium/e as well, but no specific reason.

https://github.com/mozilla/policy-templates/blob/master/docs/index.md#generativeai

On FF I think the settings to look for are:

browser.ml.chat.enabled, browser.ml.linkPreview.optin, browser.tabs.groups.smart.userEnabled

Have you tried on a different browser or the Electron client?

Not withstanding any of that, I suspect this issue may be a duplicate of this (always worth a search before opening an issue):

#36643

FWIW the OP there has a large multi instance setup.

If you have EE you can of course open a bug with Rocket themselves as well.

[andreaswolf]

I reacted to your comment, that's why I @-ed you. Seemed logical to me, but I'll refrain from that now.

As for your other points:

1. High number of channels in the user end causes client to slow down excessively #36643 seems to be exactly the same issue. I tried finding similary issues, but with ~2.6k open that's not always easy. I was also pretty confident at first that this is a relatively new issue, since I did not know that it only happens for lots of channels/conversations.
2. I disabled all settings you mentioned; no noticeable change – the browser is still largely unresponsive. A reload fixed that for a few seconds, but switching channels let everything freeze again
3. Of course I can try to completely change our deployment now, but to me it seems a clear issue within the browser. The OP of High number of channels in the user end causes client to slow down excessively #36643 using a multi-node setup, and still experiencing the same contributes to that.
4. I haven't fleeced the server-side logs yet; again, I see little point here since this is pretty surely an issue in the client-side code.
5. I'll try loading our live data locally now to reproduce the issue
6. One thing I see though (in Chromium only) is a few "414 URL too long" errors for presence-related endpoints (need to look again what endpoint exactly; anyways, I'll try exchanging our ancient Apache reverse proxy by Træfik now.)

[andreaswolf]

Ad 5: I imported our production data into a simplified setup locally, and indeed the performance seems to be a lot better.

This might also be due to the downgrade to community – initially, the performance was as bad as on the production instance, but after a short while, it improved (definitely after I confirmed "this is a new workspace" in the dialog that appeared).
That's why I suspect it's probably related to an enterprise feature that got disabled when the license was removed after I confirmed it's a new workspace.

[Gummikavalier]

Please note that if you are hit by this:
#36643
the issue can be worked around by user just by hiding excess amounts of channels in the users channel bar, and then refreshing the browser window. Just to confirm the issue in question really is the same one.

Disabling AI features in Firefox did not help with it, unfortunately, but that was also worth a try.