Description
Connections work great, until they don't.
I'm having a hard time diagnosing what is happening and wish to get some help please.
The application averages ~10GB memory usage, and after a random amount of time, sometimes days, it will spike up and consume almost all the memory it has available to it, in my case, the docker container has a limit of 72 GB.
Sometimes this will crash the application with OOM, other times this will simply grind it to an almost halt and becomes unresponsive or very slow and requires a manual restart.
A memory dump opened in dotMemory:
Context:
The application connects to ~1200 gameservers.
Each gameserver has 6x subscriptions.
At the request of the hosting provider, each gameserver needs its own websocket connection for load balancing purposes.
The application registers GraphQLHttpClient as transient in DI
We routinely flush all connections creating new objects every 45 minutes, again, at the request from the server.
Lib Version: 6.1.0
Dotnet Runtime: .NET 9.0.2
Operating System: Unix 6.8.0.51 x64
Hardware: Bare Metal, 2x AMD EPYC™ 7282 Gen 2 @ 3,20GHZ, 128 GB RAM
Container: Docker