.NET Core 3.1 linux .NET process using much more memory than GC.GetGCMemoryInfo().TotalAvailableMemoryBytes

[dave-yotta]

Specifically ASP.NET Core. I've got the GC apparently using more memory than GC.GetGCMemoryInfo().TotalAvailableMemoryBytes. Still investigating but wondered if anyone had any hints.

I've set (though it should be the default) export COMPlus_GCHeapHardLimitPercent=75 as documented.

I've got a container with 1000mb memory limit, I've tried looking at TotalAvailableMemoryBytes in dotnet script on cli exec'ed in - it's 750mb.

But the application (tricky to debug) sails way past this, near 100%, and gets killed my oomkiller. I've done a console writeline of GC.GetGCMemoryInfo().TotalAvailableMemoryBytes in the controller I'm testing with and it's indeed 750mb. But the exceptions just arent raised.

I've got catch(OutOfMemoryException){ Environment.FailFast(); } in exception handler, to terminate and get process dump, in case you're wondering.

Testing on a local kubernetes cluster (DEBUG configuration), I get the OOMException at the expected 750mb. Not in our "staging" cluster (it's AKS if that matters).

Edit: Basically, the memory usage of the .net process is a few hundred MB (400mb?) greater than the total GC size - is there any way to know this non-GC allocation size in order to set correct memory limits?

Edit 2: Even stranger, my local kube cluster running practically the same image (possible env differences) is stopping at about 700mb shown resident in htop - with a 500mb GC heap limit. While in the AKS cluster the same situation the .net process goes up to 1000mb (this is the cgroups mem limit) and gets oomkiler killed. There must be a reason for this unpredictable behaviour?

[akirayamamoto]

I believe we are experiencing this issue as well.