Low memory notifications for GC on Unix

[sergiy-k]

On Windows GC receives Low Memory notifications from the OS. Such functionality does not seem to exist on Unix. We need to investigate ideas how to deliver such information to the GC (user-mode polling strategy?) to help alleviate OOM and/or changes to allocation behaviors to reduce our OOM killer score on the process.

[janvorli]

It looks like there is no low memory notification API in Linux and the only way is to poll /proc/meminfo regularly.

[itsjbe]

The Same thing is happening on LXD. I had a quick chat with @janvorli at WUG Brno last week and he had pointed this issue out. Should I report it as a different issue?

root@brnpreQA:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.2 LTS
Release:        16.04
Codename:       xenial

root@brnpreQA:~# lxc version
2.12

[janvorli]

@jbednarik1 actually, I was thinking about your issue after we've met and I've realized in your case, it might be a different issue that was fixed for 2.0.0. But I am not sure if you've said you were hitting it with 2.0.0 or with previous versions.

[itsjbe]

Hi @janvorli. It seems to me we have solved our issue by setting GC to workstation mode.
Feel free to check https://github.com/dotnet/coreclr/issues/9719, if you'd be interested. Thank you anyway. Cheers!

[janvorli]

We can add the polling into the finalizer thread's FinalizerThread::WaitForFinalizerEvent where we check the low memory notification event for windows. On Linux, the WaitForMultipleObjectsEx already has a timeout that's used for heap dump trigger checks. This timeout is currently set to 10 seconds. We could change it to a lower value (e.g. 100ms - this would need to be determined experimentally based on the impact on the CPU load) and then poll the memory consumption each time it timeouts. We would also need to change the #ifdef to just FEATURE_PAL. If we find that the consumption is approaching the physical memory limit, we would do the same thing that we do for the kLowMemoryNotification event.
We would still want to keep the heap dump trigger to be checked with the original frequency of 10 secs (but only for defined(__linux__) && defined(FEATURE_EVENT_TRACE) to keep the behavior unchanged). This can be easily achieved by a counter counting the timeouts of the wait and checking the g_TriggerHeapDump only when the counter multiplied by the new timeout would reach the 10 secs. Or we can make it more responsive to the g_TriggerHeapDump - instead of checking it on a fixed interval, we would check it on each timeout, unless the previous heap dump was triggered less than 10 secs ago.

For polling the current memory consumption, we can use the PAL_GetWorkingSetSize function.
For getting the physical memory limit, we can use the PAL_GetRestrictedPhysicalMemoryLimit. This function works correctly even in docker container with physical memory limit being set. But it is quite heavy in parsing the cgroups files and so we should read it just once and keep it for the whole lifetime of coreclr.

As for the threshold to trigger GC, we should probably use a percentage of the physical memory limit. We could make the percentage configurable via a new COMPlus_xxxx variable. We can try to set the default to something like 75%.

To test the effect of the change, a simple application that allocates e.g. a byte array of a large size (e.g. 10 MB) in a loop and assigns it to the same variable, waiting for say 0.5 secs between the allocations would probably work. Running this application in a docker container with memory limited to e.g. 100MB on a host system with swap disabled and OOM killer enabled would result in the application getting killed without this change, but should result in its infinite survival with this change.

[kriskalish]

Is there any plan to assign a milestone for this? I believe low memory notifications would resolve issues I'm having while running services in Docker that deal with a high volume of requests.

I'm currently working around it by invoking the garbage collector manually on a schedule.

[alefranz]

Has anything changed around this in .NET Core 3.0?
Looking at the issue tracker I get the impression that some work has been done for docker, but what about regular servers/VMs?
Is there any special setting needed to have the GC invoked and avoid the memory to grow until it get OOM-killed?

[alefranz]

OK, I was wrong, the GC is taking by default into accounts memory limits outside of containers as well, as pointed out to me by @Maoni0 on Twitter https://twitter.com/maoni0/status/1164939641352404992?s=20

[Maoni0]

been talking to @lpereira about the PSI stuff on linux and he believes we can implement the low mem notification on linux that way. CC @janvorli