Set worker memory limit to current free memory, not total memory?

[gjoseph92]

In coiled/feedback#185, we found that system services were taking up enough memory that the worker setting its memory limit to to the total amount of virtual memory available often caused it to put the system under memory pressure and freeze it before the terminate limit could kick in.

Perhaps dask should take this approach in general, and rather than setting its default memory limit to the total system memory (which is always going to be too high), it should use the amount of currently-available memory (maybe plus its own current RSS)?

We'd just change .total here to .available:

distributed/distributed/system.py

Line 19 in 969aa46

limit = psutil.virtual_memory().total

cc @fjetter @ntabris

[fjetter]

This is certainly interesting. Judging by the docs, this indeed reads as a saner value to pick https://psutil.readthedocs.io/en/latest/index.html#psutil.virtual_memory

total: total physical memory (exclusive swap).
available: the memory that can be given instantly to processes without the system going into swap. This is calculated by summing different memory values depending on the platform and it is supposed to be used to monitor actual memory usage in a cross platform fashion.

What does concern me slightly is that modern OSes are working with things like in-memory compression of pages. I'm wondering how this fits into the picture.

I did a few tests on my local machine (M1 macbook, 16GB total RAM) with this simple snippet. The data size of this example is just enough to stress my system but small enough to run through.

from distributed import Client
from dask.datasets import timeseries
from distributed import performance_report
ddf = timeseries( # ~5.5GB
    start='2020-01-01',
    end='2021-12-31',
)
with performance_report("mem-limit-total"):
    ddf.shuffle(on="x").y.sum().compute()

With the default / total meassure, this spawned four workers with 4GB memory_limit each.
I have a lot of things running in the background so my idle memory usage is already pretty high. In fact, if I use available, I will get about half of that.

Running the default, I see that the shuffle is just being processed as one would expect. There is not a lot of unmanaged memory, there is transfer as one would expected, there is little to no spilling. The memory is comfortably managed since the data size fits nicely into the 16GB total memory.
Wall time of this is about 224s

Now, using available memory, I see memory pressure going up high. The memory limit is set to about 2GB, i.e. total cluster memory is at 8GB which would fit the data in memory once but does not allow multiple copies as we require for a shuffle.
I see a lot of spill to disk and a lot of unmanaged memory. In fact, at times there is almost exlusively unmanaged memory tracked. This looks awful.
Wall time of this is about 117s

We're twice as fast and my system is stressed much less even though all alarms go off. That's a surprising result. I would love somebody with a linux and/or Intel MacBook to confirm these results

perf-reports.zip

[fjetter]

I didn't know about virtual_memory().available but if this performs as I saw right now, this opens doors to a different kind of memory monitoring. Instead of watching RSS and comparing this to a pre-set memory_limit we may only want to watch available memory and pause/spill/terminate based on that.

cc @crusaderky

[fjetter]

An interesting other observation about unmanaged memory: Without spilling, unmanaged memory is almost not there. I'm curious if there is something in our spillage layer that causes too many data copies or holds on to data much longer than it should

[crusaderky]

I didn't know about virtual_memory().available but if this performs as I saw right now, this opens doors to a different kind of memory monitoring. Instead of watching RSS and comparing this to a pre-set memory_limit we may only want to watch available memory and pause/spill/terminate based on that.

I would not want it on a laptop where I prototype my code with a LocalCluster.
My IDE, browser, etc. will use a highly volatile amount of memory and using current virtual memory will cause the whole experience to be highly random and irriproducibile.
Also, swapping out is a healthy thing to do for idling processes - which are the majority on a desktop environment.

On a dedicated, one-service server such as a docker container, this becomes more sensible.
AMM would most likely need to be aware of it.
The GUI would also need a rethinking, or users will spend a lot of time wondering why worker A has paused at 2GB while worker B on the same host got to reach 40GB.

TLDR, I don't think it's a great idea. In another post, I had suggested that, if there is no swap file, dask-worker could assume it's running on a docker container and auto-calculate memory_limit based on a single, initial sample of available virtual memory instead of total memory.

IMPORTANT - if you're going to do a one-off sample when you start dask-worker, you need to make sure it's before you fork, or you'll introduce even more randomness as the nannies spin up at the same time (+/- 200MB per process).

[fjetter]

Just in case my comment above was not clear, by reducing the memory limits to available (about 50% of total) I was able to accelerate my workload be a factor of ~2

available: 117s
total: 224s