Dashboard memory/CPU increases when dashboard is used, making scheduler unresponsive

[mazar]

What happened:

Running Dask-Scheduler on a dedicated host, with dashboard enabled and used, results in CPU utilization increase to the point where scheduler slows down to a halt.

What you expected to happen:

Using dashboard should not result in so much CPU utilization, resulting in unresponsive scheduler.

If dashboard is not intended to be used in a production environment, a warning must be added to documentation (or even the CLI) to inform the users.

Minimal Complete Verifiable Example:

1. Use any recent release (at least for the past year, including main dev branch).
2. Install in a venv to make sure all requirements are installed with the most recent available version
3. Start dask-scheduler with no extra CLI parameters
4. Don't even need to start a worker, just scheduler by itself is enough to reproduce the problem
5. Visit the dashboard, browse to multiple pages
6. Observe that with each page click, memory and CPU utilization increase

Anything else we need to know?:

In a production environment, we have to constantly monitor the scheduler machine and restart dask-scheduler due to this issue.
Not sure if it helps to know: when CPU usage goes above 50%, scheduler stops doing some optimizations: https://github.com/dask/distributed/blob/main/distributed/scheduler.py#L8048

Environment:

• Dask version: Many, including latest 2022.4.0
• Python version: Tested with 3.8.8 and 3.9.7
• Operating System: Tested with Ubuntu 18.04 and MacOS 12.3.1
• Install method (conda, pip, source): tested with both pip and source

[fjetter]

Hi @mazar ,

Sorry for the late reply.

I can reproduce some of what you are describing. I get my CPU util up to about 10-15% when the scheduler is idling which is obviously a bit much when nothing is actually happening. Can you tell us a bit more about what you are seeing? Is there any particular plot that spikes your CPU usage?

Can you have a look at <dashboard-address>/individual-profile-server once your CPU spikes and tell us what you can see there?

[mazar]

Using 2022.5.0 on MacOS, Python 3.9.7, after letting the scheduler run for about 9-10 minutes, with browser pointed to /system page, I see memory and CPU slowly rising:

And the /individual-profile-server looks like this:

Is there a way to dump all details from this profiler page?

[mazar]

After letting it run for another 20 minutes:

[fjetter]

Thanks, this is helpful. It appears this has been introduced a long time ago in #4561

[quasiben]

@charlesbluca if you have a few minutes can you take a look at this ?

[charlesbluca]

Yeah I'm digging into what's causing this

[charlesbluca]

@fjetter you were able to verify that without dask/dask#4561 these issues aren't occuring? I'm trying things out with that commit reverted and still seem to be seeing gradual CPU/memory increases

[fjetter]

So far, I couldn't reproduce it myself. I suspected dask/dask#4561 because this is where the line that is popping up in the profile above was introduced.

If I let my scheduler running for ~10min, the CPU load doesn't go up significantly. There are two things I'm suspecting

• The amount of data for the system monitor is just too large and we should decrease sampling rate
• The memory formatter function is too slow and applying it to all values is just not fast enough

---

@mazar when generating the above profile, did you run any computation? Did you connect any workers?

[mazar]

@fjetter No computations and no workers were present for the above profile.