Repeated calls to `memory_color` take around 12% of CPU time of scheduler

[jonded94]

Describe the issue:

Some follow-up to: #8761

After fixing above issue already in #8762, the next big thing that takes very much CPU power with a scheduler with lots of workers (>2000), are the calls to _cluster_memory_color, more specifically _memory_color.

distributed/distributed/dashboard/components/scheduler.py

Line 391 in 782050a

def _cluster_memory_color(self) -> str:

As far as I can see, this is about coloring the memory bar of a specific worker depending if it's deemed "good", "almost full" or "full".

Again, speedscope stuff (this was without the fix from PR 8762):

speedscope.json

Is this something that could be solved by binning the memory load & size (surely coloring doesn't have to be so exact that is has to be based on exact bytes of memory) and caching the result of this memory coloring process too?

Surely, one don't has to recalculate which color a worker process with for example 1024/4096MiB RAM shall have hundreds of times per second, especially since the coloring result doesn't change at all.

Environment:

• Dask version: 2024.7.0
• Python version: 3.10
• Operating System: Linux, Debian
• Install method (conda, pip, source): poetry / pip

[jonded94]

Ah, so what takes time here is asking the scheduler what the current memory situation across all workers is. This also happens on the next biggest call, taking around 7.0% CPU time:

distributed/distributed/dashboard/components/scheduler.py

Line 413 in 782050a

meminfo = self.scheduler.memory

So there are at least two separate calls which result in measuring the total cluster memory situation, is that correct? Is that something that one could cache at least per tick or even calculate only one every 10 ticks or so?

[jonded94]

Other tangential question: Can the tick rate of the dashboard be adjusted somehow? I would be totally fine with an update every 10s or so, I'm mostly using internal Grafana dashboards fueled from the Prometheus metrics anyways.
On top of that, we have very long running tasks, so frequent updates are not even helping.

[jonded94]

There is --no-show for entirely disabling the dashboard, but this parameter is apparently unused :(

distributed/distributed/cli/dask_scheduler.py

Line 127 in 782050a

show,

Using --no-dashboard will disable metrics too, which is undesirable :/ We'd want prometheus metrics that we can look at asynchronously.

[fjetter]

Other tangential question: Can the tick rate of the dashboard be adjusted somehow? I would be totally fine with an update every 10s or so, I'm mostly using internal Grafana dashboards fueled from the Prometheus metrics anyways.
On top of that, we have very long running tasks, so frequent updates are not even helping.

This is not currently exposed. It's also a bit messy since the update interval for every widget is controlled individually, see for example

distributed/distributed/dashboard/scheduler.py

Lines 74 to 116 in 8564dc7

	"/individual-task-stream": individual_doc(
	TaskStream, 100, n_rectangles=1000, clear_interval="10s"
	),
	"/individual-progress": individual_doc(TaskProgress, 100, height=160),
	"/individual-graph": individual_doc(TaskGraph, 200),
	"/individual-groups": individual_doc(TaskGroupGraph, 200),
	"/individual-group-progress": individual_doc(TaskGroupProgress, 200),
	"/individual-workers-memory": individual_doc(WorkersMemory, 100),
	"/individual-cluster-memory": individual_doc(ClusterMemory, 100),
	"/individual-workers-transfer-bytes": individual_doc(WorkersTransferBytes, 100),
	"/individual-cpu": individual_doc(CurrentLoad, 100, fig_attr="cpu_figure"),
	"/individual-nprocessing": individual_doc(
	CurrentLoad, 100, fig_attr="processing_figure"
	),
	"/individual-occupancy": individual_doc(Occupancy, 100),
	"/individual-workers": individual_doc(WorkerTable, 500),
	"/individual-exceptions": individual_doc(ExceptionsTable, 1000),
	"/individual-bandwidth-types": individual_doc(BandwidthTypes, 500),
	"/individual-bandwidth-workers": individual_doc(BandwidthWorkers, 500),
	"/individual-workers-network": individual_doc(
	WorkerNetworkBandwidth, 500, fig_attr="bandwidth"
	),
	"/individual-workers-disk": individual_doc(
	WorkerNetworkBandwidth, 500, fig_attr="disk"
	),
	"/individual-workers-network-timeseries": individual_doc(
	SystemTimeseries, 500, fig_attr="bandwidth"
	),
	"/individual-workers-cpu-timeseries": individual_doc(
	SystemTimeseries, 500, fig_attr="cpu"
	),
	"/individual-workers-memory-timeseries": individual_doc(
	SystemTimeseries, 500, fig_attr="memory"
	),
	"/individual-workers-disk-timeseries": individual_doc(
	SystemTimeseries, 500, fig_attr="disk"
	),
	"/individual-memory-by-key": individual_doc(MemoryByKey, 500),
	"/individual-compute-time-per-key": individual_doc(ComputePerKey, 500),
	"/individual-aggregate-time-per-action": individual_doc(AggregateAction, 500),
	"/individual-scheduler-system": individual_doc(SystemMonitor, 500),
	"/individual-contention": individual_doc(Contention, 500),
	"/individual-fine-performance-metrics": individual_doc(FinePerformanceMetrics, 500),

However, some docs are hard coding this, e.g. system_monitor here

distributed/distributed/dashboard/components/scheduler.py

Lines 4543 to 4551 in 8564dc7

	def systemmonitor_doc(scheduler, extra, doc):
	sysmon = SystemMonitor(scheduler, sizing_mode="stretch_both")
	doc.title = "Dask: Scheduler System Monitor"
	add_periodic_callback(doc, sysmon, 500)
	doc.add_root(sysmon.root)
	doc.template = env.get_template("simple.html")
	doc.template_variables.update(extra)
	doc.theme = BOKEH_THEME

Using --no-dashboard will disable metrics too, which is undesirable :/ We'd want prometheus metrics that we can look at asynchronously.

No, metrics are still available. At least when I try they are. You can check by opening http://localhost:<dashboard_port>/metrics

[jonded94]

This is not currently exposed. It's also a bit messy since the update interval for every widget is controlled individually

Hm, would be nice if that could be controlled somehow, but yeah, one would have to spend a bit of time here to make all of these values configurable, right?
Maybe one could include a single configurable parameter that we'd multiply/divide all of these update intervals by?

No, metrics are still available.

Ah, sorry! Just started a pipeline with --no-dashboard, the metrics indeed seem to be there. I recall that this was the case for the worker dashboards, it seemed metrics only became available when you'd enable worker dashboards.

I can keep you updated as soon as I start a really large pipeline again what the scheduler health then looks like and where it's spending a lot of compute.

[jonded94]

Hey @fjetter ! We tried a bunch of even larger scale workloads (>3k workers), tried to optimize our stack a bit by not submitting too many tasks at once, disabled the dashboard (--no-dashboard) and got the clusters to be quite stable :)

With these measures, we saw scheduler tick durations of ~3-30ms and scheduler CPU load of ~5-35%, with latter being on the lower side more often.

That is quite okay for us right now. I have to say though that enabling the dashboard and actually using it/having it open in a browser will lead to the scheduler being 100% pegged almost immediately and also the dashboard loosing connection/not updating anymore quite often.

EDIT: Nevertheless I'd still be interested in either slowing down dashboard updates through a config or improving the performance of the compute-heavy parts of the scheduler. Let me know if you know a nice place to start improving something!