Worker count prometheus metric should expose disjoint counters

[fjetter]

The current prometheus metric that exposes number workers known to the scheduler is ill defined.

distributed/distributed/http/scheduler/prometheus/core.py

Lines 35 to 42 in 3e793f7

	worker_states = GaugeMetricFamily(
	self.build_name("workers"),
	"Number of workers known by scheduler",
	labels=["state"],
	)
	worker_states.add_metric(["connected"], len(self.server.workers))
	worker_states.add_metric(["saturated"], len(self.server.saturated))
	worker_states.add_metric(["idle"], len(self.server.idle))

It's current definition is something like

connected is the number of all connected workers
idle/saturated is the number of workers in the respective state

but these sets are not disjoint. Saturated and idle are disjoint and both are subsets of connected.
Ideally, this metric would expose gauges s.t. summing all labels would yield the total number of workers similar to how we expose task states

distributed/distributed/http/scheduler/prometheus/core.py

Lines 76 to 78 in 3e793f7

	for state in ALL_TASK_STATES:
	if state != "forgotten":
	tasks.add_metric([state], task_counter.get(state, 0.0))

The implementation should ideally not iterate over all workers every time the metrics are collected but instead the scheduler should maintain the count online.

[fjetter]

A bit of additional context, the definition of saturated/idle is non trivial and a bit messy

https://github.com/dask/distributed/blob/9f72645081a3128a3b8b01bc368b3e8edd990ae1/distributed/scheduler.py#L2867-L287271

There is a relation that spells our roughly as

len(scheduler.workers) == len(scheduler.idle) + len(schduler.saturated) + <busy workers - does not relate to an attribute>

and I think we should expose the right hand side of this equation

[mercyo12]

@fjetter hi, could I make a PR for this issue? thanks!

[hendrikmakait]

@mercyo12: Please go ahead and let us know if you hit any problems! :)