Description
I did some profiling on larger clusters (see also #8725) and noticed the construction of MemoryState to take a non-trivial amount of time despite it being defined as using slots.
This shows up particularly in places where the full state object is not required, e.g.
- https://github.com/fjetter/distributed/blob/4c426a545e35c03acc6533b81a1ec8aa3ad7a664/distributed/dashboard/components/scheduler.py#L394
- https://github.com/fjetter/distributed/blob/4c426a545e35c03acc6533b81a1ec8aa3ad7a664/distributed/dashboard/components/scheduler.py#L4258
In https://github.com/fjetter/distributed/blob/4c426a545e35c03acc6533b81a1ec8aa3ad7a664/distributed/scheduler.py#L1849 we are creating possibly hundreds of those objects just to throw them away right away.
I think if we wanted to maintain such a MemoryState object we should ensure that the instances are being updated instead of recreating them on demand given that those dashboards update every 100ms.
Overall this is just 3% of CPU time in my measurement (about the same size as #8725) so it is not urgent.