The root locality is also a regular worker locality, while it has more stuff to do. It may be good to relief it from some worker tasks so it can focus more on distributing tasks.
Verify the result by looking at the trace of a model with local operations. Currently gasp occur at worker localities because the root locality is too busy computing.
(If things don't improve, it may be insightful to test dynamically querying for locality IDs instead of caching these, as we do currently. Maybe this introduces additional asynchrony which can be used to distribute tasks.)
The root locality is also a regular worker locality, while it has more stuff to do. It may be good to relief it from some worker tasks so it can focus more on distributing tasks.
Verify the result by looking at the trace of a model with local operations. Currently gasp occur at worker localities because the root locality is too busy computing.
(If things don't improve, it may be insightful to test dynamically querying for locality IDs instead of caching these, as we do currently. Maybe this introduces additional asynchrony which can be used to distribute tasks.)