Cannot gather keys error -- memory-related issue? #21
Description
I ran into the followign error while distributing a simple function:
025-11-05 22:01:54,709 - distributed.client - WARNING - Couldn't gather 3369 keys, rescheduling
The function was returning a very large dictionary, (1000*3 keys, from 13k file). I managed to overcome the error by reorgnaising my dictionary and switching intenral keys to tuples of values to reduce memory consumption. I managed to get a dummy reproducible of my problem in the jupyter notebook attached here.
The issue appears when running method 1, but not when running method 2. I tested this on coffea-casa @unl using 500 workers (max in adaptive scaling).
Warning running method 1 may crash your server!!