GPU gets counted for each worker

[nyck33]

As suggested by your moderator on the forums:

https://dask.discourse.group/t/why-am-i-seeing-16-gib-memory-when-i-have-an-nvidia-geforce-gtx-1650-gpu/1891

[jrbourbeau]

Thanks for the issue @nyck33. @charlesbluca @quasiben is this something you, or someone else, has bandwidth to look into?

[quasiben]

My guess is that each worker is reporting 4 GBs of available memory and the dashboard logic is adding all the info together. @charlesbluca do think this is what's going on here ?

[charlesbluca]

Yup, think that's exactly what's happening here - my understanding for GPU monitoring when creating a standard LocalCluster on a system with GPUs available is that all workers get mapped to whatever the first GPU in CUDA_VISIBLE_DEVICES is since we aren't making any modifications to this variable between worker processes, and then from there this GPU's total memory is double counted for each of the 4 workers:

distributed/distributed/dashboard/components/nvml.py

Line 136 in 2d4ff46

memory_total += mem_total

Wonder if there's some information available to us through the NVML diagnostics so that we can decide whether to treat the GPU(s) associated with the workers as distinct or not? cc @pentschev @jacobtomlinson not sure if this is something that's been tried before since I believe this has been Dask's behavior around GPU monitoring for a while

[jacobtomlinson]

Yeah this double counting is kinda expected when not using dask-cuda. LocalCluster will start one worker per CPU core, and each worker will be able to see the GPU. We could make things more intelligent in distributed but then we are getting close to duplicating what dask-cuda does.

We could potentially collect the UUID of the GPU and pass that back to the scheduler, then do some deduplication. But I'm not sure how beneficial that would be because users using LocalCluster probably aren't using the GPUs, and if they did want to use the GPUs they would use LocalCUDACluster instead which works as expected.