feature request: runTheMatrix.py should assign a different GPU to each job

[fwyzard]

runTheMatrix.py creates and executes jobs without any kind of GPU assignment.

On a machine with a single GPU, his is not an issue.

On a machine with more than one GPU, for example

$ rocmComputeCapabilities 
   0    gfx90a:sramecc+:xnack-    AMD Instinct MI250X
   1    gfx90a:sramecc+:xnack-    AMD Instinct MI250X
   2    gfx90a:sramecc+:xnack-    AMD Instinct MI250X
   3    gfx90a:sramecc+:xnack-    AMD Instinct MI250X
   4    gfx90a:sramecc+:xnack-    AMD Instinct MI250X
   5    gfx90a:sramecc+:xnack-    AMD Instinct MI250X
   6    gfx90a:sramecc+:xnack-    AMD Instinct MI250X
   7    gfx90a:sramecc+:xnack-    AMD Instinct MI250X

or

$ cudaComputeCapabilities 
   0     8.9    NVIDIA L4
   1     8.9    NVIDIA L4
   2     8.9    NVIDIA L4
   3     8.9    NVIDIA L4

the result is that all jobs try to use all GPUs, which is quite inefficient.

A better approach would be to assign a different GPU to each job, for example in a round-robin fashion.
If there are more concurrent jobs than GPUs, the GPUs will be shared - but to a much lesser extent than now.

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @fwyzard.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

assign Configuration/PyReleaseValidation

[makortel]

assign heterogeneous

[cmsbuild]

New categories assigned: pdmv,upgrade,heterogeneous

@AdrianoDee,@DickyChant,@fwyzard,@makortel,@miquork,@Moanwar,@srimanob,@subirsarkar you have been requested to review this Pull request/Issue and eventually sign? Thanks

[AdrianoDee]

Apart from running something like CUDA_VISIBLE_DEVICES=X cmsRun config.py do we have any mechanism in cmssw to tell a job to use a specific GPU?

[makortel]

Apart from running something like CUDA_VISIBLE_DEVICES=X cmsRun config.py do we have any mechanism in cmssw to tell a job to use a specific GPU?

No, CUDA_VISIBLE_DEVICES on NVIDIA (and one of HIP_VISIBLE_DEVICES, GPU_DEVICE_ORDINAL, ROCR_VISIBLE_DEVICES on AMD; it's not really clear to me which one would be the best choice (link)) would be the way to go (at least presently).

[AdrianoDee]

And follow-up question: how do we count the GPUs available on a node? I imagine we should do this outside any cmsRun and within runTheMatrix.py. What I've found with a dummy StackOverflow search is something like nvidia-smi -L | wc -l (for NVIDIA, didn't check for AMD).

[makortel]

I would think of building on top of our cudaComputeCapabilities program, which will include (unsupported) in the printout if a GPU is not supported by us (for whatever reason), that might be useful to exclude by runTheMatrix.py

cmssw/HeterogeneousCore/CUDAUtilities/bin/cudaComputeCapabilities.cpp

Lines 25 to 27 in 69407d5

	if (not isCudaDeviceSupported(i)) {
	std::cout << " (unsupported)";
	}

(similarly for rocmComputeCapabilities for AMD)