update handling of GPU workflows in runTheMatrix.py

[fwyzard]

runTheMatrix.py has some GPU-related options:

GPU-related options:
  These options are only meaningful when --gpu is used, and is not set to forbidden.

  --gpu [{forbidden,optional,required}], --requires-gpu [{forbidden,optional,required}]
                        Enable GPU workflows. Possible options are "forbidden" (default), "required" (implied if no argument is given), or "optional". (default: forbidden)
  --gpu-memory GPUMEMORYMB
                        Specify the minimum amount of GPU memory required by the job, in MB. (default: 8000)
  --cuda-capabilities CUDACAPABILITIES
                        Specify a comma-separated list of CUDA "compute capabilities", or GPU hardware architectures, that the job can use. (default: 6.0,6.1,6.2,7.0,7.2,7.5,8.0,8.6)
  --cuda-runtime CUDARUNTIME
                        Specify major and minor version of the CUDA runtime used to build the application. (default: 12.4)
  --force-gpu-name GPUNAME
                        Request a specific GPU model, e.g. "Tesla T4" or "NVIDIA GeForce RTX 2080". The default behaviour is to accept any supported GPU. (default: )
  --force-cuda-driver-version CUDADRIVERVERSION
                        Request a specific CUDA driver version, e.g. 470.57.02. The default behaviour is to accept any supported CUDA driver version. (default: )
  --force-cuda-runtime-version CUDARUNTIMEVERSION
                        Request a specific CUDA runtime version, e.g. 11.4. The default behaviour is to accept any supported CUDA runtime version. (default: )

However, they affect only the creation of WMAgent (?) workflows, not the actual content of the workflow generated by cmsDriver.py and executed by cmsRun.

---

I would like to propose two changes:

1. change the default for the --gpu option from forbidden to optional;
2. propagate the meaning of the --gpu option to cmsDriver, via the --accelerators option.

---

The first change is IMHO something we should do in its own right, but here it is motivated by minimising the impact of the second change on the cmsDriver workflows.

---

The second change proposes to map:

• --gpu optional to the current behaviour, that is, no extra cmsDriver options
• --gpu forbidden to cmsDriver.py --accelerators cpu
• --gpu required to cmsDriver.py --accelerators gpu-*

By default cmsDriver does not impose any restrictions on the usage of GPUs.
Passing --accelerators cpu sets the job's process.options.accelerators to [ 'cpu' ], which prevents the use of GPUs in a CUDA or Alpaka workflow.
Passing --accelerators gpu-* sets the job's process.options.accelerators to [ 'gpu-*' ], which requires the use of GPUs in a CUDA or Alpaka workflow.

The advantage of this approach is that we no longer need to triplicate all Alpaka-related workflows: one version to run on any backend, one version to run only on CPU, one version to run only on GPUs.

---

As this change would affect O&C and PPD operations, what is their opinion ?

[fwyzard]

assign core,pdmv

[cmsbuild]

New categories assigned: core,pdmv

@Dr15Jones,@AdrianoDee,@sunilUIET,@miquork,@makortel,@smuzaffar,@kskovpen you have been requested to review this Pull request/Issue and eventually sign? Thanks

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @fwyzard.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[fwyzard]

@sextonkennedy @srimanob @AdrianoDee FYI

[fwyzard]

@vlimant @malbouis FYI

[fwyzard]

@ckoraka FYI

[makortel]

What is the question here for core?

[fwyzard]

What is the question here for core?

Mhm, does it look OK to map --gpu=required to --accelerators gpu-* to process.options.accelerators = [ 'gpu-*' ] and so on ?

[makortel]

What is the question here for core?

Mhm, does it look OK to map --gpu=required to --accelerators gpu-* to process.options.accelerators = [ 'gpu-*' ] and so on ?

Making the workflow generation (cmsDriver options) and the communication towards WMAgent consistent makes a lot of sense to me. I'd also prefer this way over triplicating the workflows. (then we should probably change the GPU IB to also use --gpu=required. I don't remember what we do now there, maybe we rely on "one version to run only on GPUs" workflows)

But we can comment only from the runTheMatrix.py user side, we have no idea of possible implications elsewhere.

[fwyzard]

Thanks @makortel.

I don't know if other people should be added to the thread.

[fwyzard]

@tulika176 FYI