Configure `scram` to disable one of the GPU bakends in a development area ?

[fwyzard]

The ROCm (and to some extend CUDA) alpaka backends add a noticeable amount to the time it takes to build some packages.

For users that do not care about running on (AMD) GPUs, we could speed up the compilation process disabling the ROCm (or CUDA) alpaka backend(s).

Also note that it could be much worse if we manage to add the SYCL/oneAPI backend...

This could be implemented in scram, with a syntax like

scram b disable-backend {cuda,rocm}
scram b enable-backend {cuda,rocm}

?

An other way to speed up the compilation would be to target only one actual GPU type, like an NVIDIA T4 or an AMD Mi250.

This could be implemented with a syntax like

scram b enable-backend cuda=sm_89
scram b enable-backend rocm=gfx90a

We could also get the hardware type from cudaComputeCapabilities or rocmComputeCapabilities with a syntax like

scram b enable-backend cuda=native
scram b enable-backend rocm=native

@smuzaffar do you think this could be implemented in scram ?

If you think so, we can discuss the implementation detail here or in person.

[fwyzard]

assign core,heterogeneous

[cmsbuild]

New categories assigned: core,heterogeneous

@Dr15Jones,@fwyzard,@makortel,@makortel,@smuzaffar you have been requested to review this Pull request/Issue and eventually sign? Thanks

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @fwyzard.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[smuzaffar]

@fwyzard , yes we should be able to implement this via scram b ..... How about , for local development (e.g. where user only wants to test things on local host) we can just use scram build enable-alpaka-native which on host with

• nvidia gpu: disable rocm, use cudaComputeCapabilities to get the get actual gpu and only build for that gpu type
• amd gpu: disable cuda, use rocmComputeCapabilities to get the get actual gpu and only build for that gpu type
• without gpu: disable both rocm and cuda backends

we can also add scram b {enable|disable}-alpaka-{rocm|cuda} for expicitly enable/disable rocm/cuda backend build

If needed, we can discuss this in core sw meeting tomorrow

[fwyzard]

If needed, we can discuss this in core sw meeting tomorrow

Sounds good.

[fwyzard]

About updating the flags in cuda.xml and rocm.xml tools.

cuda.xml

The syntax for enabling sm_## is -gencode arch=compute_##,code=[sm_##,compute_##].
So, calling e.g.

scram b enable-backend cuda=sm_89

should remove all the CUDA_FLAGS of the form -gencode arch=compute_[0-9]+,code=[sm_[0-9]+,compute_[0-9]+], and add -gencode arch=compute_89,code=[sm_89,compute_89].

The "native" CUDA architectures used by the NVIDIA GPUs in the local machine can be extracted from cudaComputeCapabilities:

$ cudaComputeCapabilities 
   0     8.9    NVIDIA L4
   1     7.5    Tesla T4

should use the architecture sm_75.

Currently there is a script cmsCudaSetup.sh, that does part of what scram b enable-backend cuda=native should do.

rocm.xml

The syntax for enabling gfx#### is --offload-arch=gfx####, so

scram b enable-backend rocm=gfx1100

should remove all the ROCM_FLAGS of the form --offload-arch=gfx[0-9a-f]+, and add --offload-arch=gfx1100.

Note that the value after gfx can have 3 or 4 hexadecimal digits.

The "native" ROCm architectures used by the AMD GPUs in the local machine can be extracted from rocmComputeCapabilities:

$ rocmComputeCapabilities 
   0     gfx1100    AMD Radeon Pro W7800 (unsupported)

[smuzaffar]

@fwyzard , thanks for the hints in #45859 (comment).
As scram build ... passes every thing to gmake as build targets so it is not easy to implement scram build enable-backend cuda as in this case cuda becomes a build target and gmake will try to run it OR scram build enable-backend cuda=sm_89 in this case cuda becomes a variable overriding its value set by cuda tool. Instead how about

• scram build {en,dis}able-backend-{cuda,rocm}: To enable/disable cuda/rocm alpaka backends
• scram build enable-backend-{cuda,rocm}-[comma-separated-compute-capabilities] e.g
  • scram build enable-backend-cuda-sm_75 or scram build enable-backend-cuda-sm_75,sm_89
  • scram build enable-backend-rocm-gfx1100 or scram build enable-backend-rocm-gfx1100,gfx90a
  • scram build enable-backend-cuda-native: To find the native compute capabilities and use those
  • scram build enable-backend-cuda-reset: To reset the compute capabilities to their original value ( from the release area)
• scram build enable-backend-native: To disable the backend not available and call enable-backend-cuda-native for the backend which is available

[fwyzard]

I see.

Maybe we could shorten the commands, like

• scram build {en,dis}able-{cuda,rocm}
• scram build enable-cuda-sm_75
• scram build enable-rocm-gfx1100,gfx90a

etc?

And it might be more clear if we split the backend and individual targets with a :

• scram build enable-cuda:sm_75
• scram build enable-rocm:gfx1100,gfx90a

(I would suggest using = but Make would interpret it as setting a variable)

What do you think ?

[smuzaffar]

sounds good, so I will drop -backend from the target and use : for the compute capabilities

[smuzaffar]

@fwyzard , for now I have enable-alpaka:native to automatically enable/disable cuda/rocm backend and set the native compute capabilities. Is this a good target name of should I change it to enable-alpaka-native ( enable-native sounds very generic )

[fwyzard]

Maybe enable-gpus:native ?