[GPU_X] 'cudaErrorNotSupported': 'operation not supported'

[iarspider]

RelVals 29634.402, 29634.403, 29634.404, 29634.406, 29661.402, 29834.402, 29834.403, 29834.404 failed in CMSSW_15_0_GPU_X_2025-02-04-2300 with StdException:

----- Begin Fatal Exception 05-Feb-2025 02:33:03 CET-----------------------
An exception of category 'StdException' occurred while
   [0] Processing  Event run: 1 lumi: 8 event: 703 stream: 3
   [1] Running path 'MC_Ele5_Open_Unseeded'
   [2] Calling method for module HGCalSoARecHitsLayerClustersProducer@alpaka/'hltHgcalSoARecHitsLayerClustersProducer'
Exception Message:
A std::exception was thrown.
/data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/external/alpaka/1.2.0-92470b733c547768aafa79b7bf3f2362/include/alpaka/mem/buf/BufUniformCudaHipRt.hpp(302) 'TApi::mallocAsync( &memPtr, static_cast<std::size_t>(width) * sizeof(TElem), queue.getNativeHandle())' returned error  : 'cudaErrorNotSupported': 'operation not supported'!
----- End Fatal Exception -------------------------------------------------

[iarspider]

assign heterogeneous

[cmsbuild]

New categories assigned: heterogeneous

@fwyzard,@makortel you have been requested to review this Pull request/Issue and eventually sign? Thanks

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @iarspider.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[fwyzard]

Looks like the jobs are running on a 1/8th slice of an H100, with only 1 GB or GPU memory:

CUDA device 0: NVIDIA H100L-1-12C MIG 1g.12gb (sm_90)

Maybe that is not enough for the Phase-2 workflow with 4 concurrent streams ?

@rovere ?

[makortel]

I would have expected out-of-memory lead to a different error code (documentation lists cudaErrorOutOfMemory). For cudaErrorNotSupported the documentation tells

This error indicates the attempted operation is not supported on the current system or device.

According to the exception message the called Alpaka function was TApi::mallocAsync(). Why are we calling mallocAsync()? I thought we are using the caching allocator by default.

[fwyzard]

I think the allocations come from the CLUE external, not from CMSSW.

[fwyzard]

That said, I would expect an H100 to support cudaMallocAsync.

[fwyzard]

Google found a similar report about cudaMallocAsync failing in a MIG environment: kokkos/kokkos#6865 (comment) .

[makortel]

I think the allocations come from the CLUE external, not from CMSSW.

I could be concerned of proliferation of externals frequently using direct CUDA allocation functions (even if via Alpaka). I understand we could have externals for which we won't have control (say ML frameworks), but in this case CLUE has CMS developers, no? Could we think of some allocator abstraction in CLUE for which CMS could provide an implementation that forwards the allocations to our caching allocator?

[fwyzard]

My current approach is to try and "upstream" the caching allocator into Alpaka, so the externals can make use of it.

[fwyzard]

Digging further, I found a mention of a similar issue on the Spark Rapids repository: NVIDIA/spark-rapids#7636 .

Apparently the use of stream ordered memory allocations requires support for Unified Memory (not sure whether it means just the virtual memory address space or full managed memory), though I could not find any documentation in that regard.

And, by default the NVIDIA Virtual GPU Software (vGPU) disables unified memory:

6.4. Enabling Unified Memory for a vGPU

Unified memory is disabled by default. If used, you must enable unified memory individually for each vGPU that requires it by setting a vGPU plugin parameter. How to enable unified memory for a vGPU depends on the hypervisor that you are using.

@iarspider @smuzaffar can you ask IT to enable unified memory on the hypervisor they use for the vGPU H100 nodes ?

@makortel is running on a host without support for unified memory and asynchronous memory allocations something that we want to support ?

[makortel]

is running on a host without support for unified memory and asynchronous memory allocations something that we want to support ?

Good question. In CMSSW code itself we could rather easily (at this point) have a functional setup for a host without unified memory or async memory allocations (I'd believe we are there already :) ). After that, we are constrained by external factors: on one hand if one (or more) of our externals do not support such host (which is the case in this issue), on the other hand if we have to use a system whose administrators refuse to enable the vGPU unified memory support (I have no idea how probably that would be).