Issue with DeviceResource (`cudaMallocAsync`) on vGPU without memory pool support

[swahtz]

When attempting to use the AWS EC2 g6f instance type for CI in fvdb-core (working prototype on this branch), we receive an error at the first call to cudaMallocAsync at nanovdb/cuda/DeviceResource.h:22: CUDA error 801: operation not supported

Through some investigation, it appears that the stream-ordered memory pools capability is not exposed by the vGPU (g6f is a fractional L4 GPU instance). Confirmed when running:

> #include <cuda_runtime.h>
> #include <stdio.h>
>
> int main() {
>   int supported = 0, driver = 0;
>   cudaDriverGetVersion(&driver);
>   cudaError_t e = cudaDeviceGetAttribute(
>       &supported, cudaDevAttrMemoryPoolsSupported, 0);
>   printf("driver=%d, query=%s, memoryPoolsSupported=%d\n",
>          driver, cudaGetErrorString(e), supported);
>   return e != cudaSuccess || !supported;
> }

driver=13000, query=no error, memoryPoolsSupported=0

These instances are running the latest AWS-supplied GRID driver (version 19.5/580.159.03) and can confirm the GPU device is available:

>nvidia-smi
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.159.03             Driver Version: 580.159.03     CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA L4-3Q                   On  |   00000000:31:00.0 Off |                    0 |
| N/A   N/A    P0            N/A  /  N/A  |       0MiB /   3072MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

From reading, it appears that memory pools can be enabled/disabled via hypervisor settings on the vGPU's setup. This seems likely to be disabled due to security concerns; I have opened up an AWS support ticket to confirm if this behaviour is intentional.

Regardless, support for this operating scenario (driver/hardware/CUDA support for cudaMallocAsync but unavailable capability) would require utilizing cudaMalloc (or a custom allocator) instead. This is a related issue in the work for #2232 and allowing NanoVDB to use an alternative allocator to enable successful execution on a vGPU configured in this way could be acceptance criteria for that work.

[harrism]

There's an immediate unblock available: NANOVDB_USE_SYNC_CUDA_MALLOC (in nanovdb/util/cuda/Util.h), which redefines the util::cuda::mallocAsync/freeAsync wrappers as plain cudaMalloc/cudaFree. It was added for exactly this scenario — a vGPU ask in #1798 led to PR #1799, which was superseded by the implementation that shipped in #1807. Every device allocation in NanoVDB routes through those wrappers (DeviceResource — including the DeviceResource.h line in your report — DeviceBuffer, GridHandle.cuh, NodeManager, and the CUDA tools), so defining the macro covers the whole library. Define it project-wide so all TUs agree (the wrappers are inline, so inconsistent definitions across TUs are an ODR violation): e.g. -DNANOVDB_USE_SYNC_CUDA_MALLOC in your CI toolchain. That should get the g6f branch running today.

One adjacent caveat for that environment: these vGPU configurations typically disable unified memory as well, so cuda::UnifiedBuffer (cudaMallocManaged) paths will fail independently of this fix — worth avoiding in the same CI jobs.

Beyond the macro, two follow-ups are planned under #2232:

1. Runtime fallback (small standalone PR): the wrappers gain a cached per-device cudaDevAttrMemoryPoolsSupported query — the same attribute your repro checks — and fall back to cudaMalloc/cudaFree when pools are unavailable. One binary then serves both environments with no build flags; error 801 becomes "works, synchronously." A synchronous allocation is immediately valid on every stream, so this preserves the wrappers' stream-ordered contract rather than weakening it.
2. The injectable answer: step 2 of NanoVDB: injectable CUDA memory resources — design & roadmap #2232 adds a synchronous device resource and the sync/async dispatch in the builders, so environments like this can inject a cudaMalloc-backed resource explicitly rather than relying on fallback. Adopting your framing: "runs on pool-less vGPU via an injected synchronous resource" is now an acceptance criterion on NanoVDB: injectable CUDA memory resources — design & roadmap #2232.

Once the runtime-fallback PR is up, it would be great to validate it on your ci/g6f_instances branch — that hardware configuration is the real test.

[harrism]

The runtime-fallback PR is up: #2256. On a pool-less device the wrappers now select cudaMalloc/cudaFree automatically (cached per-device cudaDevAttrMemoryPoolsSupported query), so no build flags are needed. It would be great to get a validation run on the ci/g6f_instances branch — that hardware is the real test for this.

[swahtz]

Got this response from AWS support:

I understand you are experiencing CUDA error 801 ("operation not supported") when calling cudaMallocAsync on a g6f.xlarge instance in us-east-2. Additionally, cudaDeviceGetAttribute(cudaDevAttrMemoryPoolsSupported) returns 0, indicating that CUDA memory pools (cudaMallocAsync, cudaFreeAsync, cudaMemPool_t) are not available, despite using the AWS-provided NVIDIA GRID driver 19.5/580.159.03 with CUDA runtime 13.0.

The behavior you are observing is expected and by design. It stems from an inherent limitation of NVIDIA's vGPU (Virtual GPU) architecture used on G6f instances.

Your g6f.xlarge instance receives a 1/8 fractional GPU slice (3 GB out of 24 GB total L4 GPU memory) through NVIDIA's time-sliced vGPU partitioning. CUDA stream-ordered memory allocation relies on low-level GPU Virtual Memory Management (VMM) capabilities that require direct, exclusive control over GPU memory hardware. This conflicts with the vGPU hypervisor's memory isolation model, which partitions and manages GPU resources across multiple virtual GPU instances.

As a result, the vGPU driver does not expose memory pool capabilities to the guest, and cudaDevAttrMemoryPoolsSupported correctly reports 0.

Key Points:

• There is no current or planned GRID driver release that enables CUDA memory pools on fractional time-sliced vGPU profiles. This is an architectural constraint, not a driver gap.
• This limitation is documented by NVIDIA: https://docs.nvidia.com/vgpu/latest/grid-vgpu-release-notes-generic-linux-kvm/index.html#unified-memory-support-limitations

Migrate to a full-GPU instance like G6 (recommended if cudaMallocAsync is required).

I hope this clarifies the technical constraints and provides viable paths forward. Please feel free to reach out if you have any further questions or concerns.

[harrism]

For pool-less instances like g6f, CI can set NANOVDB_USE_SYNC_CUDA_MALLOC automatically rather than hard-coding it per instance type. A CMake try_run at configure time compiles and runs the same cudaDevAttrMemoryPoolsSupported check from the report above and defines the macro when it returns 0:

# runs on the configuring machine — valid when build and test share the GPU instance
try_run(pools_run pools_compile ${CMAKE_BINARY_DIR}/probe ${CMAKE_SOURCE_DIR}/cmake/probe_memory_pools.cu
        RUN_OUTPUT_VARIABLE pools_out)
if(pools_compile AND pools_run EQUAL 0 AND pools_out MATCHES "memoryPoolsSupported=0")
  add_compile_definitions(NANOVDB_USE_SYNC_CUDA_MALLOC)
endif()

Two notes: try_run executes on the machine doing the configure, so this is correct when the build and the tests run on the same GPU instance (as in the g6f job) — for a split build-node/test-node pipeline, run the probe as a step on the GPU node instead and pass the flag through. And because the wrappers are inline, the macro must be defined for the whole build (a global compile definition), not per-target. This keeps the library honest — it fails loudly on a misconfigured build — while the deployer opts into synchronous allocation based on detected capability.