[BUG] Managed async default pool threshold causes allocation stalls #2510

Describe the bug

cuda_async_managed_memory_resource uses CUDA's default managed pool without modifying its release threshold. The CUDA default threshold is zero, which permits synchronization points to release unused pool backing. In an allocation-heavy Velox/cuDF workload, this causes repeated managed allocation, mapping, prefetch, and synchronization stalls. Setting the same pool's release threshold to UINT64_MAX before the first allocation restores warmed performance to the device-async baseline.

This is related to the release-threshold policy proposed for device async pools in #2497, but applies to the separate CUDA 13 managed pool used by cuda_async_managed_memory_resource.

Steps/Code to reproduce bug

Application-level reproducer:

1. Construct cuda_async_managed_memory_resource under CUDA 13.1 and wrap it in prefetch_resource_adaptor.
2. Run an allocation-heavy single-GPU workload with the default managed-pool release threshold.
3. Repeat from a fresh process after setting cudaMemPoolAttrReleaseThreshold on the default managed pool to UINT64_MAX before RMM initializes.
4. Run the workload twice in the same process to measure pool reuse.

The representative query was TPC-H SF1000 Q18. It made 18,788 allocations totaling approximately 1.47 TiB cumulatively, with about 90.8 GiB peak live memory. The prefetch adaptor and cuDF issued 29,460 cudaMemPrefetchAsync calls.

Nsight Systems comparison of fresh managed executions:

CUDA API totals overlap across worker threads and cannot be summed as wall time. Kernel work was unchanged: 11,770 kernels and 3.042 seconds aggregate with the default threshold versus 11,771 kernels and 3.074 seconds with the maximum threshold. Copy volume and copy-engine time were also effectively unchanged, and no managed-memory page faults were recorded.

The threshold was set and verified through the CUDA Driver API:

CUmemLocation location{CU_MEM_LOCATION_TYPE_DEVICE, device};
CUmemoryPool pool{};
cuMemGetDefaultMemPool(&pool, &location, CU_MEM_ALLOCATION_TYPE_MANAGED);

std::uint64_t threshold = std::numeric_limits<std::uint64_t>::max();
cuMemPoolSetAttribute(pool, CU_MEMPOOL_ATTR_RELEASE_THRESHOLD, &threshold);

The full application report is available at https://gist.github.com/bdice/911a8dc2057c71911e867b5f329dd3bf. The three .nsys-rep profiles are available on request.

Expected behavior

cuda_async_managed_memory_resource should avoid pathological pool trimming under allocation-heavy workloads. Possible policies are:

• If the default managed pool's threshold is zero, raise it to std::numeric_limits<std::uint64_t>::max(), matching the default-pool policy discussed in [FEA] Use CUDA's current async pool as RMM's default memory resource #2497 and used by CCCL/cuda-python for device async pools.
• Expose a release-threshold constructor option so applications can select retention explicitly.

The global-state and retained-memory tradeoffs should be documented. A maximum threshold retains the workload's high-water backing memory and may reduce memory available to allocations outside the pool.

Environment details

• Environment: bare metal, Docker, one NVIDIA B200 GPU
• RMM: built from source at f3310cb85b3fe15fd21f550adbbf7eeb1e374588
• CUDA toolkit: 13.1.115
• Nsight Systems: 2025.5.1
• Consumer: Presto native execution with Velox/cuDF
• Workload: single-GPU TPC-H SF1000 Q18
• Worker image: ghcr.io/rapidsai/velox-testing-images@sha256:c4742c090637eed0d9c06809a3864c468269ec1ef0e7cc72121bb6015ae5ea48

Additional context

The maximum threshold does not remove calls made by prefetch_resource_adaptor; it makes allocation and prefetch substantially cheaper when managed-pool backing can be reused. The warmed result indicates that default-pool trimming, rather than GPU kernel execution or transfer bandwidth, accounts for most of the observed slowdown.