[BUG] GPU reduction failed on GPUs without peer access #314
Description
Describe the bug
We have seen an assertion https://github.com/StanfordLegion/realm/blob/main/src/realm/cuda/cuda_internal.cc#L2365 when running GPU reductions on multiple GPUs when peer access is not enabled.
Expected behavior
If peer access is not enable across GPUs, we should not pick the GPU reduction channel for the reduction operation, instead, the data should be staged through the host buffer.
Desktop (please complete the following information):
- GPU: two 4090