[Issue]: Discrepancy between hipGetDeviceProperties and rocminfo

[Rombur]

Problem Description

In AMD's documentation, it says that Maximum x-, y- or z-dimension of a grid is 2^{32}-1. These are also the values you get when using rocminfo. hipGetDeviceProperties used to return the same values but it was changed in 31b362b. Which values should I use?

Operating System

NA

CPU

NA

GPU

MI 250

ROCm Version

ROCm 6.3

ROCm Component

No response

Steps to Reproduce

No response

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

No response

Additional Information

No response

[Rombur]

From what I can tell the change was made to accommodate hipModuleLaunchKernel. The GridDim size can be larger if you use hipLaunchKernel. It seems wrong to return a smaller value of GridDim size just because of the requirement of one function. Especially when hipModuleLaunchKernel is only used for some Fortran compatibility stuffs.

[dalg24]

Could we get please some guidance from AMD? This is impacting Kokkos.
cc @crtrott @lucbv

[Rombur]

I just realized that the values actually never matched. The documentation says that the maximum number is 2^{32}-1 but hipGetDeviceProperties returns std::numeric_limits<int32_t>::max() for GridDim.x which is 2^{31}-1. It should probably be uint32t_t instead of int32_t.

[darren-amd]

Hi @Rombur,

I had a chat with the internal team and the recommendation is to use the values from hipGetDeviceProperties, namely: 2^{31} - 1, 2^{16} - 1, and 2^{16} - 1 for the x, y, and z dimensions. This is in line with CUDA: Technical Specifications per Compute Capability. I am working on getting the documentation updated to reflect this change. Thanks!

[Rombur]

@darren-amd Thanks for the update. Will rocminfo also be update, so that we don't get conflicting information?

[darren-amd]

Hi @Rombur,

Yes, I have an internal PR to get rocminfo updated as well.

[Rombur]

Great. Thank you @darren-amd

[dalg24]

What version of ROCm did the fix land in?