INIT or FIRST on MI300X

[ianbmacdonald]

Figured it was worth a spin during an idle moment; Even if just to peek at the 910.5GB GTT allocated to each NUMA island reachable via llvmpipe.

I hit CTRL-C after a few minutes.

Ubuntu 24.04.3

https://github.com/GpuZelenograd/memtest_vulkan v0.5.0 by GpuZelenograd
To finish testing use Ctrl+C

1: Bus=0x00:00 DevId=0x0000   1821GB llvmpipe (LLVM 20.1.2, 256 bits)
^C
memtest_vulkan: INIT OR FIRST testing failed due to runtime error

# dmesg | grep amdgpu | grep memory
[ 149.221603] [drm] amdgpu: 196048M of VRAM memory ready
[ 149.221606] [drm] amdgpu: 932325M of GTT memory ready.
[ 151.074052] amdgpu: HMM registered 196048MB device memory
[ 153.810224] [drm] amdgpu: 196048M of VRAM memory ready
[ 153.810229] [drm] amdgpu: 932325M of GTT memory ready.
[ 155.664940] amdgpu: HMM registered 196048MB device memory
[ 158.375524] [drm] amdgpu: 196048M of VRAM memory ready
[ 158.375529] [drm] amdgpu: 932325M of GTT memory ready.
[ 160.496592] amdgpu: HMM registered 196048MB device memory
[ 163.224584] [drm] amdgpu: 196048M of VRAM memory ready
[ 163.224589] [drm] amdgpu: 932325M of GTT memory ready.
[ 165.341745] amdgpu: HMM registered 196048MB device memory
[ 168.205036] [drm] amdgpu: 196048M of VRAM memory ready
[ 168.205039] [drm] amdgpu: 932325M of GTT memory ready.
[ 170.187421] amdgpu: HMM registered 196048MB device memory
[ 173.205868] [drm] amdgpu: 196048M of VRAM memory ready
[ 173.205873] [drm] amdgpu: 932325M of GTT memory ready.
[ 175.239164] amdgpu: HMM registered 196048MB device memory
[ 178.107449] [drm] amdgpu: 196048M of VRAM memory ready
[ 178.107454] [drm] amdgpu: 932325M of GTT memory ready.
[ 180.274584] amdgpu: HMM registered 196048MB device memory
[ 183.166721] [drm] amdgpu: 196048M of VRAM memory ready
[ 183.166723] [drm] amdgpu: 932325M of GTT memory ready.
[ 185.336213] amdgpu: HMM registered 196048MB device memory
# amd-smi

+------------------------------------------------------------------------------+
| AMD-SMI 26.0.0+37d158ab      amdgpu version: 6.14.14  ROCm version: 7.0.1    |
| Platform: Linux Guest                                                        |
|-------------------------------------+----------------------------------------|
| BDF                        GPU-Name | Mem-Uti   Temp   UEC       Power-Usage |
| GPU  HIP-ID  OAM-ID  Partition-Mode | GFX-Uti    Fan               Mem-Usage |
|=====================================+========================================|
| 0002:00:00.0 AMD Instinct MI300X VF | 0 %      40 °C   0           140/750 W |
|   0       0       2        SPX/NPS1 | 0 %        N/A           282/196048 MB |
|-------------------------------------+----------------------------------------|
| 0003:00:00.0 AMD Instinct MI300X VF | 0 %      37 °C   0           141/750 W |
|   1       1       1        SPX/NPS1 | 0 %        N/A           282/196048 MB |
|-------------------------------------+----------------------------------------|
| 0004:00:00.0 AMD Instinct MI300X VF | 0 %      39 °C   0           136/750 W |
|   2       2       0        SPX/NPS1 | 0 %        N/A           282/196048 MB |
|-------------------------------------+----------------------------------------|
| 0005:00:00.0 AMD Instinct MI300X VF | 0 %      40 °C   0           142/750 W |
|   3       3       3        SPX/NPS1 | 0 %        N/A           282/196048 MB |
|-------------------------------------+----------------------------------------|
| 0006:00:00.0 AMD Instinct MI300X VF | 0 %      38 °C   0           138/750 W |
|   4       4       7        SPX/NPS1 | 0 %        N/A           282/196048 MB |
|-------------------------------------+----------------------------------------|
| 0007:00:00.0 AMD Instinct MI300X VF | 0 %      40 °C   0           137/750 W |
|   5       5       5        SPX/NPS1 | 0 %        N/A           282/196048 MB |
|-------------------------------------+----------------------------------------|
| 0008:00:00.0 AMD Instinct MI300X VF | 0 %      39 °C   0           136/750 W |
|   6       6       4        SPX/NPS1 | 0 %        N/A           282/196048 MB |
|-------------------------------------+----------------------------------------|
| 0009:00:00.0 AMD Instinct MI300X VF | 0 %      39 °C   0           141/750 W |
|   7       7       6        SPX/NPS1 | 0 %        N/A           282/196048 MB |
+-------------------------------------+----------------------------------------+
+------------------------------------------------------------------------------+
| Processes:                                                                   |
|  GPU        PID  Process Name          GTT_MEM  VRAM_MEM  MEM_USAGE     CU % |
|==============================================================================|
|  No running processes found                                                  |
+------------------------------------------------------------------------------+

[ianbmacdonald]

# ./memtest_vulkan_verbose > output.txt
ERROR:             loader_validate_layers: Layer 0 does not exist in the list of available layers
ERROR:             loader_validate_layers: Layer 0 does not exist in the list of available layers
ERROR:             loader_validate_layers: Layer 0 does not exist in the list of available layers
ERROR:             loader_validate_layers: Layer 0 does not exist in the list of available layers
ERROR:             loader_validate_layers: Layer 0 does not exist in the list of available layers
ERROR:             loader_validate_layers: Layer 0 does not exist in the list of available layers
ERROR:             loader_validate_layers: Layer 0 does not exist in the list of available layers
...

output.txt

[galkinvv]

1: Bus=0x00:00 DevId=0x0000 1821GB llvmpipe (LLVM 20.1.2, 256 bits)

Seems that only software-one implementaition runnign on CPU is available, the AMD .icd is not even suggested, like a "non-functional AMD driver installation", neither AMDVLK nor RADV.

Try running as "VK_LOADER_DEBUG=all ./memtest_vulkan" and provide its output. Maybe the vulkan loader tell some info about icd's it tries. According to dmesg log amd kernel driver is reprobing devices on each attempt, so there is something done by vulkan loader, but eventually it does not list MI300X as a discovered vulkan device.

Now regarding the software-only llvmpipe:

• the free VRAM reported by llvmpipe budget 1820.9GB usage 39.0GB should be ~related to the output of the free command I suppose. Please provide the free command output.
• however, actual allocation seems fails even at 1683.858GB with ERROR_OUT_OF_DEVICE_MEMORY while executing vk1_0::Buffer create_buffer. And then it retries with a "0.5GB" smaller size, which is obviousely too slow guessing algorithm for such huge VRAM amount. This should be resolved on the memtest_vulkan's side by using non-linear search strategy for available memory, smilar to radv/amdgpu: Failed to allocate a buffer #16 (comment)