Problem Description
Every hipLaunchKernel call crashes with a 0xC0000005 access violation in
hipProfilerRegisterChunkCallbackExt. No Python exception is raised — the
process dies immediately.
Root cause (confirmed via WinDbg live kernel debugging):
hipLaunchKernel+0x85 calls into the profiler callback registry via call r10
(41 FF D2). The callback linked list contains a corrupted/garbage pointer:
rcx = 7b2c450fc892ac84 (garbage — not NULL, not valid usermode address)
crash: mov rax, qword ptr [rcx+8] → [7b2c450fc892ac8c] = ????????
This is a race condition or uninitialized node in the profiler callback
linked list during first kernel dispatch.
Workaround (confirmed working):
NOP out the call instruction at hipLaunchKernel+0x85 in amdhip64_7.dll:
File offset 0x4549B5: 41 FF D2 → 90 90 90
After this patch, all PyTorch training operations work correctly.
FlashAttention backward (aotriton_v2.dll) also crashes separately —
workaround: attn_implementation="eager" in from_pretrained().
Operating System
Windows 11 10.0.26200
CPU
AMD Ryzen 9 7900 12-Core Processor
GPU
AMD Radeon RX 9070 XT (gfx1201)
ROCm Version
7.14.0 (torch 2.12.0+rocm7.14.0, torch.version.hip = 7.14.60850)
ROCm Component
HIP
Steps to Reproduce
- Install ROCm 7.14.0 wheel on Windows 11 with RX 9070 XT (gfx1201)
- Load any PyTorch model to CUDA:
model = AutoModelForCausalLM.from_pretrained(..., device_map={"": 0})
- Run any operation that dispatches a HIP kernel (tensor copy, matmul, etc.)
- Process exits immediately with 0xC0000005, no Python exception caught
WinDbg crash stack:
hipLaunchKernel+0x88
hipRegisterTracerCallback+0x14A0E8 (x5 frames)
hipProfilerRegisterChunkCallbackExt+0x5C257 ← CRASH
mov rax, qword ptr [rcx+8]
rcx = 7b2c450fc892ac84 (garbage pointer)
(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support
Additional Information
The crash is NOT caused by missing gfx1201 kernels (that is a separate issue).
The crash occurs in the profiler/tracer callback dispatch mechanism inside
hipLaunchKernel itself, before any actual compute kernel is launched.
Environment variables ROCPROFILER_DISABLE=1, HSA_TOOLS_LIB="",
HIP_LAUNCH_BLOCKING=1 do NOT fix the issue — the crash is in amdhip64's
internal callback system, not in external profiler tools.
The NOP patch at file offset 0x4549B5 confirms the call site.
Problem Description
Every hipLaunchKernel call crashes with a 0xC0000005 access violation in
hipProfilerRegisterChunkCallbackExt. No Python exception is raised — the
process dies immediately.
Root cause (confirmed via WinDbg live kernel debugging):
hipLaunchKernel+0x85 calls into the profiler callback registry via
call r10(41 FF D2). The callback linked list contains a corrupted/garbage pointer:
rcx = 7b2c450fc892ac84 (garbage — not NULL, not valid usermode address)
crash: mov rax, qword ptr [rcx+8] → [7b2c450fc892ac8c] = ????????
This is a race condition or uninitialized node in the profiler callback
linked list during first kernel dispatch.
Workaround (confirmed working):
NOP out the call instruction at hipLaunchKernel+0x85 in amdhip64_7.dll:
File offset 0x4549B5: 41 FF D2 → 90 90 90
After this patch, all PyTorch training operations work correctly.
FlashAttention backward (aotriton_v2.dll) also crashes separately —
workaround: attn_implementation="eager" in from_pretrained().
Operating System
Windows 11 10.0.26200
CPU
AMD Ryzen 9 7900 12-Core Processor
GPU
AMD Radeon RX 9070 XT (gfx1201)
ROCm Version
7.14.0 (torch 2.12.0+rocm7.14.0, torch.version.hip = 7.14.60850)
ROCm Component
HIP
Steps to Reproduce
model = AutoModelForCausalLM.from_pretrained(..., device_map={"": 0})
WinDbg crash stack:
hipLaunchKernel+0x88
hipRegisterTracerCallback+0x14A0E8 (x5 frames)
hipProfilerRegisterChunkCallbackExt+0x5C257 ← CRASH
mov rax, qword ptr [rcx+8]
rcx = 7b2c450fc892ac84 (garbage pointer)
(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support
Additional Information
The crash is NOT caused by missing gfx1201 kernels (that is a separate issue).
The crash occurs in the profiler/tracer callback dispatch mechanism inside
hipLaunchKernel itself, before any actual compute kernel is launched.
Environment variables ROCPROFILER_DISABLE=1, HSA_TOOLS_LIB="",
HIP_LAUNCH_BLOCKING=1 do NOT fix the issue — the crash is in amdhip64's
internal callback system, not in external profiler tools.
The NOP patch at file offset 0x4549B5 confirms the call site.