Skip to content

Commit 612e5c0

Browse files
[ROCm] Fix MIOpen CTC loss crash on Windows (pytorch#179264)
<h2>Fix MIOpen CTC loss access violation on Windows discrete GPUs</h2> <h3>Problem</h3> <p>A failing unit test on Windows started showing a couple weeks ago and a missing <code>#include</code> was added in [](pytorch#178284), but CI on TheRock kept failing. The fix was tested on gfx1151 (APU), where the test passed, but CI showed failures on gfx1100. </p> <p><code>test_CTCLoss_no_batch_dim</code> (and any code path hitting <code>miopen_ctc_loss</code>) crashes with a fatal access violation on Windows systems with discrete AMD GPUs:</p> <pre><code>Windows fatal exception: access violation Exception Code: 0xC0000005 #0 miopen::CTCLossDescriptor::GetCTCLossWorkspaceSize (MIOpen.dll+0x14fde4) #1 miopenGetCTCLossWorkspaceSize (MIOpen.dll+0x150912) #2 at::native::miopen_ctc_loss (torch_hip.dll) </code></pre> <h3>Root Cause</h3> <p><code>miopenGetCTCLossWorkspaceSize</code> and <code>miopenCTCLoss</code> read the <code>labels</code>, <code>label_lengths</code>, and <code>input_lengths</code> arrays <strong>on the host side</strong> to plan the computation and calculate workspace requirements. The existing code copies these arrays to GPU memory and passes device pointers:</p> <pre><code>Tensor labels_gpu = targets_t.to(Device(at::kCUDA), at::kInt); // ... hipMemcpy to GPU ... MIOPEN_CHECK(miopenGetCTCLossWorkspaceSize(..., labels_gpu.data_ptr&lt;int&gt;(), // device pointer label_lengths_gpu.data_ptr&lt;int&gt;(), // device pointer input_lengths_gpu.data_ptr&lt;int&gt;() // device pointer )); </code></pre> <p>This works on:</p> <ul> <li><strong>Linux</strong> — HSA (Heterogeneous System Architecture) maps GPU allocations into the process virtual address space, making device pointers host-readable</li> <li><strong>Windows APUs</strong> — CPU and iGPU share system RAM, so device pointers point to host-accessible memory</li> </ul> <p>This crashes on:</p> <ul> <li><strong>Windows dGPUs</strong> — GPU has dedicated VRAM across PCIe; device pointers are opaque handles that cannot be dereferenced from host code</li> </ul> <h3>Verification</h3> <p>Tested on gfx1201:</p> <table border="1" cellpadding="6" cellspacing="0"> <tr><th>Check</th><th>Result</th></tr> <tr><td><code>hipDeviceAttributeIntegrated</code></td><td><code>0</code> (discrete GPU)</td></tr> <tr><td><code>hipDeviceAttributeCanUseHostPointerForRegisteredMem</code></td><td><code>0</code></td></tr> <tr><td><code>hipDeviceAttributeManagedMemory</code></td><td><code>0x7FFFFFFF</code> (unsupported)</td></tr> <tr><td><code>hipDeviceAttributeUnifiedAddressing</code></td><td><code>0x7FFFFFFF</code> (unsupported)</td></tr> <tr><td>Host read of <code>hipMalloc</code> pointer via <code>ctypes</code></td><td>Access violation</td></tr> <tr><td>CTC loss with CPU pointers</td><td>Pass (forward + backward)</td></tr> </table> <h3>Fix</h3> <p>Use host pointers since this is what MIOpen expects should be used.</p> <h3>Testing</h3> <p>Run all existing CTCLoss unit tests.</p> Pull Request resolved: pytorch#179264 Approved by: https://github.com/jeffdaily Co-authored-by: Jeff Daily <jeff.daily@amd.com>
1 parent 291f197 commit 612e5c0

1 file changed

Lines changed: 7 additions & 26 deletions

File tree

aten/src/ATen/native/miopen/LossCTC_miopen.cpp

Lines changed: 7 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -207,35 +207,16 @@ std::tuple<Tensor, Tensor> miopen_ctc_loss(
207207
Tensor costs = at::empty({batch_size}, log_probs->options());
208208
Tensor grad = at::empty_like(log_probs_t, LEGACY_CONTIGUOUS_MEMORY_FORMAT);
209209

210-
// MIOpen requires labels and lengths on GPU
211-
Tensor labels_gpu = targets_t.to(Device(at::kCUDA), at::kInt);
212-
Tensor label_lengths_gpu = at::empty(
213-
{static_cast<int64_t>(target_lengths.size())},
214-
at::TensorOptions().dtype(at::kInt).device(at::kCUDA));
215-
Tensor input_lengths_gpu = at::empty(
216-
{static_cast<int64_t>(input_lengths.size())},
217-
at::TensorOptions().dtype(at::kInt).device(at::kCUDA));
218-
219-
C10_CUDA_CHECK(hipMemcpy(
220-
label_lengths_gpu.data_ptr<int>(),
221-
target_lengths.data(),
222-
target_lengths.size() * sizeof(int),
223-
hipMemcpyHostToDevice));
224-
C10_CUDA_CHECK(hipMemcpy(
225-
input_lengths_gpu.data_ptr<int>(),
226-
input_lengths.data(),
227-
input_lengths.size() * sizeof(int),
228-
hipMemcpyHostToDevice));
229-
210+
// MIOpen reads labels/lengths on the host.
230211
size_t workspace_size;
231212
(void)deterministic; // MIOpen only supports deterministic algorithm
232213
MIOPEN_CHECK(miopenGetCTCLossWorkspaceSize(
233214
handle,
234215
probs_desc,
235216
grads_desc,
236-
labels_gpu.data_ptr<int>(),
237-
label_lengths_gpu.data_ptr<int>(),
238-
input_lengths_gpu.data_ptr<int>(),
217+
targets_t.data_ptr<int>(),
218+
target_lengths.data(),
219+
input_lengths.data(),
239220
MIOPEN_CTC_LOSS_ALGO_DETERMINISTIC,
240221
ctc_desc,
241222
&workspace_size));
@@ -246,9 +227,9 @@ std::tuple<Tensor, Tensor> miopen_ctc_loss(
246227
handle,
247228
probs_desc,
248229
log_probs_t.data_ptr(),
249-
labels_gpu.data_ptr<int>(),
250-
label_lengths_gpu.data_ptr<int>(),
251-
input_lengths_gpu.data_ptr<int>(),
230+
targets_t.data_ptr<int>(),
231+
target_lengths.data(),
232+
input_lengths.data(),
252233
costs.data_ptr(),
253234
grads_desc,
254235
grad.data_ptr(),

0 commit comments

Comments
 (0)