Skip to content

[release/2.7] support experimental CU carveout #2700

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jithunnair-amd merged 2 commits into from
Oct 8, 2025

[ROCm] fix carveout feature (#164303)

47700f3
Select commit
Loading
Failed to load commit list.
Merged

[release/2.7] support experimental CU carveout #2700

[ROCm] fix carveout feature (#164303)
47700f3
Select commit
Loading
Failed to load commit list.
ROCm Repo Management API / Tests / Tests / Test Distributed / Run pytorch_distributed_2 failed Oct 8, 2025 in 0s

TestDistBackendWithSpawn.test_ddp_apply_optim_in_backward_ignored_params failed

TestDistBackendWithSpawn.test_ddp_apply_optim_in_backward_ignored_params failed

Details

TestDistBackendWithSpawn.test_ddp_apply_optim_in_backward_ignored_params

AssertionError: Scalars are not equal!

Expected 0 but got -6.
Absolute difference: 6
Relative difference: inf
Expected zero exit code but got -6 for pid: 550066
Stack trace 
Traceback (most recent call last):
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 605, in wrapper
    self._join_processes(fn)
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 845, in _join_processes
    self._check_return_codes(elapsed_time)
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 920, in _check_return_codes
    self.assertEqual(
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 4123, in assertEqual
    raise error_metas.pop()[0].to_error(  # type: ignore[index]
AssertionError: Scalars are not equal!

Expected 0 but got -6.
Absolute difference: 6
Relative difference: inf
Expected zero exit code but got -6 for pid: 550066
Standard error 
/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/backends/cudnn/__init__.py:144: UserWarning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (Triggered internally at /var/lib/jenkins/pytorch/torch/csrc/cuda/Module.cpp:1920.)
  torch._C._cuda_set_cudnn_benchmark_limit(_benchmark_limit)
View more details on ROCm Repo Management API