CI: Add ROCm nightly docker workflow by leo-automation · Pull Request #3115 · ROCm/pytorch

leo-automation · 2026-03-31T14:20:59Z

Motivation

Migrating from Jenkins. This PR adds ROCm-specific nightly docker automation so we can build and validate a dedicated ROCm nightly image outside the standard PyTorch docker release flow

Technical Details

Adds .github/workflows/pytorch-nightly-docker.yml to build and push a ROCm nightly image, support an optional rocm_version override, and run a ROCm smoke-test job against the produced image.
Adds .ci/docker/pytorch-nightly-docker.Dockerfile to build PyTorch, torchvision, and torchaudio on top of the ROCm nightly base image.

Test Result

jithunnair-amd · 2026-04-14T17:34:08Z

+        working-directory: .ci/docker
+        run: |
+          export PYTORCH_ROCM_ARCH="${{ env.PYTORCH_ROCM_ARCH }}"
+          ./build.sh pytorch-linux-jammy-rocm${{ env.ROCM_VERSION }}-py${{ env.PYTHON_VERSION }} \


Suggested change

./build.sh pytorch-linux-jammy-rocm${{ env.ROCM_VERSION }}-py${{ env.PYTHON_VERSION }} \

./build.sh pytorch-linux-jammy-rocm-n-py3 \

along with a sed in the build.sh to replace with ROCM_VERSION we want to build in this workflow: https://github.com/pytorch/pytorch/blob/5e73467572a7b3a5508f1216cedf5fe6f1ac5ce0/.ci/docker/build.sh#L162

jithunnair-amd · 2026-04-14T17:40:00Z

+      - .github/scripts/rocm_nightly_debug_build.sh
+
+env:
+  ROCM_VERSION: '7.2.1'


Suggested change

ROCM_VERSION: '7.2.1'

ROCM_VERSION: '7.2.2'

jithunnair-amd · 2026-04-14T17:53:11Z

@leo-automation to check if increasing MTU size on runner nodes might resolve network issues such as https://github.com/ROCm/pytorch/actions/runs/24400425702/job/71269225000

rocm-repo-management-api · 2026-04-23T11:58:39Z

Jenkins build for 807c7a11d7e2982b3c5e653b91aa4b61b61879a2 commit finished as NOT_BUILT
Links: Pipeline Overview / Build artifacts / Test Results

jithunnair-amd · 2026-04-23T13:26:59Z

+ENV CI=1
+ENV PYTORCH_TEST_WITH_ROCM=1
+ENV PYTORCH_TESTING_DEVICE_ONLY_FOR="cuda"
+ENV USE_NVSHMEM=0


@leo-automation Please add a comment stating that this is TODO and TEMPORARY and a reason why it's there

jithunnair-amd · 2026-04-23T13:29:19Z

+RUN git clone https://github.com/pytorch/pytorch --recursive \
+    && cd pytorch \
+    # Bypass sccache on torch_rocshmem: its -fgpu-rdc + mixed xnack± offload-arch flags break sccache's argv parser.
+    && sed -i 's|set_target_properties(torch_rocshmem PROPERTIES LINKER_LANGUAGE HIP)|set_target_properties(torch_rocshmem PROPERTIES LINKER_LANGUAGE HIP CXX_COMPILER_LAUNCHER "" HIP_COMPILER_LAUNCHER "")|' caffe2/CMakeLists.txt \


@leo-automation Do we still need this if we have the USE_NVSHMEM=0 above? If not, we could leave this in a comment if you don't want to lose it

jithunnair-amd · 2026-04-23T13:30:51Z

+    && git config --local user.email "amd@amd.com" \
+    && git remote add rocm https://github.com/ROCm/pytorch.git \
+    && git fetch rocm \
+    && git cherry-pick 519160d466782f5a62365be051fcb3ef90fa0b00 \


@leo-automation Do we need this as well?

jithunnair-amd · 2026-04-23T13:46:09Z

+env:
+  ROCM_VERSION: '7.2.2'
+  PYTHON_VERSION: '3.10'
+  PYTORCH_ROCM_ARCH: 'gfx906;gfx908;gfx90a;gfx942;gfx950;gfx1030;gfx1100;gfx1101;gfx1102;gfx1150;gfx1151;gfx1200;gfx1201'


@pruthvistony @jeffdaily @pragupta @jerrymannil @jataylo Since these images are primarily for dev use, do we really need to build for all archs by default? We can always build for a specific arch if needed, by manually triggering the workflow (would need to ensure that manual triggers use a different docker image tag so it doesn't overwrite the cron-based ones). I propose we build only for the most-commonly required GFX archs for dev use.
My proposal: "gfx90a gfx942 gfx950"
Any Radeon ones needed?

rocm-repo-management-api · 2026-04-23T16:01:06Z

Jenkins build for 807c7a11d7e2982b3c5e653b91aa4b61b61879a2 commit finished as FAILURE
Links: Pipeline Overview / Build artifacts / Test Results

rocm-repo-management-api · 2026-04-24T13:18:34Z

Jenkins build for fb1c009dec41f8dd2083d9e28be2fb7714fe464e commit finished as NOT_BUILT
Links: Pipeline Overview / Build artifacts / Test Results

rocm-repo-management-api · 2026-04-24T14:05:36Z

Jenkins build for 20df855ecfe6114da39c5403157eba72e5f49853 commit finished as NOT_BUILT
Links: Pipeline Overview / Build artifacts / Test Results

rocm-repo-management-api · 2026-04-24T16:04:57Z

Jenkins build for 20df855ecfe6114da39c5403157eba72e5f49853 commit finished as FAILURE
Links: Pipeline Overview / Build artifacts / Test Results

rocm-repo-management-api · 2026-04-27T13:18:04Z

Jenkins build for 077f47cf15cb4d2fb62b9f3fb8f7de17520b5650 commit finished as FAILURE
Links: Pipeline Overview / Build artifacts / Test Results

rocm-repo-management-api · 2026-04-27T17:16:56Z

Jenkins build for 012f03505ff5cafb0aceb085dcc30230540c51a9 commit finished as FAILURE
Links: Pipeline Overview / Build artifacts / Test Results

rocm-repo-management-api · 2026-04-28T14:38:42Z

Jenkins build for 88ec33078145d1c899001ecbc60a0bce195af050 commit finished as FAILURE
Links: Pipeline Overview / Build artifacts / Test Results

Detected error during Pytorch building:

[7710/8176] Building CXX object caffe2/CMakeFiles/torch_hip.dir/__/torch/csrc/distributed/c10d/FlightRecorderCuda.cpp.o
cc1plus: warning: command-line option ‘-Wno-duplicate-decl-specifier’ is valid for C/ObjC but not for C++
[7711/8176] Building CXX object caffe2/CMakeFiles/stride_properties_test.dir/__/aten/src/ATen/test/stride_properties_test.cpp.o
[7712/8176] Building CXX object caffe2/CMakeFiles/cpu_profiling_allocator_test.dir/__/aten/src/ATen/test/cpu_profiling_allocator_test.cpp.o
[7713/8176] Building CXX object caffe2/CMakeFiles/memory_overlapping_test.dir/__/aten/src/ATen/test/memory_overlapping_test.cpp.o
FAILED: caffe2/CMakeFiles/memory_overlapping_test.dir/__/aten/src/ATen/test/memory_overlapping_test.cpp.o 
/opt/cache/bin/sccache /opt/cache/bin/c++ -DHAVE_MALLOC_USABLE_SIZE=1 -DHAVE_MMAP=1 -DHAVE_POSIX_FALLOCATE=1 -DHAVE_SHM_OPEN=1 -DHAVE_SHM_UNLINK=1 -DHIPBLASLT_USE_ROCROLLER -DIDEEP_USE_MKL -DMINIZ_DISABLE_ZIP_READER_CRC32_CHECKS -DONNXIFI_ENABLE_EXT=1 -DONNX_ML=1 -DONNX_NAMESPACE=onnx_torch -DROCM_VERSION=70202 -DTORCH_HIP_VERSION=702 -DUSE_C10D_GLOO -DUSE_C10D_NCCL -DUSE_DISTRIBUTED -DUSE_EXTERNAL_MZCRC -DUSE_LAYERNORM_FAST_RECIPROCAL -DUSE_PROF_API=1 -DUSE_RPC -DUSE_TENSORPIPE -D_FILE_OFFSET_BITS=64 -D__HIP_PLATFORM_AMD__ -D__HIP_PLATFORM_AMD__=1 -I/var/lib/jenkins/pytorch/build/aten/src -I/var/lib/jenkins/pytorch/aten/src -I/var/lib/jenkins/pytorch/build -I/var/lib/jenkins/pytorch -I/var/lib/jenkins/pytorch/nlohmann -I/var/lib/jenkins/pytorch/moodycamel -I/var/lib/jenkins/pytorch/build/include -I/var/lib/jenkins/pytorch/build/caffe2/aten/src -I/var/lib/jenkins/pytorch/aten/src/ATen/.. -I/var/lib/jenkins/pytorch/third_party/miniz-3.0.2 -I/var/lib/jenkins/pytorch/torch/csrc/api -I/var/lib/jenkins/pytorch/torch/csrc/api/include -I/var/lib/jenkins/pytorch/c10/.. -I/var/lib/jenkins/pytorch/c10/hip/../.. -isystem /opt/rocm-7.2.2/include -isystem /var/lib/jenkins/pytorch/build/third_party/gloo -isystem /var/lib/jenkins/pytorch/cmake/../third_party/gloo -isystem /var/lib/jenkins/pytorch/cmake/../third_party/tensorpipe/third_party/libuv/include -isystem /var/lib/jenkins/pytorch/cmake/../third_party/googletest/googlemock/include -isystem /var/lib/jenkins/pytorch/cmake/../third_party/googletest/googletest/include -isystem /var/lib/jenkins/pytorch/third_party/protobuf/src -isystem /opt/conda/envs/py_3.12/include -isystem /var/lib/jenkins/pytorch/third_party/XNNPACK/include -isystem /var/lib/jenkins/pytorch/third_party/ittapi/include -isystem /var/lib/jenkins/pytorch/cmake/../third_party/eigen -isystem /opt/rocm/include -isystem /var/lib/jenkins/pytorch/third_party/ideep/mkl-dnn/include/oneapi/dnnl -isystem /var/lib/jenkins/pytorch/third_party/ideep/include -isystem /var/lib/jenkins/pytorch/INTERFACE -isystem /var/lib/jenkins/pytorch/third_party/nlohmann/include -isystem /var/lib/jenkins/pytorch/third_party/concurrentqueue -isystem /opt/rocm-7.2.2/include/hiprand -isystem /opt/rocm-7.2.2/include/rocrand -isystem /var/lib/jenkins/pytorch/third_party/googletest/googletest/include -isystem /var/lib/jenkins/pytorch/third_party/googletest/googletest -isystem /var/lib/jenkins/pytorch/third_party/googletest/googlemock/include -isystem /var/lib/jenkins/pytorch/third_party/googletest/googlemock -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_MSLK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -DC10_NODEPRECATED -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -faligned-new -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-dangling-reference -Wno-error=dangling-reference -Wno-stringop-overflow -DHAVE_AVX512_CPU_DEFINITION -DHAVE_AVX2_CPU_DEFINITION -O3 -DNDEBUG -DNDEBUG -std=gnu++20 -fPIE -fdiagnostics-color=always -DMKL_HAS_SBGEMM -DMKL_HAS_SHGEMM -DTORCH_USE_LIBUV -DCAFFE2_USE_GLOO -fPIC -D__HIP_PLATFORM_AMD__=1 -DCUDA_HAS_FP16=1 -DUSE_ROCM -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -DTORCH_HIP_VERSION=702 -Wno-shift-count-negative -Wno-shift-count-overflow -DCAFFE2_USE_MIOPEN -DTHRUST_DEVICE_SYSTEM=THRUST_DEVICE_SYSTEM_HIP -DHIPBLAS_V2 -DHIP_ENABLE_WARP_SYNC_BUILTINS -DHIPBLASLT_OUTER_VEC -DUSE_ROCM_CK_GEMM -MD -MT caffe2/CMakeFiles/memory_overlapping_test.dir/__/aten/src/ATen/test/memory_overlapping_test.cpp.o -MF caffe2/CMakeFiles/memory_overlapping_test.dir/__/aten/src/ATen/test/memory_overlapping_test.cpp.o.d -o caffe2/CMakeFiles/memory_overlapping_test.dir/__/aten/src/ATen/test/memory_overlapping_test.cpp.o -c /var/lib/jenkins/pytorch/aten/src/ATen/test/memory_overlapping_test.cpp
thread 'main' panicked at 'failed to shut down worker thread', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/jobserver-0.1.9/src/lib.rs:650:16
note: Run with `RUST_BACKTRACE=1` for a backtrace.
[7714/8176] Building CXX object caffe2/CMakeFiles/native_test.dir/__/aten/src/ATen/test/native_test.cpp.o
[7715/8176] Building CXX object caffe2/CMakeFiles/torch_hip.dir/__/torch/csrc/jit/tensorexpr/cuda_codegen.cpp.o

rocm-repo-management-api · 2026-04-28T16:37:07Z

Jenkins build for 93b2d31c392008b0cfe4788a1a332f67553d4996 commit finished as NOT_BUILT
Links: Pipeline Overview / Build artifacts / Test Results

rocm-repo-management-api · 2026-04-28T19:24:22Z

Jenkins build for 93b2d31c392008b0cfe4788a1a332f67553d4996 commit finished as FAILURE
Links: Pipeline Overview / Build artifacts / Test Results

rocm-repo-management-api · 2026-04-29T08:24:33Z

Jenkins build for 7b8dd188ab294f8c1d36cb5b5b898fdadc06f7f9 commit finished as FAILURE
Links: Pipeline Overview / Build artifacts / Test Results

First draft

507fd86

leo-automation requested a review from jeffdaily as a code owner March 31, 2026 14:21

leo-automation added 2 commits March 31, 2026 17:30

Pass artfiact and test before pushing image

59590af

Matrix

ef2aefd

leo-automation removed the request for review from jeffdaily April 1, 2026 13:31

leo-automation added 15 commits April 1, 2026 15:45

Remove notifications for now

fc2c16d

Temporary trigger

2a6a9e6

Self hosted runners smoke test

a7223b2

Delete smokes

c818f73

Verboose and runner change

454a5fa

Updae dockerfile

03393e9

Remove tty

551548d

Use older buildx with better build logging

e53f83c

Verboose image build troublshooting

2e68950

Debug

1784746

More logging

28cbb19

FIx permissions and have main jib disable sccache

ffdf09a

Debug

865f60d

Fix debug script

8711232

Debug script fix

9b82418

jithunnair-amd reviewed Apr 14, 2026

View reviewed changes

leo-automation added 7 commits April 16, 2026 14:23

Implement Jithun's suggestions

1fbfc09

Updated timeout

d74afa3

Remove debug

717a478

pin sscache version

2cd758b

Debug

c28afaa

buildx fix

66ffb00

Debug buildx

554e8f5

ROCm deleted a comment from rocm-repo-management-api Bot Apr 22, 2026

jithunnair-amd reviewed Apr 23, 2026

View reviewed changes

Comment thread .ci/docker/pytorch-nightly-docker.Dockerfile

jithunnair-amd reviewed Apr 23, 2026

View reviewed changes

leo-automation added 3 commits April 23, 2026 17:31

All in one job

758f32b

try 7.2.0

af18af1

7.2

807c7a1

Bypass sschache on rochsmem torch target

391c1d3

Remove cherry pick

fb1c009

sscacge workaround

20df855

Address comments

077f47c

leo-automation requested a review from jithunnair-amd April 27, 2026 14:45

Trivy increase context size

012f035

Try removing use_preprocessor_cache_mode from sccache

88ec330

Cleanup

7b8dd18

leo-automation force-pushed the rocm-nightly-gha branch from 93b2d31 to 7b8dd18 Compare April 29, 2026 08:20

	./build.sh pytorch-linux-jammy-rocm${{ env.ROCM_VERSION }}-py${{ env.PYTHON_VERSION }} \
	./build.sh pytorch-linux-jammy-rocm-n-py3 \

Conversation

leo-automation commented Mar 31, 2026

Motivation

Technical Details

Test Result

Uh oh!

jithunnair-amd Apr 14, 2026

Choose a reason for hiding this comment

Uh oh!

jithunnair-amd Apr 14, 2026

Choose a reason for hiding this comment

Uh oh!

jithunnair-amd commented Apr 14, 2026

Uh oh!

rocm-repo-management-api Bot commented Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jithunnair-amd Apr 23, 2026

Choose a reason for hiding this comment

Uh oh!

jithunnair-amd Apr 23, 2026

Choose a reason for hiding this comment

Uh oh!

jithunnair-amd Apr 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jithunnair-amd Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rocm-repo-management-api Bot commented Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rocm-repo-management-api Bot commented Apr 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rocm-repo-management-api Bot commented Apr 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rocm-repo-management-api Bot commented Apr 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rocm-repo-management-api Bot commented Apr 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rocm-repo-management-api Bot commented Apr 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rocm-repo-management-api Bot commented Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rocm-repo-management-api Bot commented Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rocm-repo-management-api Bot commented Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rocm-repo-management-api Bot commented Apr 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

rocm-repo-management-api Bot commented Apr 23, 2026 •

edited

Loading

jithunnair-amd Apr 23, 2026 •

edited

Loading

rocm-repo-management-api Bot commented Apr 23, 2026 •

edited

Loading

rocm-repo-management-api Bot commented Apr 24, 2026 •

edited

Loading

rocm-repo-management-api Bot commented Apr 24, 2026 •

edited

Loading

rocm-repo-management-api Bot commented Apr 24, 2026 •

edited

Loading

rocm-repo-management-api Bot commented Apr 27, 2026 •

edited

Loading

rocm-repo-management-api Bot commented Apr 27, 2026 •

edited

Loading

rocm-repo-management-api Bot commented Apr 28, 2026 •

edited

Loading

rocm-repo-management-api Bot commented Apr 28, 2026 •

edited

Loading

rocm-repo-management-api Bot commented Apr 28, 2026 •

edited

Loading

rocm-repo-management-api Bot commented Apr 29, 2026 •

edited

Loading