[CI] Parity: support sha=latest by resolving the latest green run on main#3393
Open
ethanwee1 wants to merge 1 commit into
Open
[CI] Parity: support sha=latest by resolving the latest green run on main#3393ethanwee1 wants to merge 1 commit into
ethanwee1 wants to merge 1 commit into
ROCm Repo Management API / Jenkins
failed
Jul 6, 2026 in 3h 23m 57s
Tests/Test Inductor/Run pytorch_inductor_1: error in 'error' step
Tests / Test PyTorch / Test PyTorch / Run pytorch_test_1 / Shell Script
Error in sh step, with arguments ./test_pytorch_test.sh.
script returned exit code 1
Build log
[2026-07-06T21:00:04.368Z] + ./test_pytorch_test.sh
[2026-07-06T21:00:04.369Z] + [[ -z '' ]]
[2026-07-06T21:00:04.369Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls'
[2026-07-06T21:00:04.369Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=default -e SHARD_NUMBER=1 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-07-06T21:00:04.369Z] ++ set -ex
[2026-07-06T21:00:04.369Z] ++ set -o pipefail
[2026-07-06T21:00:04.369Z] ++ copy_whls=0
[2026-07-06T21:00:04.369Z] ++ envvars=()
[2026-07-06T21:00:04.369Z] ++ [[ 13 -gt 0 ]]
[2026-07-06T21:00:04.369Z] ++ key=-e
[2026-07-06T21:00:04.369Z] ++ case $key in
[2026-07-06T21:00:04.369Z] ++ envvars+=("-e $2")
[2026-07-06T21:00:04.369Z] ++ shift
[2026-07-06T21:00:04.369Z] ++ shift
[2026-07-06T21:00:04.369Z] ++ [[ 11 -gt 0 ]]
[2026-07-06T21:00:04.369Z] ++ key=-e
[2026-07-06T21:00:04.369Z] ++ case $key in
[2026-07-06T21:00:04.369Z] ++ envvars+=("-e $2")
[2026-07-06T21:00:04.369Z] ++ shift
[2026-07-06T21:00:04.369Z] ++ shift
[2026-07-06T21:00:04.369Z] ++ [[ 9 -gt 0 ]]
[2026-07-06T21:00:04.369Z] ++ key=-e
[2026-07-06T21:00:04.369Z] ++ case $key in
[2026-07-06T21:00:04.369Z] ++ envvars+=("-e $2")
[2026-07-06T21:00:04.369Z] ++ shift
[2026-07-06T21:00:04.369Z] ++ shift
[2026-07-06T21:00:04.369Z] ++ [[ 7 -gt 0 ]]
[2026-07-06T21:00:04.369Z] ++ key=-e
[2026-07-06T21:00:04.369Z] ++ case $key in
[2026-07-06T21:00:04.369Z] ++ envvars+=("-e $2")
[2026-07-06T21:00:04.369Z] ++ shift
[2026-07-06T21:00:04.369Z] ++ shift
[2026-07-06T21:00:04.369Z] ++ [[ 5 -gt 0 ]]
[2026-07-06T21:00:04.369Z] ++ key=-e
[2026-07-06T21:00:04.369Z] ++ case $key in
[2026-07-06T21:00:04.369Z] ++ envvars+=("-e $2")
[2026-07-06T21:00:04.369Z] ++ shift
[2026-07-06T21:00:04.369Z] ++ shift
[2026-07-06T21:00:04.369Z] ++ [[ 3 -gt 0 ]]
[2026-07-06T21:00:04.369Z] ++ key=-e
[2026-07-06T21:00:04.369Z] ++ case $key in
[2026-07-06T21:00:04.369Z] ++ envvars+=("-e $2")
[2026-07-06T21:00:04.369Z] ++ shift
[2026-07-06T21:00:04.369Z] ++ shift
[2026-07-06T21:00:04.369Z] ++ [[ 1 -gt 0 ]]
[2026-07-06T21:00:04.369Z] ++ key=--copy-whls
[2026-07-06T21:00:04.369Z] ++ case $key in
[2026-07-06T21:00:04.369Z] ++ copy_whls=1
[2026-07-06T21:00:04.369Z] ++ shift
[2026-07-06T21:00:04.369Z] ++ [[ 0 -gt 0 ]]
[2026-07-06T21:00:04.369Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-07-06T21:00:04.369Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-07-06T21:00:04.369Z] ++ [[ false == \t\r\u\e ]]
[2026-07-06T21:00:04.369Z] ++ [[ true == \t\r\u\e ]]
[2026-07-06T21:00:04.369Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
[2026-07-06T21:00:04.369Z] ++ docker ps -a
[2026-07-06T21:00:04.369Z] CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[2026-07-06T21:00:04.369Z] ++ docker rm -f pytorch-ci-container
[2026-07-06T21:00:04.369Z] ++ docker pull rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-06T21:00:09.688Z] pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942: Pulling from rocm/pytorch-ci-private
[2026-07-06T21:00:10.081Z] 99e6462d03e0: Pulling fs layer
[2026-07-06T21:00:10.081Z] 8d78d529f04d: Pulling fs layer
[2026-07-06T21:00:10.081Z] 906ac4de0a2b: Pulling fs layer
[2026-07-06T21:00:10.081Z] cb259a83ac3d: Pulling fs layer
[2026-07-06T21:00:10.081Z] 5f8dd10c9c78: Pulling fs layer
[2026-07-06T21:00:10.081Z] 78cb638d6686: Pulling fs layer
[2026-07-06T21:00:10.081Z] c608e00b1282: Pulling fs layer
[2026-07-06T21:00:10.081Z] dedc784bccba: Pulling fs layer
[2026-07-06T21:00:10.081Z] 0fcc3b23bd99: Pulling fs layer
[2026-07-06T21:00:10.081Z] bf3e4cf00cd6: Pulling fs layer
[2026-07-06T21:00:10.081Z] 284e8cbb8a05: Pulling fs layer
[2026-07-06T21:00:10.081Z] db0af168a1a4: Pulling fs layer
[2026-07-06T21:00:10.081Z] ed86e5a5d9b1: Pulling fs layer
[2026-07-06T21:00:10.081Z] 42239a12ed73: Pulling fs layer
[2026-07-06T21:00:10.081Z] 65f281ce2e70: Pulling fs layer
[2026-07-06T21:00:10.081Z] 4f4fb700ef54: Pulling fs layer
[2026-07-06T21:00:10.081Z] f790d498bebb: Pulling fs layer
[2026-07-06T21:00:10.081Z] 3d56742d201d: Pulling fs layer
[2026-07-06T21:00:10.081Z] 0e6d9b5f5e8d: Pulling fs layer
[2026-07-06T21:00:10.081Z] 5edf52847c35: Pulling fs layer
[2026-07-06T21:00:10.081Z] d7df6379f9cc: Pulling fs layer
[2026-07-06T21:00:10.081Z] 9548298aa5f4: Pulling fs layer
[2026-07-06T21:00:10.081Z] 34e29023481f: Pulling fs layer
[2026-07-06T21:00:10.081Z] 4c8a34755929: Pulling fs layer
[2026-07-06T21:00:10.081Z] 3d56742d201d: Pulling fs layer
[2026-07-06T21:00:10.081Z] 07b45639fc6f: Pulling fs layer
[2026-07-06T21:00:10.081Z] f3058198fba5: Pulling fs layer
[2026-07-06T21:00:10.081Z] 3d56742d201d: Pulling fs layer
[2026-07-06T21:00:10.081Z] 4106a853273f: Pulling fs layer
[2026-07-06T21:00:10.081Z] 0a0ffadd8b4f: Pulling fs layer
[2026-07-06T21:00:10.081Z] f5afebc61536: Pulling fs layer
[2026-07-06T21:00:10.081Z] 8b3d007242f8: Pulling fs layer
[2026-07-06T21:00:10.081Z] d9a8b776bea7: Pulling fs layer
[2026-07-06T21:00:10.081Z] b96eaed0e751: Pulling fs layer
[2026-07-06T21:00:10.081Z] c0664a7365cb: Pulling fs layer
[2026-07-06T21:00:10.081Z] 747272797eac: Pulling fs layer
[2026-07-06T21:00:10.081Z] 86cc05da26d1: Pulling fs layer
[2026-07-06T21:00:10.081Z] eab367f1ca39: Pulling fs layer
[2026-07-06T21:00:10.081Z] 95d704643610: Pulling fs layer
[2026-07-06T21:00:10.081Z] b09b7d04a8d9: Pulling fs layer
[2026-07-06T21:00:10.081Z] cccff27d82fb: Pulling fs layer
[2026-07-06T21:00:10.081Z] ba1fabf878bd: Pulling fs layer
[2026-07-06T21:00:10.081Z] 423e26ddb517: Pulling fs layer
[2026-07-06T21:00:10.081Z] 1057fba0c7df: Pulling fs layer
[2026-07-06T21:00:10.081Z] 4cfd21f31c8b: Pulling fs layer
[2026-07-06T21:00:10.081Z] 3d56742d201d: Pulling fs layer
[2026-07-06T21:00:10.081Z] 8d998c79d309: Pulling fs layer
[2026-07-06T21:00:10.081Z] 27fcad4d96a6: Pulling fs layer
[2026-07-06T21:00:10.081Z] f0ef294e4f9b: Pulling fs layer
[2026-07-06T21:00:10.081Z] 241cbb1c2f26: Pulling fs layer
[2026-07-06T21:00:10.081Z] 9ae773ccc4b2: Pulling fs layer
[2026-07-06T21:00:10.081Z] 4f4fb700ef54: Pulling fs layer
[2026-07-06T21:00:10.081Z] 3d56742d201d: Pulling fs layer
[2026-07-06T21:00:10.081Z] 78a3c5370db4: Pulling fs layer
[2026-07-06T21:00:10.081Z] ed874654c321: Pulling fs layer
[2026-07-06T21:00:10.081Z] fef65a673dc9: Pulling fs layer
[2026-07-06T21:00:10.081Z] f391428cfe0f: Pulling fs layer
[2026-07-06T21:00:10.081Z] 69145e0460f6: Download complete
[2026-07-06T21:00:10.081Z] c608e00b1282: Download complete
[2026-07-06T21:00:10.466Z] 34e29023481f: Download complete
[2026-07-06T21:00:10.466Z] 8d78d529f04d: Download complete
[2026-07-06T21:00:10.466Z] 99e6462d03e0: Download complete
[2026-07-06T21:00:10.466Z] ed874654c321: Download complete
[2026-07-06T21:00:10.466Z] f3058198fba5: Download complete
[2026-07-06T21:00:10.466Z] f5afebc61536: Download complete
[2026-07-06T21:00:10.466Z] 86cc05da26d1: Download complete
[2026-07-06T21:00:10.466Z] 8b3d007242f8: Download complete
[2026-07-06T21:00:10.466Z] 9548298aa5f4: Download complete
[2026-07-06T21:00:10.466Z] c0664a7365cb: Download complete
[2026-07-06T21:00:10.466Z] 4f4fb700ef54: Download complete
[2026-07-06T21:00:10.466Z] 4cfd21f31c8b: Download complete
[2026-07-06T21:00:10.466Z] 78a3c5370db4: Download complete
[2026-07-06T21:00:10.466Z] f391428cfe0f: Download complete
[2026-07-06T21:00:10.466Z] 3d56742d201d: Download complete
[2026-07-06T21:00:10.466Z] 0fcc3b23bd99: Download complete
[2026-07-06T21:00:10.466Z] 9ae773ccc4b2: Download complete
[2026-07-06T21:00:10.466Z] f0ef294e4f9b: Download complete
[2026-07-06T21:00:10.466Z] ed86e5a5d9b1: Download complete
[2026-07-06T21:00:10.466Z] 5f8dd10c9c78: Download complete
[2026-07-06T21:00:10.466Z] b96eaed0e751: Download complete
[2026-07-06T21:00:10.466Z] f790d498bebb: Download complete
[2026-07-06T21:00:10.466Z] bf3e4cf00cd6: Download complete
[2026-07-06T21:00:10.466Z] 8d998c79d309: Download complete
[2026-07-06T21:00:10.466Z] 27fcad4d96a6: Download complete
[2026-07-06T21:00:10.466Z] 42239a12ed73: Download complete
[2026-07-06T21:00:10.466Z] db0af168a1a4: Download complete
[2026-07-06T21:00:10.466Z] 07b45639fc6f: Download complete
[2026-07-06T21:00:10.466Z] eab367f1ca39: Download complete
[2026-07-06T21:00:10.466Z] b09b7d04a8d9: Download complete
[2026-07-06T21:00:10.466Z] 423e26ddb517: Download complete
[2026-07-06T21:00:10.466Z] 241cbb1c2f26: Download complete
[2026-07-06T21:00:10.466Z] fef65a673dc9: Download complete
[2026-07-06T21:00:10.466Z] 65f281ce2e70: Download complete
[2026-07-06T21:00:10.466Z] 284e8cbb8a05: Download complete
[2026-07-06T21:00:10.466Z] 906ac4de0a2b: Download complete
[2026-07-06T21:00:10.466Z] d7df6379f9cc: Download complete
[2026-07-06T21:00:10.466Z] 0a0ffadd8b4f: Download complete
[2026-07-06T21:00:10.466Z] 747272797eac: Download complete
[2026-07-06T21:00:10.850Z] 0e6d9b5f5e8d: Download complete
[2026-07-06T21:00:10.850Z] 4106a853273f: Download complete
[2026-07-06T21:00:11.237Z] ba1fabf878bd: Download complete
[2026-07-06T21:00:11.623Z] cb259a83ac3d: Download complete
[2026-07-06T21:00:11.623Z] 4c8a34755929: Download complete
[2026-07-06T21:00:12.002Z] 95d704643610: Download complete
[2026-07-06T21:00:12.002Z] 99e6462d03e0: Pull complete
[2026-07-06T21:00:12.002Z] cb259a83ac3d: Pull complete
[2026-07-06T21:00:13.905Z] cccff27d82fb: Download complete
[2026-07-06T21:00:18.106Z] 1057fba0c7df: Download complete
[2026-07-06T21:00:25.933Z] 1057fba0c7df: Pull complete
[2026-07-06T21:00:25.933Z] c608e00b1282: Pull complete
[2026-07-06T21:00:25.933Z] 34e29023481f: Pull complete
[2026-07-06T21:00:25.933Z] 8d78d529f04d: Pull complete
[2026-07-06T21:00:25.933Z] 3d56742d201d: Pull complete
[2026-07-06T21:00:25.933Z] 65f281ce2e70: Pull complete
[2026-07-06T21:00:25.933Z] 906ac4de0a2b: Pull complete
[2026-07-06T21:00:25.933Z] d7df6379f9cc: Pull complete
[2026-07-06T21:00:25.933Z] 0e6d9b5f5e8d: Pull complete
[2026-07-06T21:00:32.398Z] 78cb638d6686: Download complete
[2026-07-06T21:01:01.142Z] 5edf52847c35: Download complete
[2026-07-06T21:01:42.329Z] 5edf52847c35: Pull complete
[2026-07-06T21:01:42.329Z] ed874654c321: Pull complete
[2026-07-06T21:01:42.329Z] 86cc05da26d1: Pull complete
[2026-07-06T21:01:42.329Z] 8b3d007242f8: Pull complete
[2026-07-06T21:01:42.329Z] 9548298aa5f4: Pull complete
[2026-07-06T21:01:42.329Z] db0af168a1a4: Pull complete
[2026-07-06T21:01:42.329Z] 0a0ffadd8b4f: Pull complete
[2026-07-06T21:01:42.329Z] 747272797eac: Pull complete
[2026-07-06T21:02:23.943Z] d9a8b776bea7: Download complete
[2026-07-06T21:03:34.528Z] dedc784bccba: Download complete
[2026-07-06T21:04:58.948Z] 9ae773ccc4b2: Pull complete
[2026-07-06T21:04:58.948Z] bf3e4cf00cd6: Pull complete
[2026-07-06T21:04:58.948Z] 27fcad4d96a6: Pull complete
[2026-07-06T21:04:58.948Z] dedc784bccba: Pull complete
[2026-07-06T21:05:01.553Z] f3058198fba5: Pull complete
[2026-07-06T21:05:01.553Z] f5afebc61536: Pull complete
[2026-07-06T21:05:01.553Z] 78cb638d6686: Pull complete
[2026-07-06T21:05:02.946Z] c0664a7365cb: Pull complete
[2026-07-06T21:05:02.946Z] 07b45639fc6f: Pull complete
[2026-07-06T21:05:02.946Z] cccff27d82fb: Pull complete
[2026-07-06T21:05:02.946Z] b96eaed0e751: Pull complete
[2026-07-06T21:05:02.946Z] f790d498bebb: Pull complete
[2026-07-06T21:05:03.380Z] 95d704643610: Pull complete
[2026-07-06T21:05:03.380Z] 4f4fb700ef54: Pull complete
[2026-07-06T21:05:03.380Z] 4cfd21f31c8b: Pull complete
[2026-07-06T21:05:03.380Z] 78a3c5370db4: Pull complete
[2026-07-06T21:05:03.380Z] 0fcc3b23bd99: Pull complete
[2026-07-06T21:05:03.380Z] f0ef294e4f9b: Pull complete
[2026-07-06T21:05:03.380Z] 5f8dd10c9c78: Pull complete
[2026-07-06T21:05:03.380Z] 42239a12ed73: Pull complete
[2026-07-06T21:05:03.380Z] b09b7d04a8d9: Pull complete
[2026-07-06T21:05:03.380Z] 423e26ddb517: Pull complete
[2026-07-06T21:05:03.380Z] 241cbb1c2f26: Pull complete
[2026-07-06T21:05:03.380Z] 284e8cbb8a05: Pull complete
[2026-07-06T21:05:03.380Z] 4106a853273f: Pull complete
[2026-07-06T21:06:02.700Z] f391428cfe0f: Pull complete
[2026-07-06T21:06:02.700Z] eab367f1ca39: Pull complete
[2026-07-06T21:06:02.700Z] d9a8b776bea7: Pull complete
[2026-07-06T21:06:02.700Z] ed86e5a5d9b1: Pull complete
[2026-07-06T21:06:02.700Z] 8d998c79d309: Pull complete
[2026-07-06T21:06:02.700Z] fef65a673dc9: Pull complete
[2026-07-06T21:06:02.700Z] ba1fabf878bd: Pull complete
[2026-07-06T21:06:02.700Z] 4c8a34755929: Pull complete
[2026-07-06T21:06:02.700Z] Digest: sha256:b836aa0e221ac85a2184ec24a67b9d5089f3caa45d182cf521dee7d38112aa81
[2026-07-06T21:06:02.700Z] Status: Downloaded newer image for rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-06T21:06:02.700Z] docker.io/rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-06T21:06:02.700Z] ++ GPU_FLAGS=
[2026-07-06T21:06:02.700Z] ++ echo 'NODE_LABELS: k8s-mi250-1-fb2qx gfx90a mlir k8s-mi250-1-fb2qx gfx90a mlir'
[2026-07-06T21:06:02.700Z] NODE_LABELS: k8s-mi250-1-fb2qx gfx90a mlir k8s-mi250-1-fb2qx gfx90a mlir
[2026-07-06T21:06:02.700Z] ++ [[ ! k8s-mi250-1-fb2qx gfx90a mlir =~ build-only ]]
[2026-07-06T21:06:02.700Z] ++ GPU_FLAGS='--group-add video --device=/dev/kfd --device=/dev/dri'
[2026-07-06T21:06:02.700Z] ++++ dirname prepare_docker_env.sh
[2026-07-06T21:06:02.700Z] +++ cd .
[2026-07-06T21:06:02.700Z] +++ pwd
[2026-07-06T21:06:02.700Z] ++ script_dir=/home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3393
[2026-07-06T21:06:02.700Z] ++ docker run --name pytorch-ci-container -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=default -e SHARD_NUMBER=1 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip -e PREBUILT=false -e TEST_CORE=true -e TEST_CORE=true -v /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3393:/host_workspace -v /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3393:/scripts --entrypoint bash -d --network=host --group-add video --device=/dev/kfd --device=/dev/dri --ipc=host --pid=host --shm-size 8G --cap-add=SYS_PTRACE --security-opt seccomp=unconfined rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942 -lc 'sleep infinity'
[2026-07-06T21:06:02.700Z] 9b7ec9351d0c777f5fa379f62e464b039ce2b28896b76e332253f9249823d642
[2026-07-06T21:06:03.080Z] ++ env -i python3 /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch/tools/stats/export_test_times.py
[2026-07-06T21:06:04.951Z] Exporting test times from test-infra
[2026-07-06T21:06:04.951Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch/.additional_ci_files/test-times.json
[2026-07-06T21:06:04.951Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch/.additional_ci_files/test-class-times.json
[2026-07-06T21:06:04.951Z] ++ python3 /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch/tools/stats/export_test_times.py
[2026-07-06T21:06:04.951Z] Exporting test times from test-infra
[2026-07-06T21:06:04.951Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch/.additional_ci_files/test-times.json
[2026-07-06T21:06:04.951Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch/.additional_ci_files/test-class-times.json
[2026-07-06T21:06:04.951Z] ++ [[ -d /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch/.additional_ci_files ]]
[2026-07-06T21:06:04.951Z] ++ sed -i -E 's/"\S+rocm\S+"/""/g' /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch/.additional_ci_files/test-times.json
[2026-07-06T21:06:04.951Z] ++ [[ false != \t\r\u\e ]]
[2026-07-06T21:06:04.951Z] ++ docker exec -u root pytorch-ci-container bash -c 'rm -rf /var/lib/jenkins/pytorch'
[2026-07-06T21:06:04.951Z] ++ docker cp /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-06T21:06:10.115Z] ++ docker exec -u root pytorch-ci-container bash -c 'chown root:root -R /var/lib/jenkins/pytorch'
[2026-07-06T21:06:10.115Z] ++ [[ 1 -eq 1 ]]
[2026-07-06T21:06:10.115Z] ++ docker cp /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3393/rocm_composable_kernel-7.1.1.dev0+g4266f8-py3-none-any.whl /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3393/torch-2.14.0a0+git3ec517c-cp312-cp312-linux_x86_64.whl pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-06T21:06:10.115Z] docker: 'docker cp' requires 2 arguments
[2026-07-06T21:06:10.115Z]
[2026-07-06T21:06:10.115Z] Usage: docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
[2026-07-06T21:06:10.115Z] docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH
[2026-07-06T21:06:10.115Z]
[2026-07-06T21:06:10.115Z] Run 'docker cp --help' for more information
Tests / Test PyTorch / Test PyTorch / Run pytorch_test_1 / Error signal
Error in error step, with arguments pytorch_test_1 failed.
pytorch_test_1 failed
Tests / Test PyTorch / Test PyTorch / Run pytorch_test_2 / Shell Script
Error in sh step, with arguments rocminfo.
script returned exit code 1
Build log
[2026-07-06T21:06:19.213Z] + rocminfo
[2026-07-06T21:06:19.213Z] ROCk module version 6.10.5 is loaded
[2026-07-06T21:06:19.213Z] Unable to open /dev/kfd read-write: Invalid argument
[2026-07-06T21:06:19.213Z] jenkins is member of video group
Tests / Test PyTorch / Test PyTorch / Run pytorch_test_2 / Shell Script
Error in sh step, with arguments ./test_pytorch_test.sh.
script returned exit code 1
Build log
[2026-07-06T21:18:49.525Z] + ./test_pytorch_test.sh
[2026-07-06T21:18:49.525Z] + [[ -z '' ]]
[2026-07-06T21:18:49.525Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls'
[2026-07-06T21:18:49.525Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=default -e SHARD_NUMBER=2 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-07-06T21:18:49.525Z] ++ set -ex
[2026-07-06T21:18:49.525Z] ++ set -o pipefail
[2026-07-06T21:18:49.525Z] ++ copy_whls=0
[2026-07-06T21:18:49.525Z] ++ envvars=()
[2026-07-06T21:18:49.525Z] ++ [[ 13 -gt 0 ]]
[2026-07-06T21:18:49.525Z] ++ key=-e
[2026-07-06T21:18:49.525Z] ++ case $key in
[2026-07-06T21:18:49.525Z] ++ envvars+=("-e $2")
[2026-07-06T21:18:49.525Z] ++ shift
[2026-07-06T21:18:49.525Z] ++ shift
[2026-07-06T21:18:49.525Z] ++ [[ 11 -gt 0 ]]
[2026-07-06T21:18:49.525Z] ++ key=-e
[2026-07-06T21:18:49.525Z] ++ case $key in
[2026-07-06T21:18:49.525Z] ++ envvars+=("-e $2")
[2026-07-06T21:18:49.525Z] ++ shift
[2026-07-06T21:18:49.525Z] ++ shift
[2026-07-06T21:18:49.525Z] ++ [[ 9 -gt 0 ]]
[2026-07-06T21:18:49.525Z] ++ key=-e
[2026-07-06T21:18:49.525Z] ++ case $key in
[2026-07-06T21:18:49.525Z] ++ envvars+=("-e $2")
[2026-07-06T21:18:49.525Z] ++ shift
[2026-07-06T21:18:49.525Z] ++ shift
[2026-07-06T21:18:49.525Z] ++ [[ 7 -gt 0 ]]
[2026-07-06T21:18:49.525Z] ++ key=-e
[2026-07-06T21:18:49.525Z] ++ case $key in
[2026-07-06T21:18:49.525Z] ++ envvars+=("-e $2")
[2026-07-06T21:18:49.525Z] ++ shift
[2026-07-06T21:18:49.525Z] ++ shift
[2026-07-06T21:18:49.525Z] ++ [[ 5 -gt 0 ]]
[2026-07-06T21:18:49.525Z] ++ key=-e
[2026-07-06T21:18:49.525Z] ++ case $key in
[2026-07-06T21:18:49.525Z] ++ envvars+=("-e $2")
[2026-07-06T21:18:49.525Z] ++ shift
[2026-07-06T21:18:49.525Z] ++ shift
[2026-07-06T21:18:49.525Z] ++ [[ 3 -gt 0 ]]
[2026-07-06T21:18:49.525Z] ++ key=-e
[2026-07-06T21:18:49.525Z] ++ case $key in
[2026-07-06T21:18:49.525Z] ++ envvars+=("-e $2")
[2026-07-06T21:18:49.525Z] ++ shift
[2026-07-06T21:18:49.525Z] ++ shift
[2026-07-06T21:18:49.525Z] ++ [[ 1 -gt 0 ]]
[2026-07-06T21:18:49.525Z] ++ key=--copy-whls
[2026-07-06T21:18:49.525Z] ++ case $key in
[2026-07-06T21:18:49.525Z] ++ copy_whls=1
[2026-07-06T21:18:49.525Z] ++ shift
[2026-07-06T21:18:49.525Z] ++ [[ 0 -gt 0 ]]
[2026-07-06T21:18:49.525Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-07-06T21:18:49.525Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-07-06T21:18:49.525Z] ++ [[ false == \t\r\u\e ]]
[2026-07-06T21:18:49.525Z] ++ [[ true == \t\r\u\e ]]
[2026-07-06T21:18:49.525Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
[2026-07-06T21:18:49.525Z] ++ docker ps -a
[2026-07-06T21:18:49.525Z] CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[2026-07-06T21:18:49.525Z] 29f0c6fb79dd rocm/pytorch-private:ubuntu20.04_py3.9_pip3.9 "bash -lc 'sleep inf…" 21 hours ago Exited (143) 15 hours ago pytorch-ci-container
[2026-07-06T21:18:49.525Z] 33af8c9e058f dashboard-dashboard-ui "/docker-entrypoint.…" 2 weeks ago Up 5 hours 80/tcp, 0.0.0.0:3000->3000/tcp, [::]:3000->3000/tcp rocagents-dashboard-ui
[2026-07-06T21:18:49.525Z] 7c6be83a2db1 dashboard-dashboard-api "uvicorn main:app --…" 2 weeks ago Up 5 hours (healthy) 0.0.0.0:8080->8080/tcp, [::]:8080->8080/tcp rocagents-dashboard-api
[2026-07-06T21:18:49.525Z] 780dc8fe5a7b rocm/eigen-test:ubuntu20.04-rocm5.3.0-eigen "/scripts/run_tests_…" 4 weeks ago Exited (1) 4 weeks ago infallible_chatelet
[2026-07-06T21:18:49.525Z] 2d7786cfd066 deployment-rocagents "python src/multi_re…" 7 weeks ago Up 5 hours (healthy) rocagents-orchestrator
[2026-07-06T21:18:49.525Z] ++ docker rm -f pytorch-ci-container
[2026-07-06T21:18:50.082Z] ++ docker pull rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-06T21:18:50.641Z] pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942: Pulling from rocm/pytorch-ci-private
[2026-07-06T21:18:50.641Z] Digest: sha256:b836aa0e221ac85a2184ec24a67b9d5089f3caa45d182cf521dee7d38112aa81
[2026-07-06T21:18:50.641Z] Status: Image is up to date for rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-06T21:18:50.641Z] docker.io/rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-06T21:18:50.641Z] ++ GPU_FLAGS=
[2026-07-06T21:18:50.641Z] ++ echo 'NODE_LABELS: multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250 multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250'
[2026-07-06T21:18:50.641Z] NODE_LABELS: multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250 multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250
[2026-07-06T21:18:50.641Z] ++ [[ ! multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250 =~ build-only ]]
[2026-07-06T21:18:50.641Z] ++ GPU_FLAGS='--group-add video --device=/dev/kfd --device=/dev/dri'
[2026-07-06T21:18:50.641Z] ++++ dirname prepare_docker_env.sh
[2026-07-06T21:18:50.641Z] +++ cd .
[2026-07-06T21:18:50.641Z] +++ pwd
[2026-07-06T21:18:50.641Z] ++ script_dir=/home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393
[2026-07-06T21:18:50.641Z] ++ docker run --name pytorch-ci-container -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=default -e SHARD_NUMBER=2 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip -e PREBUILT=false -e TEST_CORE=true -e TEST_CORE=true -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393:/host_workspace -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393:/scripts --entrypoint bash -d --network=host --group-add video --device=/dev/kfd --device=/dev/dri --ipc=host --pid=host --shm-size 8G --cap-add=SYS_PTRACE --security-opt seccomp=unconfined rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942 -lc 'sleep infinity'
[2026-07-06T21:18:51.165Z] c3c943b50fb095868e2367c090fd8ae150c2592f258696124f6e194cd7e4e4eb
[2026-07-06T21:18:51.165Z] ++ env -i python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch/tools/stats/export_test_times.py
[2026-07-06T21:18:52.262Z] Exporting test times from test-infra
[2026-07-06T21:18:52.262Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch/.additional_ci_files/test-times.json
[2026-07-06T21:18:52.262Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch/.additional_ci_files/test-class-times.json
[2026-07-06T21:18:52.262Z] ++ python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch/tools/stats/export_test_times.py
[2026-07-06T21:18:52.262Z] Exporting test times from test-infra
[2026-07-06T21:18:52.262Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch/.additional_ci_files/test-times.json
[2026-07-06T21:18:52.262Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch/.additional_ci_files/test-class-times.json
[2026-07-06T21:18:52.262Z] ++ [[ -d /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch/.additional_ci_files ]]
[2026-07-06T21:18:52.262Z] ++ sed -i -E 's/"\S+rocm\S+"/""/g' /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch/.additional_ci_files/test-times.json
[2026-07-06T21:18:52.262Z] ++ [[ false != \t\r\u\e ]]
[2026-07-06T21:18:52.262Z] ++ docker exec -u root pytorch-ci-container bash -c 'rm -rf /var/lib/jenkins/pytorch'
[2026-07-06T21:18:52.262Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-06T21:18:57.452Z] ++ docker exec -u root pytorch-ci-container bash -c 'chown root:root -R /var/lib/jenkins/pytorch'
[2026-07-06T21:18:57.452Z] ++ [[ 1 -eq 1 ]]
[2026-07-06T21:18:57.452Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393/rocm_composable_kernel-7.1.1.dev0+g4266f8-py3-none-any.whl /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393/torch-2.14.0a0+git3ec517c-cp312-cp312-linux_x86_64.whl pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-06T21:18:57.452Z] docker: 'docker cp' requires 2 arguments
[2026-07-06T21:18:57.452Z]
[2026-07-06T21:18:57.452Z] Usage: docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
[2026-07-06T21:18:57.452Z] docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH
[2026-07-06T21:18:57.452Z]
[2026-07-06T21:18:57.452Z] Run 'docker cp --help' for more information
Tests / Test PyTorch / Test PyTorch / Run pytorch_test_2 / Error signal
Error in error step, with arguments pytorch_test_2 failed.
pytorch_test_2 failed
Tests / Test Distributed / Test Distributed / Run pytorch_distributed_1 / Shell Script
Error in sh step, with arguments ./test_pytorch_test_distributed.sh.
script returned exit code 1
Build log
[2026-07-06T20:56:50.935Z] + ./test_pytorch_test_distributed.sh
[2026-07-06T20:56:50.935Z] + [[ -z '' ]]
[2026-07-06T20:56:50.935Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls'
[2026-07-06T20:56:50.935Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=distributed -e SHARD_NUMBER=1 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-07-06T20:56:50.935Z] ++ set -ex
[2026-07-06T20:56:50.935Z] ++ set -o pipefail
[2026-07-06T20:56:50.935Z] ++ copy_whls=0
[2026-07-06T20:56:50.935Z] ++ envvars=()
[2026-07-06T20:56:50.935Z] ++ [[ 13 -gt 0 ]]
[2026-07-06T20:56:50.935Z] ++ key=-e
[2026-07-06T20:56:50.935Z] ++ case $key in
[2026-07-06T20:56:50.935Z] ++ envvars+=("-e $2")
[2026-07-06T20:56:50.935Z] ++ shift
[2026-07-06T20:56:50.935Z] ++ shift
[2026-07-06T20:56:50.935Z] ++ [[ 11 -gt 0 ]]
[2026-07-06T20:56:50.935Z] ++ key=-e
[2026-07-06T20:56:50.935Z] ++ case $key in
[2026-07-06T20:56:50.935Z] ++ envvars+=("-e $2")
[2026-07-06T20:56:50.935Z] ++ shift
[2026-07-06T20:56:50.935Z] ++ shift
[2026-07-06T20:56:50.935Z] ++ [[ 9 -gt 0 ]]
[2026-07-06T20:56:50.935Z] ++ key=-e
[2026-07-06T20:56:50.935Z] ++ case $key in
[2026-07-06T20:56:50.935Z] ++ envvars+=("-e $2")
[2026-07-06T20:56:50.935Z] ++ shift
[2026-07-06T20:56:50.935Z] ++ shift
[2026-07-06T20:56:50.935Z] ++ [[ 7 -gt 0 ]]
[2026-07-06T20:56:50.935Z] ++ key=-e
[2026-07-06T20:56:50.935Z] ++ case $key in
[2026-07-06T20:56:50.935Z] ++ envvars+=("-e $2")
[2026-07-06T20:56:50.935Z] ++ shift
[2026-07-06T20:56:50.935Z] ++ shift
[2026-07-06T20:56:50.935Z] ++ [[ 5 -gt 0 ]]
[2026-07-06T20:56:50.935Z] ++ key=-e
[2026-07-06T20:56:50.935Z] ++ case $key in
[2026-07-06T20:56:50.935Z] ++ envvars+=("-e $2")
[2026-07-06T20:56:50.935Z] ++ shift
[2026-07-06T20:56:50.935Z] ++ shift
[2026-07-06T20:56:50.935Z] ++ [[ 3 -gt 0 ]]
[2026-07-06T20:56:50.935Z] ++ key=-e
[2026-07-06T20:56:50.935Z] ++ case $key in
[2026-07-06T20:56:50.935Z] ++ envvars+=("-e $2")
[2026-07-06T20:56:50.935Z] ++ shift
[2026-07-06T20:56:50.935Z] ++ shift
[2026-07-06T20:56:50.935Z] ++ [[ 1 -gt 0 ]]
[2026-07-06T20:56:50.935Z] ++ key=--copy-whls
[2026-07-06T20:56:50.935Z] ++ case $key in
[2026-07-06T20:56:50.935Z] ++ copy_whls=1
[2026-07-06T20:56:50.935Z] ++ shift
[2026-07-06T20:56:50.935Z] ++ [[ 0 -gt 0 ]]
[2026-07-06T20:56:50.935Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-07-06T20:56:50.935Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-07-06T20:56:50.935Z] ++ [[ false == \t\r\u\e ]]
[2026-07-06T20:56:50.935Z] ++ [[ true == \t\r\u\e ]]
[2026-07-06T20:56:50.935Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
[2026-07-06T20:56:50.935Z] ++ docker ps -a
[2026-07-06T20:56:50.935Z] CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[2026-07-06T20:56:50.935Z] c412cbfc81a5 rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-1b618e3c916dd1605bea8b2bec1c54e0e08d0eb3-gfx908_gfx90a_gfx942 "bash -lc 'sleep inf…" 3 days ago Up 3 days pytorch-ci-container
[2026-07-06T20:56:50.935Z] ++ docker rm -f pytorch-ci-container
[2026-07-06T20:56:56.234Z] ++ docker pull rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-06T20:56:57.379Z] pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942: Pulling from rocm/pytorch-ci-private
[2026-07-06T20:56:57.379Z] Digest: sha256:b836aa0e221ac85a2184ec24a67b9d5089f3caa45d182cf521dee7d38112aa81
[2026-07-06T20:56:57.379Z] Status: Image is up to date for rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-06T20:56:57.379Z] docker.io/rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-06T20:56:57.379Z] ++ GPU_FLAGS=
[2026-07-06T20:56:57.379Z] ++ echo 'NODE_LABELS: multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908'
[2026-07-06T20:56:57.379Z] NODE_LABELS: multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908
[2026-07-06T20:56:57.379Z] ++ [[ ! multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 =~ build-only ]]
[2026-07-06T20:56:57.379Z] ++ GPU_FLAGS='--group-add video --device=/dev/kfd --device=/dev/dri'
[2026-07-06T20:56:57.379Z] ++++ dirname prepare_docker_env.sh
[2026-07-06T20:56:57.379Z] +++ cd .
[2026-07-06T20:56:57.379Z] +++ pwd
[2026-07-06T20:56:57.379Z] ++ script_dir=/home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393
[2026-07-06T20:56:57.379Z] ++ docker run --name pytorch-ci-container -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=distributed -e SHARD_NUMBER=1 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip -e PREBUILT=false -e TEST_CORE=true -e TEST_CORE=true -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393:/host_workspace -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393:/scripts --entrypoint bash -d --network=host --group-add video --device=/dev/kfd --device=/dev/dri --ipc=host --pid=host --shm-size 8G --cap-add=SYS_PTRACE --security-opt seccomp=unconfined rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942 -lc 'sleep infinity'
[2026-07-06T20:56:57.379Z] 2a64dc7fa3cef1831e286c5ba05b416abedd7e0ba192a70395c85243d5bd079c
[2026-07-06T20:56:57.911Z] ++ env -i python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch/tools/stats/export_test_times.py
[2026-07-06T20:56:59.102Z] Exporting test times from test-infra
[2026-07-06T20:56:59.102Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch/.additional_ci_files/test-times.json
[2026-07-06T20:56:59.102Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch/.additional_ci_files/test-class-times.json
[2026-07-06T20:56:59.638Z] ++ python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch/tools/stats/export_test_times.py
[2026-07-06T20:56:59.638Z] Exporting test times from test-infra
[2026-07-06T20:56:59.638Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch/.additional_ci_files/test-times.json
[2026-07-06T20:56:59.638Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch/.additional_ci_files/test-class-times.json
[2026-07-06T20:56:59.638Z] ++ [[ -d /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch/.additional_ci_files ]]
[2026-07-06T20:56:59.638Z] ++ sed -i -E 's/"\S+rocm\S+"/""/g' /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch/.additional_ci_files/test-times.json
[2026-07-06T20:56:59.638Z] ++ [[ false != \t\r\u\e ]]
[2026-07-06T20:56:59.638Z] ++ docker exec -u root pytorch-ci-container bash -c 'rm -rf /var/lib/jenkins/pytorch'
[2026-07-06T20:56:59.638Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-06T20:57:06.107Z] ++ docker exec -u root pytorch-ci-container bash -c 'chown root:root -R /var/lib/jenkins/pytorch'
[2026-07-06T20:57:06.663Z] ++ [[ 1 -eq 1 ]]
[2026-07-06T20:57:06.663Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393/rocm_composable_kernel-7.1.1.dev0+g4266f8-py3-none-any.whl /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393/torch-2.14.0a0+git3ec517c-cp312-cp312-linux_x86_64.whl pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-06T20:57:06.663Z] "docker cp" requires exactly 2 arguments.
[2026-07-06T20:57:06.663Z] See 'docker cp --help'.
[2026-07-06T20:57:06.663Z]
[2026-07-06T20:57:06.663Z] Usage: docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
[2026-07-06T20:57:06.663Z] docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH
[2026-07-06T20:57:06.663Z]
[2026-07-06T20:57:06.663Z] Copy files/folders between a container and the local filesystem
Tests / Test Distributed / Test Distributed / Run pytorch_distributed_1 / Error signal
Error in error step, with arguments pytorch_distributed_1 failed.
pytorch_distributed_1 failed
Tests / Test Distributed / Test Distributed / Run pytorch_distributed_2 / Shell Script
Error in sh step, with arguments ./test_pytorch_test_distributed.sh.
script returned exit code 1
Build log
[2026-07-06T21:01:29.475Z] + ./test_pytorch_test_distributed.sh
[2026-07-06T21:01:29.475Z] + [[ -z '' ]]
[2026-07-06T21:01:29.475Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls'
[2026-07-06T21:01:29.475Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=distributed -e SHARD_NUMBER=2 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-07-06T21:01:29.475Z] ++ set -ex
[2026-07-06T21:01:29.475Z] ++ set -o pipefail
[2026-07-06T21:01:29.475Z] ++ copy_whls=0
[2026-07-06T21:01:29.475Z] ++ envvars=()
[2026-07-06T21:01:29.475Z] ++ [[ 13 -gt 0 ]]
[2026-07-06T21:01:29.475Z] ++ key=-e
[2026-07-06T21:01:29.475Z] ++ case $key in
[2026-07-06T21:01:29.475Z] ++ envvars+=("-e $2")
[2026-07-06T21:01:29.475Z] ++ shift
[2026-07-06T21:01:29.475Z] ++ shift
[2026-07-06T21:01:29.475Z] ++ [[ 11 -gt 0 ]]
[2026-07-06T21:01:29.475Z] ++ key=-e
[2026-07-06T21:01:29.475Z] ++ case $key in
[2026-07-06T21:01:29.475Z] ++ envvars+=("-e $2")
[2026-07-06T21:01:29.475Z] ++ shift
[2026-07-06T21:01:29.475Z] ++ shift
[2026-07-06T21:01:29.475Z] ++ [[ 9 -gt 0 ]]
[2026-07-06T21:01:29.475Z] ++ key=-e
[2026-07-06T21:01:29.475Z] ++ case $key in
[2026-07-06T21:01:29.475Z] ++ envvars+=("-e $2")
[2026-07-06T21:01:29.475Z] ++ shift
[2026-07-06T21:01:29.475Z] ++ shift
[2026-07-06T21:01:29.475Z] ++ [[ 7 -gt 0 ]]
[2026-07-06T21:01:29.475Z] ++ key=-e
[2026-07-06T21:01:29.475Z] ++ case $key in
[2026-07-06T21:01:29.475Z] ++ envvars+=("-e $2")
[2026-07-06T21:01:29.475Z] ++ shift
[2026-07-06T21:01:29.475Z] ++ shift
[2026-07-06T21:01:29.475Z] ++ [[ 5 -gt 0 ]]
[2026-07-06T21:01:29.475Z] ++ key=-e
[2026-07-06T21:01:29.475Z] ++ case $key in
[2026-07-06T21:01:29.475Z] ++ envvars+=("-e $2")
[2026-07-06T21:01:29.475Z] ++ shift
[2026-07-06T21:01:29.475Z] ++ shift
[2026-07-06T21:01:29.475Z] ++ [[ 3 -gt 0 ]]
[2026-07-06T21:01:29.475Z] ++ key=-e
[2026-07-06T21:01:29.475Z] ++ case $key in
[2026-07-06T21:01:29.476Z] ++ envvars+=("-e $2")
[2026-07-06T21:01:29.476Z] ++ shift
[2026-07-06T21:01:29.476Z] ++ shift
[2026-07-06T21:01:29.476Z] ++ [[ 1 -gt 0 ]]
[2026-07-06T21:01:29.476Z] ++ key=--copy-whls
[2026-07-06T21:01:29.476Z] ++ case $key in
[2026-07-06T21:01:29.476Z] ++ copy_whls=1
[2026-07-06T21:01:29.476Z] ++ shift
[2026-07-06T21:01:29.476Z] ++ [[ 0 -gt 0 ]]
[2026-07-06T21:01:29.476Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-07-06T21:01:29.476Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-07-06T21:01:29.476Z] ++ [[ false == \t\r\u\e ]]
[2026-07-06T21:01:29.476Z] ++ [[ true == \t\r\u\e ]]
[2026-07-06T21:01:29.476Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
[2026-07-06T21:01:29.476Z] ++ docker ps -a
[2026-07-06T21:01:29.476Z] CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[2026-07-06T21:01:29.476Z] 2a64dc7fa3ce rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942 "bash -lc 'sleep inf…" 4 minutes ago Up 4 minutes pytorch-ci-container
[2026-07-06T21:01:29.476Z] ++ docker rm -f pytorch-ci-container
[2026-07-06T21:01:30.631Z] ++ docker pull rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-06T21:01:31.167Z] pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942: Pulling from rocm/pytorch-ci-private
[2026-07-06T21:01:31.167Z] Digest: sha256:b836aa0e221ac85a2184ec24a67b9d5089f3caa45d182cf521dee7d38112aa81
[2026-07-06T21:01:31.167Z] Status: Image is up to date for rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-06T21:01:31.167Z] docker.io/rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-06T21:01:31.167Z] ++ GPU_FLAGS=
[2026-07-06T21:01:31.167Z] ++ echo 'NODE_LABELS: multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908'
[2026-07-06T21:01:31.167Z] NODE_LABELS: multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908
[2026-07-06T21:01:31.167Z] ++ [[ ! multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 =~ build-only ]]
[2026-07-06T21:01:31.167Z] ++ GPU_FLAGS='--group-add video --device=/dev/kfd --device=/dev/dri'
[2026-07-06T21:01:31.167Z] ++++ dirname prepare_docker_env.sh
[2026-07-06T21:01:31.167Z] +++ cd .
[2026-07-06T21:01:31.167Z] +++ pwd
[2026-07-06T21:01:31.167Z] ++ script_dir=/home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393
[2026-07-06T21:01:31.167Z] ++ docker run --name pytorch-ci-container -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=distributed -e SHARD_NUMBER=2 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip -e PREBUILT=false -e TEST_CORE=true -e TEST_CORE=true -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393:/host_workspace -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393:/scripts --entrypoint bash -d --network=host --group-add video --device=/dev/kfd --device=/dev/dri --ipc=host --pid=host --shm-size 8G --cap-add=SYS_PTRACE --security-opt seccomp=unconfined rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942 -lc 'sleep infinity'
[2026-07-06T21:01:31.704Z] cc533b2d7cebe80cf9147b4721c90ffc0ea56ba82acf71e4e9a6e2db5fd4b80c
[2026-07-06T21:01:31.704Z] ++ env -i python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch/tools/stats/export_test_times.py
[2026-07-06T21:01:32.259Z] Exporting test times from test-infra
[2026-07-06T21:01:32.259Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch/.additional_ci_files/test-times.json
[2026-07-06T21:01:32.259Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch/.additional_ci_files/test-class-times.json
[2026-07-06T21:01:32.259Z] ++ python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch/tools/stats/export_test_times.py
[2026-07-06T21:01:32.795Z] Exporting test times from test-infra
[2026-07-06T21:01:32.795Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch/.additional_ci_files/test-times.json
[2026-07-06T21:01:32.795Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch/.additional_ci_files/test-class-times.json
[2026-07-06T21:01:32.795Z] ++ [[ -d /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch/.additional_ci_files ]]
[2026-07-06T21:01:32.795Z] ++ sed -i -E 's/"\S+rocm\S+"/""/g' /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch/.additional_ci_files/test-times.json
[2026-07-06T21:01:32.795Z] ++ [[ false != \t\r\u\e ]]
[2026-07-06T21:01:32.795Z] ++ docker exec -u root pytorch-ci-container bash -c 'rm -rf /var/lib/jenkins/pytorch'
[2026-07-06T21:01:32.795Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-06T21:01:39.336Z] ++ docker exec -u root pytorch-ci-container bash -c 'chown root:root -R /var/lib/jenkins/pytorch'
[2026-07-06T21:01:39.887Z] ++ [[ 1 -eq 1 ]]
[2026-07-06T21:01:39.887Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393/rocm_composable_kernel-7.1.1.dev0+g4266f8-py3-none-any.whl /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3393/torch-2.14.0a0+git3ec517c-cp312-cp312-linux_x86_64.whl pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-06T21:01:39.887Z] "docker cp" requires exactly 2 arguments.
[2026-07-06T21:01:39.887Z] See 'docker cp --help'.
[2026-07-06T21:01:39.887Z]
[2026-07-06T21:01:39.887Z] Usage: docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
[2026-07-06T21:01:39.887Z] docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH
[2026-07-06T21:01:39.887Z]
[2026-07-06T21:01:39.887Z] Copy files/folders between a container and the local filesystem
Tests / Test Distributed / Test Distributed / Run pytorch_distributed_2 / Error signal
Error in error step, with arguments pytorch_distributed_2 failed.
pytorch_distributed_2 failed
Tests / Test Inductor / Test Inductor / Run pytorch_inductor_1 / Shell Script
Error in sh step, with arguments rocminfo.
script returned exit code 1
Build log
[2026-07-06T21:02:58.174Z] + rocminfo
[2026-07-06T21:02:58.174Z] ROCk module version 6.10.5 is loaded
[2026-07-06T21:02:58.174Z] Unable to open /dev/kfd read-write: Invalid argument
[2026-07-06T21:02:58.174Z] jenkins is member of video group
Tests / Test Inductor / Test Inductor / Run pytorch_inductor_1 / Shell Script
Error in sh step, with arguments ./test_pytorch_inductor.sh.
script returned exit code 1
Build log
[2026-07-06T21:11:35.619Z] + ./test_pytorch_inductor.sh
[2026-07-06T21:11:35.619Z] + [[ -z '' ]]
[2026-07-06T21:11:35.619Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls'
[2026-07-06T21:11:35.619Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=inductor -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-07-06T21:11:35.619Z] ++ set -ex
[2026-07-06T21:11:35.619Z] ++ set -o pipefail
[2026-07-06T21:11:35.619Z] ++ copy_whls=0
[2026-07-06T21:11:35.619Z] ++ envvars=()
[2026-07-06T21:11:35.619Z] ++ [[ 11 -gt 0 ]]
[2026-07-06T21:11:35.619Z] ++ key=-e
[2026-07-06T21:11:35.619Z] ++ case $key in
[2026-07-06T21:11:35.619Z] ++ envvars+=("-e $2")
[2026-07-06T21:11:35.619Z] ++ shift
[2026-07-06T21:11:35.619Z] ++ shift
[2026-07-06T21:11:35.619Z] ++ [[ 9 -gt 0 ]]
[2026-07-06T21:11:35.619Z] ++ key=-e
[2026-07-06T21:11:35.619Z] ++ case $key in
[2026-07-06T21:11:35.619Z] ++ envvars+=("-e $2")
[2026-07-06T21:11:35.619Z] ++ shift
[2026-07-06T21:11:35.619Z] ++ shift
[2026-07-06T21:11:35.619Z] ++ [[ 7 -gt 0 ]]
[2026-07-06T21:11:35.619Z] ++ key=-e
[2026-07-06T21:11:35.619Z] ++ case $key in
[2026-07-06T21:11:35.619Z] ++ envvars+=("-e $2")
[2026-07-06T21:11:35.619Z] ++ shift
[2026-07-06T21:11:35.619Z] ++ shift
[2026-07-06T21:11:35.619Z] ++ [[ 5 -gt 0 ]]
[2026-07-06T21:11:35.619Z] ++ key=-e
[2026-07-06T21:11:35.619Z] ++ case $key in
[2026-07-06T21:11:35.619Z] ++ envvars+=("-e $2")
[2026-07-06T21:11:35.619Z] ++ shift
[2026-07-06T21:11:35.619Z] ++ shift
[2026-07-06T21:11:35.619Z] ++ [[ 3 -gt 0 ]]
[2026-07-06T21:11:35.619Z] ++ key=-e
[2026-07-06T21:11:35.619Z] ++ case $key in
[2026-07-06T21:11:35.619Z] ++ envvars+=("-e $2")
[2026-07-06T21:11:35.619Z] ++ shift
[2026-07-06T21:11:35.619Z] ++ shift
[2026-07-06T21:11:35.619Z] ++ [[ 1 -gt 0 ]]
[2026-07-06T21:11:35.619Z] ++ key=--copy-whls
[2026-07-06T21:11:35.619Z] ++ case $key in
[2026-07-06T21:11:35.619Z] ++ copy_whls=1
[2026-07-06T21:11:35.619Z] ++ shift
[2026-07-06T21:11:35.619Z] ++ [[ 0 -gt 0 ]]
[2026-07-06T21:11:35.619Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-07-06T21:11:35.619Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-07-06T21:11:35.619Z] ++ [[ false == \t\r\u\e ]]
[2026-07-06T21:11:35.619Z] ++ [[ true == \t\r\u\e ]]
[2026-07-06T21:11:35.619Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
[2026-07-06T21:11:35.619Z] ++ docker ps -a
[2026-07-06T21:11:35.619Z] CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[2026-07-06T21:11:35.619Z] ++ docker rm -f pytorch-ci-container
[2026-07-06T21:11:35.619Z] ++ docker pull rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-06T21:11:36.915Z] pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942: Pulling from rocm/pytorch-ci-private
[2026-07-06T21:11:37.730Z] 99e6462d03e0: Pulling fs layer
[2026-07-06T21:11:37.730Z] cb259a83ac3d: Pulling fs layer
[2026-07-06T21:11:37.730Z] 65f281ce2e70: Pulling fs layer
[2026-07-06T21:11:37.730Z] 78cb638d6686: Pulling fs layer
[2026-07-06T21:11:37.730Z] f5afebc61536: Pulling fs layer
[2026-07-06T21:11:37.730Z] 34e29023481f: Pulling fs layer
[2026-07-06T21:11:37.730Z] 0a0ffadd8b4f: Pulling fs layer
[2026-07-06T21:11:37.730Z] ed874654c321: Pulling fs layer
[2026-07-06T21:11:37.730Z] 8d78d529f04d: Pulling fs layer
[2026-07-06T21:11:37.730Z] c0664a7365cb: Pulling fs layer
[2026-07-06T21:11:37.730Z] 8d998c79d309: Pulling fs layer
[2026-07-06T21:11:37.730Z] fef65a673dc9: Pulling fs layer
[2026-07-06T21:11:37.730Z] 4106a853273f: Pulling fs layer
[2026-07-06T21:11:37.730Z] c608e00b1282: Pulling fs layer
[2026-07-06T21:11:37.730Z] 4f4fb700ef54: Pulling fs layer
[2026-07-06T21:11:37.730Z] 3d56742d201d: Pulling fs layer
[2026-07-06T21:11:37.730Z] 3d56742d201d: Pulling fs layer
[2026-07-06T21:11:37.730Z] ba1fabf878bd: Pulling fs layer
[2026-07-06T21:11:37.730Z] 0e6d9b5f5e8d: Pulling fs layer
[2026-07-06T21:11:37.730Z] 423e26ddb517: Pulling fs layer
[2026-07-06T21:11:37.730Z] 3d56742d201d: Pulling fs layer
[2026-07-06T21:11:37.730Z] 747272797eac: Pulling fs layer
[2026-07-06T21:11:37.730Z] 4c8a34755929: Pulling fs layer
[2026-07-06T21:11:37.730Z] dedc784bccba: Pulling fs layer
[2026-07-06T21:11:37.730Z] d9a8b776bea7: Pulling fs layer
[2026-07-06T21:11:37.730Z] 9548298aa5f4: Pulling fs layer
[2026-07-06T21:11:37.730Z] 8b3d007242f8: Pulling fs layer
[2026-07-06T21:11:37.730Z] 906ac4de0a2b: Pulling fs layer
[2026-07-06T21:11:37.730Z] 3d56742d201d: Pulling fs layer
[2026-07-06T21:11:37.730Z] b96eaed0e751: Pulling fs layer
[2026-07-06T21:11:37.730Z] 27fcad4d96a6: Pulling fs layer
[2026-07-06T21:11:37.730Z] 5edf52847c35: Pulling fs layer
[2026-07-06T21:11:37.730Z] 3d56742d201d: Pulling fs layer
[2026-07-06T21:11:37.730Z] db0af168a1a4: Pulling fs layer
[2026-07-06T21:11:37.730Z] d7df6379f9cc: Pulling fs layer
[2026-07-06T21:11:37.730Z] 1057fba0c7df: Pulling fs layer
[2026-07-06T21:11:37.730Z] f790d498bebb: Pulling fs layer
[2026-07-06T21:11:37.730Z] 42239a12ed73: Pulling fs layer
[2026-07-06T21:11:37.730Z] cccff27d82fb: Pulling fs layer
[2026-07-06T21:11:37.730Z] f3058198fba5: Pulling fs layer
[2026-07-06T21:11:37.730Z] 07b45639fc6f: Pulling fs layer
[2026-07-06T21:11:37.730Z] bf3e4cf00cd6: Pulling fs layer
[2026-07-06T21:11:37.730Z] 86cc05da26d1: Pulling fs layer
[2026-07-06T21:11:37.730Z] 9ae773ccc4b2: Pulling fs layer
[2026-07-06T21:11:37.730Z] 4cfd21f31c8b: Pulling fs layer
[2026-07-06T21:11:37.730Z] eab367f1ca39: Pulling fs layer
[2026-07-06T21:11:37.730Z] 78a3c5370db4: Pulling fs layer
[2026-07-06T21:11:37.730Z] 5f8dd10c9c78: Pulling fs layer
[2026-07-06T21:11:37.730Z] f0ef294e4f9b: Pulling fs layer
[2026-07-06T21:11:37.730Z] 284e8cbb8a05: Pulling fs layer
[2026-07-06T21:11:37.730Z] b09b7d04a8d9: Pulling fs layer
[2026-07-06T21:11:37.730Z] ed86e5a5d9b1: Pulling fs layer
[2026-07-06T21:11:37.730Z] 241cbb1c2f26: Pulling fs layer
[2026-07-06T21:11:37.730Z] 4f4fb700ef54: Pulling fs layer
[2026-07-06T21:11:37.730Z] 95d704643610: Pulling fs layer
[2026-07-06T21:11:37.730Z] 0fcc3b23bd99: Pulling fs layer
[2026-07-06T21:11:37.730Z] f391428cfe0f: Pulling fs layer
[2026-07-06T21:11:38.113Z] 69145e0460f6: Download complete
[2026-07-06T21:11:38.113Z] 65f281ce2e70: Download complete
[2026-07-06T21:11:38.495Z] ed874654c321: Download complete
[2026-07-06T21:11:38.495Z] 34e29023481f: Download complete
[2026-07-06T21:11:38.495Z] f790d498bebb: Download complete
[2026-07-06T21:11:38.495Z] 4f4fb700ef54: Download complete
[2026-07-06T21:11:38.495Z] f5afebc61536: Download complete
[2026-07-06T21:11:38.495Z] 284e8cbb8a05: Download complete
[2026-07-06T21:11:38.495Z] c608e00b1282: Download complete
[2026-07-06T21:11:38.495Z] bf3e4cf00cd6: Download complete
[2026-07-06T21:11:38.495Z] b96eaed0e751: Download complete
[2026-07-06T21:11:38.495Z] 27fcad4d96a6: Download complete
[2026-07-06T21:11:38.495Z] 9ae773ccc4b2: Download complete
[2026-07-06T21:11:38.495Z] 8d998c79d309: Download complete
[2026-07-06T21:11:38.495Z] 07b45639fc6f: Download complete
[2026-07-06T21:11:38.495Z] 241cbb1c2f26: Download complete
[2026-07-06T21:11:38.495Z] db0af168a1a4: Download complete
[2026-07-06T21:11:38.495Z] f0ef294e4f9b: Download complete
[2026-07-06T21:11:38.495Z] eab367f1ca39: Download complete
[2026-07-06T21:11:38.495Z] 0fcc3b23bd99: Download complete
[2026-07-06T21:11:38.495Z] 86cc05da26d1: Download complete
[2026-07-06T21:11:38.495Z] 78a3c5370db4: Download complete
[2026-07-06T21:11:38.495Z] f391428cfe0f: Download complete
[2026-07-06T21:11:38.495Z] 423e26ddb517: Download complete
[2026-07-06T21:11:38.495Z] 747272797eac: Download complete
[2026-07-06T21:11:38.495Z] 0a0ffadd8b4f: Download complete
[2026-07-06T21:11:38.495Z] c0664a7365cb: Download complete
[2026-07-06T21:11:38.495Z] 4cfd21f31c8b: Download complete
[2026-07-06T21:11:38.495Z] 8b3d007242f8: Download complete
[2026-07-06T21:11:38.495Z] 5f8dd10c9c78: Download complete
[2026-07-06T21:11:38.495Z] b09b7d04a8d9: Download complete
[2026-07-06T21:11:38.495Z] 3d56742d201d: Download complete
[2026-07-06T21:11:38.495Z] d7df6379f9cc: Download complete
[2026-07-06T21:11:38.495Z] 9548298aa5f4: Download complete
[2026-07-06T21:11:38.495Z] 42239a12ed73: Download complete
[2026-07-06T21:11:38.495Z] f3058198fba5: Download complete
[2026-07-06T21:11:38.495Z] 99e6462d03e0: Download complete
[2026-07-06T21:11:38.495Z] 8d78d529f04d: Download complete
[2026-07-06T21:11:38.873Z] fef65a673dc9: Download complete
[2026-07-06T21:11:38.873Z] ed86e5a5d9b1: Download complete
[2026-07-06T21:11:38.873Z] 906ac4de0a2b: Download complete
[2026-07-06T21:11:38.873Z] 0e6d9b5f5e8d: Download complete
[2026-07-06T21:11:40.198Z] 4106a853273f: Download complete
[2026-07-06T21:11:41.020Z] cb259a83ac3d: Download complete
[2026-07-06T21:11:41.830Z] 99e6462d03e0: Pull complete
[2026-07-06T21:11:41.830Z] cb259a83ac3d: Pull complete
[2026-07-06T21:11:43.713Z] 95d704643610: Download complete
[2026-07-06T21:11:46.320Z] ba1fabf878bd: Download complete
[2026-07-06T21:11:48.197Z] 4c8a34755929: Download complete
[2026-07-06T21:11:50.702Z] cccff27d82fb: Download complete
[2026-07-06T21:12:04.611Z] 1057fba0c7df: Download complete
[2026-07-06T21:12:11.009Z] 1057fba0c7df: Pull complete
[2026-07-06T21:12:11.009Z] 65f281ce2e70: Pull complete
[2026-07-06T21:12:11.009Z] 34e29023481f: Pull complete
[2026-07-06T21:12:11.009Z] c608e00b1282: Pull complete
[2026-07-06T21:12:11.009Z] 3d56742d201d: Pull complete
[2026-07-06T21:12:11.009Z] d7df6379f9cc: Pull complete
[2026-07-06T21:12:11.009Z] 8d78d529f04d: Pull complete
[2026-07-06T21:12:11.009Z] 906ac4de0a2b: Pull complete
[2026-07-06T21:12:11.009Z] 0e6d9b5f5e8d: Pull complete
[2026-07-06T21:12:39.639Z] 78cb638d6686: Download complete
[2026-07-06T21:13:50.036Z] 5edf52847c35: Download complete
[2026-07-06T21:14:31.289Z] ed874654c321: Pull complete
[2026-07-06T21:14:31.289Z] db0af168a1a4: Pull complete
[2026-07-06T21:14:31.289Z] 86cc05da26d1: Pull complete
[2026-07-06T21:14:31.289Z] 747272797eac: Pull complete
[2026-07-06T21:14:31.289Z] 0a0ffadd8b4f: Pull complete
[2026-07-06T21:14:31.289Z] 8b3d007242f8: Pull complete
[2026-07-06T21:14:31.289Z] 9548298aa5f4: Pull complete
[2026-07-06T21:14:31.289Z] 5edf52847c35: Pull complete
[2026-07-06T21:15:56.874Z] d9a8b776bea7: Download complete
[2026-07-06T21:16:38.376Z] dedc784bccba: Download complete
[2026-07-06T21:18:02.819Z] bf3e4cf00cd6: Pull complete
[2026-07-06T21:18:02.819Z] 27fcad4d96a6: Pull complete
[2026-07-06T21:18:02.819Z] 9ae773ccc4b2: Pull complete
[2026-07-06T21:18:02.819Z] dedc784bccba: Pull complete
[2026-07-06T21:18:07.964Z] f5afebc61536: Pull complete
[2026-07-06T21:18:07.964Z] f3058198fba5: Pull complete
[2026-07-06T21:18:07.964Z] 78cb638d6686: Pull complete
[2026-07-06T21:18:09.259Z] 07b45639fc6f: Pull complete
[2026-07-06T21:18:09.259Z] c0664a7365cb: Pull complete
[2026-07-06T21:18:09.259Z] cccff27d82fb: Pull complete
[2026-07-06T21:18:09.259Z] f790d498bebb: Pull complete
[2026-07-06T21:18:09.259Z] b96eaed0e751: Pull complete
[2026-07-06T21:18:09.636Z] 95d704643610: Pull complete
[2026-07-06T21:18:10.012Z] 4f4fb700ef54: Pull complete
[2026-07-06T21:18:10.012Z] 284e8cbb8a05: Pull complete
[2026-07-06T21:18:10.012Z] 241cbb1c2f26: Pull complete
[2026-07-06T21:18:10.012Z] f0ef294e4f9b: Pull complete
[2026-07-06T21:18:10.012Z] 0fcc3b23bd99: Pull complete
[2026-07-06T21:18:10.012Z] 78a3c5370db4: Pull complete
[2026-07-06T21:18:10.012Z] 423e26ddb517: Pull complete
[2026-07-06T21:18:10.012Z] 4cfd21f31c8b: Pull complete
[2026-07-06T21:18:10.012Z] 5f8dd10c9c78: Pull complete
[2026-07-06T21:18:10.012Z] b09b7d04a8d9: Pull complete
[2026-07-06T21:18:10.012Z] 42239a12ed73: Pull complete
[2026-07-06T21:18:10.012Z] 4106a853273f: Pull complete
[2026-07-06T21:19:08.872Z] d9a8b776bea7: Pull complete
[2026-07-06T21:19:08.872Z] eab367f1ca39: Pull complete
[2026-07-06T21:19:08.872Z] f391428cfe0f: Pull complete
[2026-07-06T21:19:08.872Z] 8d998c79d309: Pull complete
[2026-07-06T21:19:08.872Z] fef65a673dc9: Pull complete
[2026-07-06T21:19:08.872Z] ed86e5a5d9b1: Pull complete
[2026-07-06T21:19:08.872Z] ba1fabf878bd: Pull complete
[2026-07-06T21:19:08.872Z] 4c8a34755929: Pull complete
[2026-07-06T21:19:08.872Z] Digest: sha256:b836aa0e221ac85a2184ec24a67b9d5089f3caa45d182cf521dee7d38112aa81
[2026-07-06T21:19:08.872Z] Status: Downloaded newer image for rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-06T21:19:08.872Z] docker.io/rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-06T21:19:08.872Z] ++ GPU_FLAGS=
[2026-07-06T21:19:08.872Z] ++ echo 'NODE_LABELS: k8s-mi250-1-mv3pl gfx90a mlir k8s-mi250-1-mv3pl gfx90a mlir'
[2026-07-06T21:19:08.872Z] NODE_LABELS: k8s-mi250-1-mv3pl gfx90a mlir k8s-mi250-1-mv3pl gfx90a mlir
[2026-07-06T21:19:08.872Z] ++ [[ ! k8s-mi250-1-mv3pl gfx90a mlir =~ build-only ]]
[2026-07-06T21:19:08.872Z] ++ GPU_FLAGS='--group-add video --device=/dev/kfd --device=/dev/dri'
[2026-07-06T21:19:08.872Z] ++++ dirname prepare_docker_env.sh
[2026-07-06T21:19:08.872Z] +++ cd .
[2026-07-06T21:19:08.872Z] +++ pwd
[2026-07-06T21:19:08.872Z] ++ script_dir=/home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3393
[2026-07-06T21:19:08.872Z] ++ docker run --name pytorch-ci-container -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=inductor -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip -e PREBUILT=false -e TEST_CORE=true -e TEST_CORE=true -v /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3393:/host_workspace -v /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3393:/scripts --entrypoint bash -d --network=host --group-add video --device=/dev/kfd --device=/dev/dri --ipc=host --pid=host --shm-size 8G --cap-add=SYS_PTRACE --security-opt seccomp=unconfined rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942 -lc 'sleep infinity'
[2026-07-06T21:19:08.872Z] 93e266fb4c09b84df9e8923e78530f65a8c77d89513784c5f9129e772605f6c5
[2026-07-06T21:19:09.413Z] ++ env -i python3 /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch/tools/stats/export_test_times.py
[2026-07-06T21:19:20.069Z] Exporting test times from test-infra
[2026-07-06T21:19:20.069Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch/.additional_ci_files/test-times.json
[2026-07-06T21:19:20.069Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch/.additional_ci_files/test-class-times.json
[2026-07-06T21:19:20.069Z] ++ python3 /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch/tools/stats/export_test_times.py
[2026-07-06T21:19:20.069Z] Exporting test times from test-infra
[2026-07-06T21:19:20.069Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch/.additional_ci_files/test-times.json
[2026-07-06T21:19:20.069Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch/.additional_ci_files/test-class-times.json
[2026-07-06T21:19:20.069Z] ++ [[ -d /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch/.additional_ci_files ]]
[2026-07-06T21:19:20.069Z] ++ sed -i -E 's/"\S+rocm\S+"/""/g' /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch/.additional_ci_files/test-times.json
[2026-07-06T21:19:20.069Z] ++ [[ false != \t\r\u\e ]]
[2026-07-06T21:19:20.069Z] ++ docker exec -u root pytorch-ci-container bash -c 'rm -rf /var/lib/jenkins/pytorch'
[2026-07-06T21:19:20.069Z] ++ docker cp /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3393/pytorch pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-06T21:19:24.305Z] ++ docker exec -u root pytorch-ci-container bash -c 'chown root:root -R /var/lib/jenkins/pytorch'
[2026-07-06T21:19:24.306Z] ++ [[ 1 -eq 1 ]]
[2026-07-06T21:19:24.306Z] ++ docker cp /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3393/rocm_composable_kernel-7.1.1.dev0+g4266f8-py3-none-any.whl /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3393/torch-2.14.0a0+git3ec517c-cp312-cp312-linux_x86_64.whl pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-06T21:19:24.306Z] docker: 'docker cp' requires 2 arguments
[2026-07-06T21:19:24.306Z]
[2026-07-06T21:19:24.306Z] Usage: docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
[2026-07-06T21:19:24.306Z] docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH
[2026-07-06T21:19:24.306Z]
[2026-07-06T21:19:24.306Z] Run 'docker cp --help' for more information
Tests / Test Inductor / Test Inductor / Run pytorch_inductor_1 / Error signal
Error in error step, with arguments pytorch_inductor_1 failed.
pytorch_inductor_1 failed
Details
- Kill older PR Builds (1.9 sec)
- Initialize (2 hr 56 min)
- Download CI scripts (27 sec)
- Checkout Pytorch (1 min 37 sec)
- Check base Docker image existence (7.7 sec)
- Pull Docker Image (7 min 51 sec)
- Build PyTorch (2 hr 45 min)
- Tests (26 min)
- Test PyTorch (6 ms)
- Test Distributed (8 ms)
- Test Inductor (7 ms)
- Test PyTorch Slow (7 ms)
- Test PyTorch Slow (8.6 sec)
- Microbenchmark (16 sec)
- Microbenchmark (8.6 sec)
- Post Build (3.1 sec)
- Declarative: Post Actions (4.9 sec)
Loading