Skip to content

CI

CI #4985

Triggered via schedule October 25, 2025 09:30
Status Failure
Total duration 1h 38m 49s
Artifacts 46

ci.yaml

on: schedule
metadata
3s
metadata
bump-manifest
18s
bump-manifest
Matrix: amd64 / test-distribution
Matrix: arm64 / test-distribution
amd64  /  ...  /  build-base
2m 52s
amd64 / build-base / build-base
arm64  /  ...  /  build-base
3m 34s
arm64 / build-base / build-base
amd64  /  ...  /  build-mpi-operator-compatible-base
2m 41s
amd64 / test-nccl / build-mpi-operator-compatible-base
amd64  /  ...  /  build-nccl-gke
1m 33s
amd64 / test-nccl / nccl-test-gke / build-nccl-gke
arm64  /  ...  /  build-mpi-operator-compatible-base
arm64 / test-nccl / build-mpi-operator-compatible-base
arm64  /  ...  /  build-nccl-gke
arm64 / test-nccl / nccl-test-gke / build-nccl-gke
Matrix: amd64 / test-jax-cutlass-h100 / jax-cutlass-test-h100
Matrix: amd64 / test-jax / run-unit-test
Matrix: amd64 / test-te-a100 / run-unit-test
Matrix: amd64 / test-te-h100 / te-test-h100
amd64  /  ...  /  launch-slurm-runner
22m 21s
amd64 / test-jax / runner / launch-slurm-runner
amd64  /  test-nsys-jax-eks
2m 22s
amd64 / test-nsys-jax-eks
amd64  /  ...  /  launch-slurm-runner
12m 46s
amd64 / test-te-a100 / runner / launch-slurm-runner
amd64  /  build-upstream-t5x
7m 28s
amd64 / build-upstream-t5x
Matrix: amd64 / test-nsys-jax / run-unit-test
amd64  /  ...  /  launch-slurm-runner
17m 9s
amd64 / test-nsys-jax / runner / launch-slurm-runner
Matrix: amd64 / test-nccl / nccl-test
Matrix: amd64 / test-nccl / nccl-test-gke / nccl-gke
Matrix: arm64 / test-jax-cutlass-h100 / jax-cutlass-test-h100
Waiting for pending jobs
Matrix: arm64 / test-jax / run-unit-test
Waiting for pending jobs
Matrix: arm64 / test-te-a100 / run-unit-test
Waiting for pending jobs
Matrix: arm64 / test-te-h100 / te-test-h100
Waiting for pending jobs
arm64  /  test-nsys-jax-eks
0s
arm64 / test-nsys-jax-eks
arm64  /  ...  /  launch-slurm-runner
arm64 / test-jax / runner / launch-slurm-runner
arm64  /  ...  /  launch-slurm-runner
arm64 / test-te-a100 / runner / launch-slurm-runner
arm64  /  build-upstream-t5x
11m 19s
arm64 / build-upstream-t5x
Matrix: arm64 / test-nsys-jax / run-unit-test
Waiting for pending jobs
arm64  /  ...  /  launch-slurm-runner
arm64 / test-nsys-jax / runner / launch-slurm-runner
Matrix: arm64 / test-nccl / nccl-test
Waiting for pending jobs
Matrix: arm64 / test-nccl / nccl-test-gke / nccl-gke
Waiting for pending jobs
amd64  /  ...  /  maxtext-gke-xpk
2m 1s
amd64 / test-maxtext-gke / maxtext-gke-xpk
Matrix: amd64 / test-maxtext / maxtext-multinode
Matrix: amd64 / test-maxtext / single-process-multi-device
amd64  /  ...  /  build-rosetta
14m 13s
amd64 / build-rosetta-t5x / build-rosetta
amd64  /  test-axlearn-eks
2m 16s
amd64 / test-axlearn-eks
amd64  /  test-axlearn-fuji-models-eks
49s
amd64 / test-axlearn-fuji-models-eks
Matrix: amd64 / test-nsys-jax-archive
arm64  /  ...  /  maxtext-gke-xpk
arm64 / test-maxtext-gke / maxtext-gke-xpk
Matrix: arm64 / test-maxtext / maxtext-multinode
Waiting for pending jobs
Matrix: arm64 / test-maxtext / single-process-multi-device
Waiting for pending jobs
arm64  /  ...  /  build-rosetta
17m 30s
arm64 / build-rosetta-t5x / build-rosetta
arm64  /  test-axlearn-eks
0s
arm64 / test-axlearn-eks
arm64  /  test-axlearn-fuji-models-eks
0s
arm64 / test-axlearn-fuji-models-eks
Matrix: arm64 / test-nsys-jax-archive
amd64  /  ...  /  test-maxtext-metrics
15s
amd64 / test-maxtext / test-maxtext-metrics
amd64  /  collect-docker-tags
2s
amd64 / collect-docker-tags
Matrix: amd64 / test-rosetta-t5x / vit-multi-gpu-multi-node
arm64  /  ...  /  test-maxtext-metrics
arm64 / test-maxtext / test-maxtext-metrics
arm64  /  collect-docker-tags
2s
arm64 / collect-docker-tags
Matrix: arm64 / test-rosetta-t5x / vit-multi-gpu-multi-node
Waiting for pending jobs
amd64  /  ...  /  sitrep
10s
amd64 / test-maxtext / test-maxtext-sitrep / sitrep
amd64  /  ...  /  test-t5x-rosetta-summary
4s
amd64 / test-rosetta-t5x / test-t5x-rosetta-summary
amd64  /  ...  /  test-t5x-rosetta-metrics
17s
amd64 / test-rosetta-t5x / test-t5x-rosetta-metrics
arm64  /  ...  /  sitrep
arm64 / test-maxtext / test-maxtext-sitrep / sitrep
arm64  /  ...  /  test-t5x-rosetta-summary
arm64 / test-rosetta-t5x / test-t5x-rosetta-summary
arm64  /  ...  /  test-t5x-rosetta-metrics
arm64 / test-rosetta-t5x / test-t5x-rosetta-metrics
amd64  /  ...  /  test-maxtext-outcome
3s
amd64 / test-maxtext / test-maxtext-outcome
amd64  /  ...  /  sitrep
5s
amd64 / test-rosetta-t5x / test-t5x-rosetta-sitrep / sitrep
arm64  /  ...  /  test-maxtext-outcome
arm64 / test-maxtext / test-maxtext-outcome
arm64  /  ...  /  sitrep
arm64 / test-rosetta-t5x / test-t5x-rosetta-sitrep / sitrep
amd64  /  ...  /  test-t5x-rosetta-outcome
2s
amd64 / test-rosetta-t5x / test-t5x-rosetta-outcome
arm64  /  ...  /  test-t5x-rosetta-outcome
arm64 / test-rosetta-t5x / test-t5x-rosetta-outcome
make-publish-configs
4s
make-publish-configs
merge-new-manifest
9s
merge-new-manifest
Matrix: publish-containers
finalize  /  workflow-badge
6s
finalize / workflow-badge
finalize  /  report
10s
finalize / report
finalize  /  upload-badge
8s
finalize / upload-badge
finalize  /  publish-badge
4s
finalize / publish-badge
Fit to window
Zoom out
Zoom in

Annotations

11 errors and 2 warnings
amd64 / test-te-h100 / te-test-h100 (unittest, 8)
Process completed with exit code 1.
amd64 / test-jax / jax-A100-unit-test
Process completed with exit code 1.
amd64 / test-maxtext-gke / maxtext-gke-xpk
Process completed with exit code 1.
amd64 / test-nsys-jax / nsys-jax-A100-unit-test
EACCES: permission denied, scandir '/runner/_work/JAX-Toolbox/JAX-Toolbox/pytest-tmp'
amd64 / test-nsys-jax / nsys-jax-A100-unit-test
Process completed with exit code 1.
amd64 / test-nsys-jax / nsys-jax-A100-unit-test
Process completed with exit code 1.
amd64 / test-te-a100 / te-A100-unit-test
Process completed with exit code 1.
amd64 / test-maxtext / test-maxtext-metrics
Process completed with exit code 1.
amd64 / test-maxtext / test-maxtext-outcome
Process completed with exit code 1.
amd64 / test-rosetta-t5x / test-t5x-rosetta-metrics
Process completed with exit code 1.
amd64 / test-rosetta-t5x / test-t5x-rosetta-outcome
Process completed with exit code 1.
merge-new-manifest
Unexpected input(s) 'owner_and_repo', valid inputs are ['route', 'mediaType']
merge-new-manifest
Unexpected input(s) 'owner_and_repo', 'head', 'base', 'body', 'title', 'draft', valid inputs are ['route', 'mediaType']

Artifacts

Produced during runtime
Name Size Digest
artifact-axlearn-build-amd64
566 Bytes
sha256:e13b28fe52442fe40272317ec231354ac9c5696182cd11a4e37737b2f286f4d6
artifact-axlearn-build-arm64
566 Bytes
sha256:082a14e25f56dd4c2d60e00a267ca1d2750253a1281d365f00fe25f9c994059b
artifact-axlearn-test
35.9 KB
sha256:5338f21d2ecd79fcb6774883973aba52af4be32c194407af3459fd1ebbd765b5
artifact-base-build-amd64
566 Bytes
sha256:257e96790b3781e053141923cb8866b8e771b993fc8ec20759b34e789755fb8c
artifact-base-build-arm64
567 Bytes
sha256:fd80b2664ac9300c8f801bccf445f50cfffcfe143063be0f370f23e445e9b079
artifact-equinox-build-amd64
569 Bytes
sha256:5859dbd31dfe97a555dfb314f629cecfbb6bef76b2a4853d328cf2dee0c8c0c9
artifact-equinox-build-arm64
568 Bytes
sha256:b74bc07c2eccf23229e4e4a8fd649129181fed251e4e0a5b74a1299ba8ce8170
artifact-final-report
3.07 KB
sha256:53b18f7b0a313733a11557ad01a3260d081dedcc14e0e4b863c7fb5069b1a7c4
artifact-jax-build-amd64
555 Bytes
sha256:dc943085c744a6251d8ab3e126b46190780b3bce9f689e84feab4d51ef56c8eb
artifact-jax-build-arm64
552 Bytes
sha256:e1057e1f2c1ffd54067aca9591c68b11c85a1c24db349635543511850a3805ba
artifact-maxtext-build-amd64
568 Bytes
sha256:3b7db46b5eebb97f5b94c2edb910b9f0a89c5b37a726a00f583a0a017832e62a
artifact-maxtext-build-arm64
568 Bytes
sha256:7c486ca5f4161a83929d39a7d553154bea27e6113be997c105564f2faadea983
artifact-maxtext-test
631 Bytes
sha256:87f1542659d0166ef6550d86a84d66cfd745417f844324046a0dfa4f7812ae3c
artifact-mpi-operator-compatible-base-build-amd64
638 Bytes
sha256:254d15d96dce2d90a7bdc1e439328f823f16fd8e4118dd5e7f681894fe437bb5
artifact-nccl-gke-build-amd64
571 Bytes
sha256:cd49ac8a9b75ffd8a21e561f733d07c979f117f9a678d13da0058b0e987e3172
artifact-rosetta-build-t5x-amd64
584 Bytes
sha256:f7227aaa48fbdd9a25366e334f2d9aaddc2dced96b6044cd0b043eb0ab56d9f0
artifact-rosetta-build-t5x-arm64
584 Bytes
sha256:ba20cfb37376ed1b9e87c5166cc178118e6a3fe99e6ea23b2db05b50d8218bfc
artifact-rosetta-t5x-mgmn-test
624 Bytes
sha256:7313b39c5d0caf6efdf57d0737450c7b6be4f5758caa1ed5467a2fb022ffc257
artifact-t5x-build-amd64
568 Bytes
sha256:8458337376dda50aa522f2883c5f02b4da77c906a50dd4f5ff97c730578ef60a
artifact-t5x-build-arm64
567 Bytes
sha256:a2fccf56f1a0c8e5e1971abcfeb6d060c46a9ecfe1f270c587878552ae29a99e
artifact-workflow-metadata
277 Bytes
sha256:41c4db0d147d22843f0597cb37fec298cded98e5b95652454179cc2d42fe2ec9
bumped-manifest
51.5 KB
sha256:b413038aeb9dd3f6ec417c96246798f1dfcd7e8fa9858b2f6713a0d9d8f54fbf
final-axlearn
258 Bytes
sha256:18e26b33a60d0f0b96b60e548129e64c99811c523149648341c3a9decb9fb632
final-base
249 Bytes
sha256:e247e351a0856b76b01bb144b8ab4f2d79ab6c935061555be780b237e3fd7ac2
final-equinox
258 Bytes
sha256:67cd6f75c307fa03dead9c1559594268379bd3b3816e6c6806f677c15e75814f
final-jax
246 Bytes
sha256:3acfc6027b2c35eadd9e1094a4b9f621775903657657315c18c31095cf037ffe
final-maxtext
258 Bytes
sha256:5d2b609edbd5f26af65d21d8d963ec90599f63ff9e34ef4d2bb88ed6db06aef9
final-t5x
246 Bytes
sha256:16d9d904c890a19587ea7cfd64cf1277471405ecbb02ff85f50f38250d9f7b1c
final-upstream-t5x
273 Bytes
sha256:ddb99c582ff994560ca68dd0389d4e586178fd34e62e363b0f49f9bb7db07b08
jax-cutlass-test-H100
1.24 KB
sha256:4839d8e5b2dce95d11f9ae10ea745b12007fdfc5a55bdb42ee5ad6dba76bce25
jax-unit-test-A100
84 KB
sha256:a316581507f9f64b2aa0d446b212986126c306ea97d73ee6c9e5b117aaeb075c
mealkit-axlearn
268 Bytes
sha256:b2406e4f0b9d735fa5b39c992c5d9b0842aed7740ffe3c1a86850e0194dd40ec
mealkit-equinox
268 Bytes
sha256:0cc3eba42d9fd69781acf879cdb161151ae1e898869c894ee2934b1ff9a88ccf
mealkit-jax
256 Bytes
sha256:758c82aea8bf46899d73b22772bd489e6e901f099cbdec837019880ea104b947
mealkit-maxtext
268 Bytes
sha256:6e9b3082b43215360e06b698319e44c18c0650c30c63acdf9edd3461d8b7b71f
mealkit-t5x
257 Bytes
sha256:b152dbd3964b107bb5d2b1f45f82d7b932afd2a20eea3bc0ca99db2910344238
mealkit-upstream-t5x
281 Bytes
sha256:0986eec1340d6a9418a9f460e473e33ca239093a04dc3f35ce6fef33cd73e5c8
nccl-gke-all-gather
15.3 KB
sha256:b3f28389f63d86bd490c13a46e08e183034d30547e70d86cc1695d3fefcbb44b
nccl-gke-all-reduce
15.6 KB
sha256:ca6196ce2fe0a56440486e8b9ae27ad3faf215d6c5f986232a506835d582cd05
nccl-gke-broadcast
15.3 KB
sha256:248dac7044a0d478efa496d9167084d80b7b100c636219a16378ec52b32b8197
nccl-gke-reduce-scatter
15.5 KB
sha256:ee4c0b35c8b0ed728eee38cc897ae93a059e57d1a15fb08d2510e0e39925c882
rosetta-t5x-vit-18801305410-VIT8G1N
2.23 KB
sha256:ce37905a5d52749682f6f6da488ef986816d3e056314f6926d464b344951329a
te-unit-test-A100
13.1 KB
sha256:4912ce504ba4082bc87e2375f6825d23998e41bd6e9c03852f84d5e0e76bb880
te-unit-test-H100
10.3 KB
sha256:43470fc1ec0587f3d8dc36e1fce7b4b8190dd2c4392aeba6d5be6dab3791ef0e
upstream-maxtext-18801305410-1DP2FSDP4TP1PP_single_process
2.9 KB
sha256:e40087c23970287b6909cec60cc97e57e7bd3be51d4a6ca544171262d7b4fec9
upstream-maxtext-18801305410-2DP2FSDP2TP1PP
4.63 KB
sha256:faa0f28f94510cb0216f700b93102eff79dcf0e32a88ee41e57653a4104f92ed