Skip to content

[test] test: strengthen model contract regressions #16394

[test] test: strengthen model contract regressions

[test] test: strengthen model contract regressions #16394

Triggered via push June 16, 2026 00:24
Status Failure
Total duration 3h 1m 45s
Artifacts 57

cicd-main.yml

on: push
pre-flight  /  pre-flight
12s
pre-flight / pre-flight
configure
7s
configure
Lint check
20s
Lint check
cicd-wait-in-queue
4s
cicd-wait-in-queue
cicd-compute-build-matrix
3s
cicd-compute-build-matrix
Matrix: cicd-container-build
Launch_Unit_Tests_Core
6m 50s
Launch_Unit_Tests_Core
Launch_Unit_Tests_Diffusion
2m 30s
Launch_Unit_Tests_Diffusion
generate-test-matrix
8s
generate-test-matrix
generate-gb200-test-matrix
7s
generate-gb200-test-matrix
Launch_Import_Check
1m 55s
Launch_Import_Check
Matrix: cicd-functional-tests-l1
Matrix: cicd-functional-tests-l2
Matrix: cicd-functional-tests-l0
Matrix: cicd-functional-tests-flaky
Matrix: cicd-functional-tests-gb200-l1
Matrix: cicd-functional-tests-gb200-l2
Matrix: cicd-functional-tests-gb200-l0
Matrix: cicd-functional-tests-gb200-flaky
Nemo_CICD_Test
10s
Nemo_CICD_Test
Coverage_Fake
Coverage_Fake
Matrix: Coverage
Fit to window
Zoom out
Zoom in

Deployment protection rules

Reviewers, timers, and other rules protecting deployments in this run
Event Environments Comment
svcnvidia-nemo-ci
approved Jun 16, 2026
test

Automatically approved by queue manager

Annotations

7 errors, 3 warnings, and 57 notices
cicd-container-build (aws, 766267172432.dkr.ecr.us-east-1.amazonaws.com, nemo-ci-aws-gpu-x2)
WARNING: The directory '/home/runner/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
cicd-container-build (aws, 766267172432.dkr.ecr.us-east-1.amazonaws.com, nemo-ci-aws-gpu-x2)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
cicd-container-build (gcp, us-east4-docker.pkg.dev/nv-projdgxchipp-20260113193621/megatron-bridge...
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
L0_Launch_training
Process completed with exit code 1.
Result
❌ L0_Launch_training — FAILED (exit 1)
L0_Launch_training
Process completed with exit code 1.
Nemo_CICD_Test
Process completed with exit code 1.
pre-flight / pre-flight
Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: actions/checkout@v4, step-security/changed-files@v45.0.1. Actions will be forced to run with Node.js 24 by default starting June 16th, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
cicd-container-build (aws, 766267172432.dkr.ecr.us-east-1.amazonaws.com, nemo-ci-aws-gpu-x2)
Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: docker/build-push-action@v5, docker/setup-buildx-action@v3. Actions will be forced to run with Node.js 24 by default starting June 16th, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
cicd-container-build (gcp, us-east4-docker.pkg.dev/nv-projdgxchipp-20260113193621/megatron-bridge...
Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: docker/build-push-action@v5, docker/setup-buildx-action@v3. Actions will be forced to run with Node.js 24 by default starting June 16th, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
Result
✅ Launch_Unit_Tests_Diffusion — PASSED
Result
✅ Launch_Unit_Tests_Core — PASSED
Result
✅ L0_Launch_models_nemotron_omni — PASSED
Result
✅ L0_Launch_models_nemotron_omni — PASSED
Result
✅ L0_Launch_models_qwen3_omni — PASSED
Result
✅ L0_Launch_models_kimi_k25_vl — PASSED
Result
✅ L0_Launch_models_wan — PASSED
Result
✅ L0_Launch_converter_fsdp — PASSED
Result
✅ L0_Launch_recipes_gemma — PASSED
Result
✅ L0_Launch_recipes_deepseek — PASSED
Result
✅ L0_Launch_models_bailing — PASSED
Result
✅ L0_Launch_recipes_gemma4_vl — PASSED
Result
✅ L0_Launch_recipes_deepseek_fsdp — PASSED
Result
✅ L0_Launch_models_minimax_m2 — PASSED
Result
✅ L0_Launch_data — PASSED
Result
✅ L0_Launch_models_gemma4_vl — PASSED
Result
✅ L0_Launch_models_deepseek — PASSED
Result
✅ L0_Launch_models_flux — PASSED
Result
✅ L0_Launch_models_glm_vl — PASSED
Result
✅ L0_Launch_megatron_fsdp — PASSED
Result
✅ L0_Launch_models_llada15 — PASSED
Result
✅ L0_Launch_models_deepseek — PASSED
Result
✅ L0_Launch_models_kimi_k25_vl — PASSED
Result
✅ L0_Launch_models_flux — PASSED
Result
✅ L0_Launch_models_minimax_m2 — PASSED
Result
✅ L0_Launch_models_bailing — PASSED
Result
✅ L0_Launch_models_qwen3_omni — PASSED
Result
✅ L0_Launch_models_nemotronh — PASSED
Result
✅ L0_Launch_models_qwen3 — PASSED
Result
✅ L0_Launch_models_glm_vl — PASSED
Result
✅ L0_Launch_converter_generate — PASSED
Result
✅ L0_Launch_recipes_glm_vl — PASSED
Result
✅ L0_Launch_data — PASSED
Result
✅ L0_Launch_models_qwen3 — PASSED
Result
✅ L0_Launch_recipes_nemotron_omni — PASSED
Result
✅ L0_Launch_converter — PASSED
Result
✅ L0_Launch_recipes_llama_1b — PASSED
Result
✅ L0_Launch_converter — PASSED
Result
✅ L0_Launch_models_nemotronh — PASSED
Result
✅ L0_Launch_recipes_perf_config — PASSED
Result
✅ L0_Launch_models_wan — PASSED
Result
✅ L0_Launch_training_megatron_mimo — PASSED
Result
✅ L0_Launch_utils — PASSED
Result
✅ L0_Launch_vlm_inference — PASSED
Result
✅ L0_Launch_wan_ckpt_conversion — PASSED
Result
✅ L0_Launch_recipes_deepseek — PASSED
Result
✅ L0_Launch_recipes_glm_vl — PASSED
Result
✅ L0_Launch_recipes_gemma — PASSED
Result
✅ L0_Launch_recipes_llama_1b — PASSED
Result
✅ L0_Launch_recipes_gemma4_vl — PASSED
Result
✅ L0_Launch_recipes_perf_config — PASSED
Result
✅ L0_Launch_recipes_nemotron_omni — PASSED
Result
✅ L0_Launch_training_megatron_mimo — PASSED
Result
✅ L0_Launch_utils — PASSED
Result
✅ L0_Launch_wan_ckpt_conversion — PASSED
Result
✅ L0_Launch_vlm_inference — PASSED
Result
✅ L0_Launch_recipes_nemotronh — PASSED

Artifacts

Produced during runtime
Name Size Digest
coverage-e2e-27585485819-012df6c2-9a1d-4ed0-b7bc-0f4a2360764a
1.29 KB
sha256:98f6cf842ac6cb58bbd1fce9db7e4091d367ae4e6c5691c18dd2fd4e53028ef3
coverage-e2e-27585485819-09e3141b-a931-47ca-b50b-93e0228cfafe
233 KB
sha256:ebd2350cb848b4de18a9851bab2fab1d6edca4829782b14445a698193db509c5
coverage-e2e-27585485819-0c68ea9d-b698-44e4-bd6b-3c9474d4ddf1
719 KB
sha256:af3f1ce46f5d536e9bc8c88bb8fa761d6d647da7a2c2e83e8965953a80388005
coverage-e2e-27585485819-108edace-d09c-406e-8cc1-7a24d7e83237
234 KB
sha256:f324c9b509a1fb1610d13aedaae602573383a4368964445f7a5361cf4dd78d7f
coverage-e2e-27585485819-1446f306-3bdf-45b3-b2d2-4f7bdfadf99c
345 KB
sha256:1bd1a3f358173c341d6c8baca3478397dec2364137a98158bfc9de05cfefb628
coverage-e2e-27585485819-17361dae-c814-42f8-8532-f50b3933fa22
305 KB
sha256:249f11613f6962530af12e2ad1b453e5d5a5eb7fffa31d2cced299f8e1e6321a
coverage-e2e-27585485819-1d63eac9-c502-4e30-8a07-e23faf7abc5c
264 KB
sha256:85db02058cdecf10c42557530bc04874155be8156b11405f6c3e602a13f23290
coverage-e2e-27585485819-263f8ac3-b1f5-47e9-b158-2b300928b128
700 KB
sha256:88be9d865ac00fdea6f4a959c2e0bf6d4159c52724abf778e1aad250ac621190
coverage-e2e-27585485819-29986bb5-d3c1-40da-99a1-2d509da59462
311 KB
sha256:27ccfc5020308b01a64a2dd906dbe5d80d9adcf530b63cf1f868fafd90c087e2
coverage-e2e-27585485819-2ca71765-0124-4d20-a9f4-b022b6a1804b
233 KB
sha256:2ba393fd849b97b8470c357468fcb1bd9043b93a0031ec71730a15e020b0a666
coverage-e2e-27585485819-37210dbd-a3fb-4a2f-90a8-e2f2e0e525cf
314 KB
sha256:e98f9072b954e4c93198bfb2af57d9a90a21372cec4ca807d680a2b0192766db
coverage-e2e-27585485819-3d03055c-6661-4842-b815-964e7bd19ad3
759 KB
sha256:85f3bbff20b632aa8ca5967884609305a3d3a64ac09fcd3b5b4d9aa79aa18a9e
coverage-e2e-27585485819-3f930c60-f6e1-474f-9cd6-a0e4ef232bec
233 KB
sha256:52880c7de0f5b3fe8ecbbb9673c931d04c2b82d6febe9944e3520d9868ce759f
coverage-e2e-27585485819-40567597-a969-4d82-b176-ccafa91381b4
340 KB
sha256:6f2db5deb7e092f0fcfa349f5fef869e2e32c951c7432b7a8cb56eee3f2c0d66
coverage-e2e-27585485819-490cddca-736c-4156-82d1-893404deceab
348 KB
sha256:2319c536696b253c42fc0f2076319e4092190f25e79844b9552bb6ef3631df6d
coverage-e2e-27585485819-55c0b0fd-8b15-49c4-82e3-0991321f726b
348 KB
sha256:642ffd853f25cd69136b22d05fdee935ee93e365c62ef9f7a38f0fde66604e94
coverage-e2e-27585485819-5697d034-e847-4293-8bea-898601e3c83d
311 KB
sha256:52499a29198f5543de678d220c75d6c8e0f469dced43b053d973b6073b56d84c
coverage-e2e-27585485819-56d594f4-8171-4dec-aa29-8796620b4b65
760 KB
sha256:12693816f0e6f853c58ab3058405bcbc881484321eb084b6ef4677d34c0ccf39
coverage-e2e-27585485819-61a993a0-fe2e-4472-94d5-f056d6f0fa4e
253 KB
sha256:1e3c7cf2536cb77cb0aa5120feaa914906422b94206b5ccd3e5e3b7d15ffdbbb
coverage-e2e-27585485819-667da861-d951-4d91-a2df-96cd85f05229
699 KB
sha256:f00b68d2c12a21b71ab9818a129e928b50437fdc032c07a19f664d606da6e0f4
coverage-e2e-27585485819-6b522216-4051-4ae5-a46e-b8da2bb3a8cd
313 KB
sha256:c61d6024b7aebc01ec1bea56d2ca03db1a7bb7355dbfc892ac00a46b4677becd
coverage-e2e-27585485819-836e1250-424c-4da4-a4c5-10eb5375fd74
699 KB
sha256:15a78befc1600cc93639f2288979f9ab62c9ffb7376bae79b59a09e2472a9e2b
coverage-e2e-27585485819-8f54c58c-5c82-4faa-8d29-4128471a87cb
700 KB
sha256:19b16001485ce3ac3b461b3caffa032a57f347836456a7b019ed0f7be163679d
coverage-e2e-27585485819-8f845aa8-66b6-4e9e-999c-3e7d96a11a0e
760 KB
sha256:cba5109d395a30c4acf5e75d56d8d4603aeef0125e78e1c164b35b7c39039614
coverage-e2e-27585485819-9162cc3c-8978-4f4b-8b39-6989b18d1b08
235 KB
sha256:1ad0d551ec6b947dcb2e0fa42b7f251c6e591bdec88d95fc084e2734fd044760
coverage-e2e-27585485819-933f4401-861c-4877-a711-13043e949333
346 KB
sha256:0e67ca51125bcd7bf7b5c31ff2cd3186e478e8cb9ee1624cc650c6e58aec51bb
coverage-e2e-27585485819-9857159b-145a-4232-801f-3a05d6b88baf
294 KB
sha256:21d004bbc9ee85a25de927702d44218aed6538701f3a168073c38bc700db5bab
coverage-e2e-27585485819-9870188c-0aac-4e28-afea-8b0c163279fe
305 KB
sha256:3d81fe98aef3cf287d75516d57c555634e02b2259ee9e3b42c5d512875d98700
coverage-e2e-27585485819-9bb24e78-deac-41ce-ad4a-09a4231e8670
267 KB
sha256:161a4ea00b4f1adcb58d0f2a0a60f8ac5e6deddfd323c10bcf08d6b694a07d80
coverage-e2e-27585485819-9e01d149-a2c9-42eb-8f15-aa28adeeaab2
313 KB
sha256:17655fa507b96edfdbafbe1c117c56e53ad6e23bf860b022b174848f27847a6d
coverage-e2e-27585485819-a281a085-d1e1-40e4-82f0-b36518439c42
264 KB
sha256:331fd35874e4e28834046d3d567972e56062daff0a5ab701bc22b1bcd705b994
coverage-e2e-27585485819-a3da3fae-d646-4065-a61f-39a89c626dc2
355 KB
sha256:150580db41f511bb05acc469a5856c128286ecbe09fb7763b7dc37d5347c8410
coverage-e2e-27585485819-a43cb292-0619-492a-a775-e369cda8f9c1
758 KB
sha256:4785d2889876fc12378f9a5c3c3a667c2a9527aa289b47bef64cd3030a2e2267
coverage-e2e-27585485819-a878b3d5-19d0-4b06-9e01-9db56965632b
346 KB
sha256:0ada6824fc4377b35b2546632965c50b8a9b0c253fcfcffab6ee72b678684f15
coverage-e2e-27585485819-ad499aa5-6f7f-414f-9863-55b9646fc87d
719 KB
sha256:4c293cb0d265517d63bbcb5927a69ef6b72f7707021fa4bf22a21e0ebe7efcd1
coverage-e2e-27585485819-b15820c7-7001-4d5e-8dc0-83f98961f616
233 KB
sha256:8ff563a3afecd974fef9daec009ae30bdc0e81014fa70a9df396cb17926fea1b
coverage-e2e-27585485819-b218d956-99ae-4239-ae42-c0b75f3ed788
699 KB
sha256:9ce467c6983292f5cf61faa7f2496b8f56b6da5ca49d6ca8cdca8e741b25f09f
coverage-e2e-27585485819-b37e3d2b-65c5-4019-9d74-e2aa0aef4561
233 KB
sha256:dfe3d0f944036da6925cbe475f0fb0bd72cd5aa3e261693d824b1db63d9ef846
coverage-e2e-27585485819-b9e6a2e1-3d19-402d-9cab-e7de6d7b7c73
314 KB
sha256:12d76d784d568fd8f693a35a41a14dae510f3161e2177a2d72cbdb67d0697dd2
coverage-e2e-27585485819-bc10e9fb-37b3-48b9-bae5-483a1762e560
1.29 KB
sha256:47d0b2a9124fa8349f24cb6021e24846e4dc82eff79730b16fe0bd0f093c8765
coverage-e2e-27585485819-be7ab67d-3c50-4032-a94f-c9c16dc2bac8
700 KB
sha256:e94a02ae3b7adf554ef52ff300ee5346eb5e3f5ee59aa892153e2cbfab05fb11
coverage-e2e-27585485819-c78c4f29-adf6-41b6-9e55-7be3f5071e4c
701 KB
sha256:5ea120b93aa27842388ea05a49344b8de801c0dc59a8afd2618a5cc1b8cb228e
coverage-e2e-27585485819-ce8ab767-573a-416a-b374-820d8a66d62b
253 KB
sha256:3cc2fce4b1f9efe9192ca2b11e1043acc13874b0872ca8b59fa78e229ddd08e4
coverage-e2e-27585485819-d0bff67b-55d3-43d8-be84-8bf361ff37aa
233 KB
sha256:8a1364dfca549bc9f86d546793497b30a2c1db86428c5760334e943c79d8f322
coverage-e2e-27585485819-d39f2d0f-81d1-49dc-a3dc-6c04851922d5
345 KB
sha256:7f5578edcdd5d2cfb1fe95c59cf3e423f0813ff0661b8dbd6d24a0f212a6f261
coverage-e2e-27585485819-d57950fb-cb85-4d72-930c-84ba277c03ba
270 KB
sha256:6fff04e24a6e36ecd217c1dddf4f20371557b08920cd94440153c598cd9ce546
coverage-e2e-27585485819-d6006be3-9a72-4ef1-a539-c12891592cb7
355 KB
sha256:ec05efdfcc3fbfc221b2d582f26a15c5069bfdd68e710c076a522083a78ae2d8
coverage-e2e-27585485819-df7af1d7-da82-4a4f-a3a4-58237ab2cb21
232 KB
sha256:9a22caa5064c4fea7bf4a79d5bda64c983c851f9636be54f820ceb0cb3309e6c
coverage-e2e-27585485819-dff4532c-d71c-49b2-8738-013c88ab3832
762 KB
sha256:d15b7bb8b0f7b983aa70fba248cd0933ff2c41ff0a12a558e5596e5aae9fe2d2
coverage-e2e-27585485819-e2767bbb-b2cb-4cbf-9c55-a0f044235b91
270 KB
sha256:52dbfc0516979fb702c5b55c2a665ab75acd5191c13585a1b4215d117a668bec
coverage-e2e-27585485819-e64fb0ae-7940-4e01-96f3-9239158f3d9c
759 KB
sha256:4c6c3432190dd23486e7acb59007e20403e83d6589949b8cadeed48cb18e0727
coverage-e2e-27585485819-ecbc6fb7-99d9-45a0-93f7-5835e3f8f431
270 KB
sha256:b71dd5a0aca123a54fe82cd37a451e578d98b37cf71ba0589189eca91fdfb7d4
coverage-e2e-27585485819-f37f6660-8d46-4798-9bbc-aa6bf9ec45a4
700 KB
sha256:7331dca2832b6afae2c48939510d88e2c4ecba5e4a55dbb688c4c5efe99a3c7a
coverage-e2e-27585485819-f96c1ffd-4f6f-4e54-aa92-3d4a7be7e4a9
758 KB
sha256:f552defea8de781b62c6643cee16b256fcab8679d8d0300b406b97fa047d0739
coverage-e2e-27585485819-f9930eaa-134e-4c7f-acc8-b249213e676a
1.29 KB
sha256:f16788905fd7a6731c801b488862af1a4c73be56ebf462e1530265c5dee7a201
coverage-unit-test-27585485819-d42421c0-763f-4de9-8899-098ce9049949
363 KB
sha256:c115f2add6e4626c186433ad0ca0359443e84648ceca353abe9d061d54077610
coverage-unit-test-27585485819-d877f53b-4126-42f0-a041-8f1aafa0ded8
310 KB
sha256:8026dbae0059b55f8e80ae2f4b02d29c80a61161bc12f1c3be6c7fda039a7547