[test] test: strengthen model contract regressions #16394
cicd-main.yml
on: push
pre-flight
/
pre-flight
12s
Matrix: cicd-container-build
Launch_Unit_Tests_Core
6m 50s
Launch_Unit_Tests_Diffusion
2m 30s
Matrix: cicd-functional-tests-l1
Matrix: cicd-functional-tests-l2
Matrix: cicd-functional-tests-l0
Matrix: cicd-functional-tests-flaky
Matrix: cicd-functional-tests-gb200-l1
Matrix: cicd-functional-tests-gb200-l2
Matrix: cicd-functional-tests-gb200-l0
Matrix: cicd-functional-tests-gb200-flaky
Coverage_Fake
Matrix: Coverage
Deployment protection rules
Reviewers, timers, and other rules protecting deployments in this run
| Event | Environments |
|---|---|
|
svcnvidia-nemo-ci
approved
|
test |
Annotations
7 errors, 3 warnings, and 57 notices
|
cicd-container-build (aws, 766267172432.dkr.ecr.us-east-1.amazonaws.com, nemo-ci-aws-gpu-x2)
WARNING: The directory '/home/runner/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
|
|
cicd-container-build (aws, 766267172432.dkr.ecr.us-east-1.amazonaws.com, nemo-ci-aws-gpu-x2)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
|
|
cicd-container-build (gcp, us-east4-docker.pkg.dev/nv-projdgxchipp-20260113193621/megatron-bridge...
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
|
|
L0_Launch_training
Process completed with exit code 1.
|
|
Result
❌ L0_Launch_training — FAILED (exit 1)
|
|
L0_Launch_training
Process completed with exit code 1.
|
|
Nemo_CICD_Test
Process completed with exit code 1.
|
|
pre-flight / pre-flight
Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: actions/checkout@v4, step-security/changed-files@v45.0.1. Actions will be forced to run with Node.js 24 by default starting June 16th, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
cicd-container-build (aws, 766267172432.dkr.ecr.us-east-1.amazonaws.com, nemo-ci-aws-gpu-x2)
Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: docker/build-push-action@v5, docker/setup-buildx-action@v3. Actions will be forced to run with Node.js 24 by default starting June 16th, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
cicd-container-build (gcp, us-east4-docker.pkg.dev/nv-projdgxchipp-20260113193621/megatron-bridge...
Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: docker/build-push-action@v5, docker/setup-buildx-action@v3. Actions will be forced to run with Node.js 24 by default starting June 16th, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
Result
✅ Launch_Unit_Tests_Diffusion — PASSED
|
|
Result
✅ Launch_Unit_Tests_Core — PASSED
|
|
Result
✅ L0_Launch_models_nemotron_omni — PASSED
|
|
Result
✅ L0_Launch_models_nemotron_omni — PASSED
|
|
Result
✅ L0_Launch_models_qwen3_omni — PASSED
|
|
Result
✅ L0_Launch_models_kimi_k25_vl — PASSED
|
|
Result
✅ L0_Launch_models_wan — PASSED
|
|
Result
✅ L0_Launch_converter_fsdp — PASSED
|
|
Result
✅ L0_Launch_recipes_gemma — PASSED
|
|
Result
✅ L0_Launch_recipes_deepseek — PASSED
|
|
Result
✅ L0_Launch_models_bailing — PASSED
|
|
Result
✅ L0_Launch_recipes_gemma4_vl — PASSED
|
|
Result
✅ L0_Launch_recipes_deepseek_fsdp — PASSED
|
|
Result
✅ L0_Launch_models_minimax_m2 — PASSED
|
|
Result
✅ L0_Launch_data — PASSED
|
|
Result
✅ L0_Launch_models_gemma4_vl — PASSED
|
|
Result
✅ L0_Launch_models_deepseek — PASSED
|
|
Result
✅ L0_Launch_models_flux — PASSED
|
|
Result
✅ L0_Launch_models_glm_vl — PASSED
|
|
Result
✅ L0_Launch_megatron_fsdp — PASSED
|
|
Result
✅ L0_Launch_models_llada15 — PASSED
|
|
Result
✅ L0_Launch_models_deepseek — PASSED
|
|
Result
✅ L0_Launch_models_kimi_k25_vl — PASSED
|
|
Result
✅ L0_Launch_models_flux — PASSED
|
|
Result
✅ L0_Launch_models_minimax_m2 — PASSED
|
|
Result
✅ L0_Launch_models_bailing — PASSED
|
|
Result
✅ L0_Launch_models_qwen3_omni — PASSED
|
|
Result
✅ L0_Launch_models_nemotronh — PASSED
|
|
Result
✅ L0_Launch_models_qwen3 — PASSED
|
|
Result
✅ L0_Launch_models_glm_vl — PASSED
|
|
Result
✅ L0_Launch_converter_generate — PASSED
|
|
Result
✅ L0_Launch_recipes_glm_vl — PASSED
|
|
Result
✅ L0_Launch_data — PASSED
|
|
Result
✅ L0_Launch_models_qwen3 — PASSED
|
|
Result
✅ L0_Launch_recipes_nemotron_omni — PASSED
|
|
Result
✅ L0_Launch_converter — PASSED
|
|
Result
✅ L0_Launch_recipes_llama_1b — PASSED
|
|
Result
✅ L0_Launch_converter — PASSED
|
|
Result
✅ L0_Launch_models_nemotronh — PASSED
|
|
Result
✅ L0_Launch_recipes_perf_config — PASSED
|
|
Result
✅ L0_Launch_models_wan — PASSED
|
|
Result
✅ L0_Launch_training_megatron_mimo — PASSED
|
|
Result
✅ L0_Launch_utils — PASSED
|
|
Result
✅ L0_Launch_vlm_inference — PASSED
|
|
Result
✅ L0_Launch_wan_ckpt_conversion — PASSED
|
|
Result
✅ L0_Launch_recipes_deepseek — PASSED
|
|
Result
✅ L0_Launch_recipes_glm_vl — PASSED
|
|
Result
✅ L0_Launch_recipes_gemma — PASSED
|
|
Result
✅ L0_Launch_recipes_llama_1b — PASSED
|
|
Result
✅ L0_Launch_recipes_gemma4_vl — PASSED
|
|
Result
✅ L0_Launch_recipes_perf_config — PASSED
|
|
Result
✅ L0_Launch_recipes_nemotron_omni — PASSED
|
|
Result
✅ L0_Launch_training_megatron_mimo — PASSED
|
|
Result
✅ L0_Launch_utils — PASSED
|
|
Result
✅ L0_Launch_wan_ckpt_conversion — PASSED
|
|
Result
✅ L0_Launch_vlm_inference — PASSED
|
|
Result
✅ L0_Launch_recipes_nemotronh — PASSED
|
Artifacts
Produced during runtime
| Name | Size | Digest | |
|---|---|---|---|
|
coverage-e2e-27585485819-012df6c2-9a1d-4ed0-b7bc-0f4a2360764a
|
1.29 KB |
sha256:98f6cf842ac6cb58bbd1fce9db7e4091d367ae4e6c5691c18dd2fd4e53028ef3
|
|
|
coverage-e2e-27585485819-09e3141b-a931-47ca-b50b-93e0228cfafe
|
233 KB |
sha256:ebd2350cb848b4de18a9851bab2fab1d6edca4829782b14445a698193db509c5
|
|
|
coverage-e2e-27585485819-0c68ea9d-b698-44e4-bd6b-3c9474d4ddf1
|
719 KB |
sha256:af3f1ce46f5d536e9bc8c88bb8fa761d6d647da7a2c2e83e8965953a80388005
|
|
|
coverage-e2e-27585485819-108edace-d09c-406e-8cc1-7a24d7e83237
|
234 KB |
sha256:f324c9b509a1fb1610d13aedaae602573383a4368964445f7a5361cf4dd78d7f
|
|
|
coverage-e2e-27585485819-1446f306-3bdf-45b3-b2d2-4f7bdfadf99c
|
345 KB |
sha256:1bd1a3f358173c341d6c8baca3478397dec2364137a98158bfc9de05cfefb628
|
|
|
coverage-e2e-27585485819-17361dae-c814-42f8-8532-f50b3933fa22
|
305 KB |
sha256:249f11613f6962530af12e2ad1b453e5d5a5eb7fffa31d2cced299f8e1e6321a
|
|
|
coverage-e2e-27585485819-1d63eac9-c502-4e30-8a07-e23faf7abc5c
|
264 KB |
sha256:85db02058cdecf10c42557530bc04874155be8156b11405f6c3e602a13f23290
|
|
|
coverage-e2e-27585485819-263f8ac3-b1f5-47e9-b158-2b300928b128
|
700 KB |
sha256:88be9d865ac00fdea6f4a959c2e0bf6d4159c52724abf778e1aad250ac621190
|
|
|
coverage-e2e-27585485819-29986bb5-d3c1-40da-99a1-2d509da59462
|
311 KB |
sha256:27ccfc5020308b01a64a2dd906dbe5d80d9adcf530b63cf1f868fafd90c087e2
|
|
|
coverage-e2e-27585485819-2ca71765-0124-4d20-a9f4-b022b6a1804b
|
233 KB |
sha256:2ba393fd849b97b8470c357468fcb1bd9043b93a0031ec71730a15e020b0a666
|
|
|
coverage-e2e-27585485819-37210dbd-a3fb-4a2f-90a8-e2f2e0e525cf
|
314 KB |
sha256:e98f9072b954e4c93198bfb2af57d9a90a21372cec4ca807d680a2b0192766db
|
|
|
coverage-e2e-27585485819-3d03055c-6661-4842-b815-964e7bd19ad3
|
759 KB |
sha256:85f3bbff20b632aa8ca5967884609305a3d3a64ac09fcd3b5b4d9aa79aa18a9e
|
|
|
coverage-e2e-27585485819-3f930c60-f6e1-474f-9cd6-a0e4ef232bec
|
233 KB |
sha256:52880c7de0f5b3fe8ecbbb9673c931d04c2b82d6febe9944e3520d9868ce759f
|
|
|
coverage-e2e-27585485819-40567597-a969-4d82-b176-ccafa91381b4
|
340 KB |
sha256:6f2db5deb7e092f0fcfa349f5fef869e2e32c951c7432b7a8cb56eee3f2c0d66
|
|
|
coverage-e2e-27585485819-490cddca-736c-4156-82d1-893404deceab
|
348 KB |
sha256:2319c536696b253c42fc0f2076319e4092190f25e79844b9552bb6ef3631df6d
|
|
|
coverage-e2e-27585485819-55c0b0fd-8b15-49c4-82e3-0991321f726b
|
348 KB |
sha256:642ffd853f25cd69136b22d05fdee935ee93e365c62ef9f7a38f0fde66604e94
|
|
|
coverage-e2e-27585485819-5697d034-e847-4293-8bea-898601e3c83d
|
311 KB |
sha256:52499a29198f5543de678d220c75d6c8e0f469dced43b053d973b6073b56d84c
|
|
|
coverage-e2e-27585485819-56d594f4-8171-4dec-aa29-8796620b4b65
|
760 KB |
sha256:12693816f0e6f853c58ab3058405bcbc881484321eb084b6ef4677d34c0ccf39
|
|
|
coverage-e2e-27585485819-61a993a0-fe2e-4472-94d5-f056d6f0fa4e
|
253 KB |
sha256:1e3c7cf2536cb77cb0aa5120feaa914906422b94206b5ccd3e5e3b7d15ffdbbb
|
|
|
coverage-e2e-27585485819-667da861-d951-4d91-a2df-96cd85f05229
|
699 KB |
sha256:f00b68d2c12a21b71ab9818a129e928b50437fdc032c07a19f664d606da6e0f4
|
|
|
coverage-e2e-27585485819-6b522216-4051-4ae5-a46e-b8da2bb3a8cd
|
313 KB |
sha256:c61d6024b7aebc01ec1bea56d2ca03db1a7bb7355dbfc892ac00a46b4677becd
|
|
|
coverage-e2e-27585485819-836e1250-424c-4da4-a4c5-10eb5375fd74
|
699 KB |
sha256:15a78befc1600cc93639f2288979f9ab62c9ffb7376bae79b59a09e2472a9e2b
|
|
|
coverage-e2e-27585485819-8f54c58c-5c82-4faa-8d29-4128471a87cb
|
700 KB |
sha256:19b16001485ce3ac3b461b3caffa032a57f347836456a7b019ed0f7be163679d
|
|
|
coverage-e2e-27585485819-8f845aa8-66b6-4e9e-999c-3e7d96a11a0e
|
760 KB |
sha256:cba5109d395a30c4acf5e75d56d8d4603aeef0125e78e1c164b35b7c39039614
|
|
|
coverage-e2e-27585485819-9162cc3c-8978-4f4b-8b39-6989b18d1b08
|
235 KB |
sha256:1ad0d551ec6b947dcb2e0fa42b7f251c6e591bdec88d95fc084e2734fd044760
|
|
|
coverage-e2e-27585485819-933f4401-861c-4877-a711-13043e949333
|
346 KB |
sha256:0e67ca51125bcd7bf7b5c31ff2cd3186e478e8cb9ee1624cc650c6e58aec51bb
|
|
|
coverage-e2e-27585485819-9857159b-145a-4232-801f-3a05d6b88baf
|
294 KB |
sha256:21d004bbc9ee85a25de927702d44218aed6538701f3a168073c38bc700db5bab
|
|
|
coverage-e2e-27585485819-9870188c-0aac-4e28-afea-8b0c163279fe
|
305 KB |
sha256:3d81fe98aef3cf287d75516d57c555634e02b2259ee9e3b42c5d512875d98700
|
|
|
coverage-e2e-27585485819-9bb24e78-deac-41ce-ad4a-09a4231e8670
|
267 KB |
sha256:161a4ea00b4f1adcb58d0f2a0a60f8ac5e6deddfd323c10bcf08d6b694a07d80
|
|
|
coverage-e2e-27585485819-9e01d149-a2c9-42eb-8f15-aa28adeeaab2
|
313 KB |
sha256:17655fa507b96edfdbafbe1c117c56e53ad6e23bf860b022b174848f27847a6d
|
|
|
coverage-e2e-27585485819-a281a085-d1e1-40e4-82f0-b36518439c42
|
264 KB |
sha256:331fd35874e4e28834046d3d567972e56062daff0a5ab701bc22b1bcd705b994
|
|
|
coverage-e2e-27585485819-a3da3fae-d646-4065-a61f-39a89c626dc2
|
355 KB |
sha256:150580db41f511bb05acc469a5856c128286ecbe09fb7763b7dc37d5347c8410
|
|
|
coverage-e2e-27585485819-a43cb292-0619-492a-a775-e369cda8f9c1
|
758 KB |
sha256:4785d2889876fc12378f9a5c3c3a667c2a9527aa289b47bef64cd3030a2e2267
|
|
|
coverage-e2e-27585485819-a878b3d5-19d0-4b06-9e01-9db56965632b
|
346 KB |
sha256:0ada6824fc4377b35b2546632965c50b8a9b0c253fcfcffab6ee72b678684f15
|
|
|
coverage-e2e-27585485819-ad499aa5-6f7f-414f-9863-55b9646fc87d
|
719 KB |
sha256:4c293cb0d265517d63bbcb5927a69ef6b72f7707021fa4bf22a21e0ebe7efcd1
|
|
|
coverage-e2e-27585485819-b15820c7-7001-4d5e-8dc0-83f98961f616
|
233 KB |
sha256:8ff563a3afecd974fef9daec009ae30bdc0e81014fa70a9df396cb17926fea1b
|
|
|
coverage-e2e-27585485819-b218d956-99ae-4239-ae42-c0b75f3ed788
|
699 KB |
sha256:9ce467c6983292f5cf61faa7f2496b8f56b6da5ca49d6ca8cdca8e741b25f09f
|
|
|
coverage-e2e-27585485819-b37e3d2b-65c5-4019-9d74-e2aa0aef4561
|
233 KB |
sha256:dfe3d0f944036da6925cbe475f0fb0bd72cd5aa3e261693d824b1db63d9ef846
|
|
|
coverage-e2e-27585485819-b9e6a2e1-3d19-402d-9cab-e7de6d7b7c73
|
314 KB |
sha256:12d76d784d568fd8f693a35a41a14dae510f3161e2177a2d72cbdb67d0697dd2
|
|
|
coverage-e2e-27585485819-bc10e9fb-37b3-48b9-bae5-483a1762e560
|
1.29 KB |
sha256:47d0b2a9124fa8349f24cb6021e24846e4dc82eff79730b16fe0bd0f093c8765
|
|
|
coverage-e2e-27585485819-be7ab67d-3c50-4032-a94f-c9c16dc2bac8
|
700 KB |
sha256:e94a02ae3b7adf554ef52ff300ee5346eb5e3f5ee59aa892153e2cbfab05fb11
|
|
|
coverage-e2e-27585485819-c78c4f29-adf6-41b6-9e55-7be3f5071e4c
|
701 KB |
sha256:5ea120b93aa27842388ea05a49344b8de801c0dc59a8afd2618a5cc1b8cb228e
|
|
|
coverage-e2e-27585485819-ce8ab767-573a-416a-b374-820d8a66d62b
|
253 KB |
sha256:3cc2fce4b1f9efe9192ca2b11e1043acc13874b0872ca8b59fa78e229ddd08e4
|
|
|
coverage-e2e-27585485819-d0bff67b-55d3-43d8-be84-8bf361ff37aa
|
233 KB |
sha256:8a1364dfca549bc9f86d546793497b30a2c1db86428c5760334e943c79d8f322
|
|
|
coverage-e2e-27585485819-d39f2d0f-81d1-49dc-a3dc-6c04851922d5
|
345 KB |
sha256:7f5578edcdd5d2cfb1fe95c59cf3e423f0813ff0661b8dbd6d24a0f212a6f261
|
|
|
coverage-e2e-27585485819-d57950fb-cb85-4d72-930c-84ba277c03ba
|
270 KB |
sha256:6fff04e24a6e36ecd217c1dddf4f20371557b08920cd94440153c598cd9ce546
|
|
|
coverage-e2e-27585485819-d6006be3-9a72-4ef1-a539-c12891592cb7
|
355 KB |
sha256:ec05efdfcc3fbfc221b2d582f26a15c5069bfdd68e710c076a522083a78ae2d8
|
|
|
coverage-e2e-27585485819-df7af1d7-da82-4a4f-a3a4-58237ab2cb21
|
232 KB |
sha256:9a22caa5064c4fea7bf4a79d5bda64c983c851f9636be54f820ceb0cb3309e6c
|
|
|
coverage-e2e-27585485819-dff4532c-d71c-49b2-8738-013c88ab3832
|
762 KB |
sha256:d15b7bb8b0f7b983aa70fba248cd0933ff2c41ff0a12a558e5596e5aae9fe2d2
|
|
|
coverage-e2e-27585485819-e2767bbb-b2cb-4cbf-9c55-a0f044235b91
|
270 KB |
sha256:52dbfc0516979fb702c5b55c2a665ab75acd5191c13585a1b4215d117a668bec
|
|
|
coverage-e2e-27585485819-e64fb0ae-7940-4e01-96f3-9239158f3d9c
|
759 KB |
sha256:4c6c3432190dd23486e7acb59007e20403e83d6589949b8cadeed48cb18e0727
|
|
|
coverage-e2e-27585485819-ecbc6fb7-99d9-45a0-93f7-5835e3f8f431
|
270 KB |
sha256:b71dd5a0aca123a54fe82cd37a451e578d98b37cf71ba0589189eca91fdfb7d4
|
|
|
coverage-e2e-27585485819-f37f6660-8d46-4798-9bbc-aa6bf9ec45a4
|
700 KB |
sha256:7331dca2832b6afae2c48939510d88e2c4ecba5e4a55dbb688c4c5efe99a3c7a
|
|
|
coverage-e2e-27585485819-f96c1ffd-4f6f-4e54-aa92-3d4a7be7e4a9
|
758 KB |
sha256:f552defea8de781b62c6643cee16b256fcab8679d8d0300b406b97fa047d0739
|
|
|
coverage-e2e-27585485819-f9930eaa-134e-4c7f-acc8-b249213e676a
|
1.29 KB |
sha256:f16788905fd7a6731c801b488862af1a4c73be56ebf462e1530265c5dee7a201
|
|
|
coverage-unit-test-27585485819-d42421c0-763f-4de9-8899-098ce9049949
|
363 KB |
sha256:c115f2add6e4626c186433ad0ca0359443e84648ceca353abe9d061d54077610
|
|
|
coverage-unit-test-27585485819-d877f53b-4126-42f0-a041-8f1aafa0ded8
|
310 KB |
sha256:8026dbae0059b55f8e80ae2f4b02d29c80a61161bc12f1c3be6c7fda039a7547
|
|