Skip to content

Merge branch 'main' into main #16241

Merge branch 'main' into main

Merge branch 'main' into main #16241

Triggered via push June 12, 2026 07:04
Status Success
Total duration 11h 32m 23s
Artifacts 55

cicd-main.yml

on: push
pre-flight  /  pre-flight
17s
pre-flight / pre-flight
configure
6s
configure
Lint check
24s
Lint check
cicd-wait-in-queue
3s
cicd-wait-in-queue
cicd-compute-build-matrix
3s
cicd-compute-build-matrix
Matrix: cicd-container-build
Launch_Unit_Tests_Core
6m 36s
Launch_Unit_Tests_Core
Launch_Unit_Tests_Diffusion
3m 25s
Launch_Unit_Tests_Diffusion
generate-test-matrix
6s
generate-test-matrix
generate-gb200-test-matrix
0s
generate-gb200-test-matrix
Launch_Import_Check
1m 37s
Launch_Import_Check
Matrix: cicd-functional-tests-l1
Matrix: cicd-functional-tests-l2
Matrix: cicd-functional-tests-l0
Matrix: cicd-functional-tests-flaky
Matrix: cicd-functional-tests-gb200-l1
Matrix: cicd-functional-tests-gb200-l2
Matrix: cicd-functional-tests-gb200-l0
Matrix: cicd-functional-tests-gb200-flaky
Nemo_CICD_Test
10s
Nemo_CICD_Test
Coverage_Fake
0s
Coverage_Fake
Matrix: Coverage
Fit to window
Zoom out
Zoom in

Deployment protection rules

Reviewers, timers, and other rules protecting deployments in this run
Event Environments Comment
svcnvidia-nemo-ci
approved Jun 12, 2026
test

Automatically approved by queue manager

Annotations

2 errors, 4 warnings, and 53 notices
cicd-container-build (aws, 766267172432.dkr.ecr.us-east-1.amazonaws.com, nemo-ci-aws-gpu-x2-ephem...
WARNING: The directory '/home/runner/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
cicd-container-build (aws, 766267172432.dkr.ecr.us-east-1.amazonaws.com, nemo-ci-aws-gpu-x2-ephem...
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
pre-flight / pre-flight
Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: actions/checkout@v4, step-security/changed-files@v45.0.1. Actions will be forced to run with Node.js 24 by default starting June 16th, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
cicd-container-build (aws, 766267172432.dkr.ecr.us-east-1.amazonaws.com, nemo-ci-aws-gpu-x2-ephem...
Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: docker/build-push-action@v5, docker/setup-buildx-action@v3. Actions will be forced to run with Node.js 24 by default starting June 16th, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
Coverage (unit-test)
Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea. Actions will be forced to run with Node.js 24 by default starting June 16th, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
Coverage (e2e)
Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea. Actions will be forced to run with Node.js 24 by default starting June 16th, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
Result
✅ Launch_Unit_Tests_Diffusion — PASSED
Result
✅ Launch_Unit_Tests_Core — PASSED
Result
✅ L1_Launch_models_mistral — PASSED
Result
✅ L1_Launch_models_glm — PASSED
Result
✅ L0_Launch_models_llada15 — PASSED
Result
✅ L0_Launch_models_deepseek — PASSED
Result
✅ L1_Launch_models_qwen25_vl — PASSED
Result
✅ L0_Launch_models_minimax_m2 — PASSED
Result
✅ L0_Launch_models_nemotron_omni — PASSED
Result
✅ L1_Launch_models_nemotron — PASSED
Result
✅ L0_Launch_megatron_fsdp — PASSED
Result
✅ L1_Launch_models_olmoe — PASSED
Result
✅ L0_Launch_models_glm_vl — PASSED
Result
✅ L0_Launch_models_gemma4_vl — PASSED
Result
✅ L0_Launch_converter_fsdp — PASSED
Result
✅ L0_Launch_models_bailing — PASSED
Result
✅ L0_Launch_models_kimi_k25_vl — PASSED
Result
✅ L0_Launch_models_qwen3 — PASSED
Result
✅ L0_Launch_converter_generate — PASSED
Result
✅ L1_Launch_models_qwen2 — PASSED
Result
✅ L1_Launch_models_stepfun — PASSED
Result
✅ L1_Launch_models_gemma1 — PASSED
Result
✅ L1_Launch_models_gemma2 — PASSED
Result
✅ L1_Launch_models_qwen35_vl — PASSED
Result
✅ L1_Launch_models_ernie_vl — PASSED
Result
✅ L1_Launch_models_qwen3_vl — PASSED
Result
✅ L1_Launch_models_ministral3 — PASSED
Result
✅ L1_Launch_models_gemma3 — PASSED
Result
✅ L1_Launch_recipes_gpt_oss — PASSED
Result
✅ L0_Launch_recipes_gemma4_vl — PASSED
Result
✅ L0_Launch_data — PASSED
Result
✅ L0_Launch_recipes_deepseek — PASSED
Result
✅ L0_Launch_models_flux — PASSED
Result
✅ L1_Launch_models_gpt_oss — PASSED
Result
✅ L1_Launch_recipes_qwen — PASSED
Result
✅ L0_Launch_training_megatron_mimo — PASSED
Result
✅ L0_Launch_models_nemotronh — PASSED
Result
✅ L1_Launch_recipes_ministral3 — PASSED
Result
✅ L0_Launch_models_wan — PASSED
Result
✅ L1_Launch_recipes_llama_3b — PASSED
Result
✅ L0_Launch_converter — PASSED
Result
✅ L0_Launch_models_qwen3_omni — PASSED
Result
✅ L0_Launch_recipes_gemma — PASSED
Result
✅ L0_Launch_recipes_nemotron_omni — PASSED
Result
✅ L0_Launch_recipes_llama_1b — PASSED
Result
✅ L0_Launch_utils — PASSED
Result
✅ L0_Launch_recipes_perf_config — PASSED
Result
✅ L0_Launch_recipes_glm_vl — PASSED
Result
✅ L0_Launch_recipes_nemotronh — PASSED
Result
✅ L0_Launch_wan_ckpt_conversion — PASSED
Result
✅ L0_Launch_vlm_inference — PASSED
Result
✅ L0_Launch_training — PASSED
Result
✅ L1_Launch_recipes_qwen_vl — PASSED

Artifacts

Produced during runtime
Name Size Digest
coverage-e2e-27400345072-039eb6f7-8fec-4c32-9ab7-7a4b85268b74
234 KB
sha256:c9471c63fda5c4150855a39c82ffee59ea293df680e708895de0f5b206b5c0eb
coverage-e2e-27400345072-0e959b29-542a-4ef7-b953-c3d6c83db394
702 KB
sha256:6b13daad88203867ba545a2d85e2f2382c04b85b54bbf646318ab2b2f3835c86
coverage-e2e-27400345072-1270cab1-db69-423a-9d99-6715adfd4964
702 KB
sha256:5d7241f45a9db9ccb62758def5a4276b8e2f1862ccd409092a311ace11d46412
coverage-e2e-27400345072-12fe318b-4d12-4220-8071-ca684c7ae04a
356 KB
sha256:3707bf4839409ba86eb0ce7134323abb7702a6fab3fb1c36ed1e2f9c2fba2abc
coverage-e2e-27400345072-16dd3523-06d5-491e-b2b0-c6f7fd20ee38
1.29 KB
sha256:d128bc8f5ba0c5b51c41ecda91909e2119dee54cffc5e0ddd9259d4b263a4f00
coverage-e2e-27400345072-19448d37-f34a-49dc-84a0-092131346b6b
357 KB
sha256:321d322be6913cf2f0729636d34602a70894af33935e1aa21e8d5af1eb6a0713
coverage-e2e-27400345072-19fa167a-059b-4779-b018-114b11517e9d
701 KB
sha256:3a25b85dba6b9c0c4ccf82b0b0d05489d91dc957cc3affb50f0b6f78b335ea8d
coverage-e2e-27400345072-2716d3cd-c3cb-4027-a486-810f7b050c21
361 KB
sha256:245a99e09c2e822ffbed6fc2781966358591004ccd40b7fc893b1d4a0591aa51
coverage-e2e-27400345072-29fe1832-1b45-456f-a265-3d1ecde865c1
761 KB
sha256:491fef53f4a4a3ee201eb639b7ced9831dbc1807fe67d45ac96528713ce8ca89
coverage-e2e-27400345072-2b0b3b09-2c28-49b6-a899-bd49ee7a5f5d
269 KB
sha256:a53f614fede61cb99d7a9bc716d62a70a7bebd040f3899638de23e74f09eea5e
coverage-e2e-27400345072-2b680f85-b825-44c6-8422-14b3413216b8
307 KB
sha256:09c6dc00aae9afb32ce58970dee82e010b791c4ddbc13f72c9168d46f3fcac80
coverage-e2e-27400345072-37d4785e-5b7c-4d08-9459-dd0b037006b4
234 KB
sha256:9b1c5cee1939678df5b8ac5b614a034774170d20106b43fb14f25e37d3bb9fb5
coverage-e2e-27400345072-3bd5de23-5f6b-4bca-9f5e-2f6f209a9933
234 KB
sha256:efa4b0bb00d70ffff6326ab4251c5ea054cbb412d0c3e267bd62f4123f25e765
coverage-e2e-27400345072-3be9f8c2-c813-419d-9224-e17d461dde54
346 KB
sha256:3815e4c68df9b2db14ebc12f0608bba0f2b73e157bb229dfcd5a5ffdc371e66d
coverage-e2e-27400345072-4b536cc1-025b-41cc-bc61-aa39725bac82
235 KB
sha256:28fdbccc40c6c2e80604a8082afcd0fa1fe1d75ee749398d8641cbfbf78746fa
coverage-e2e-27400345072-4d088988-6c79-454e-8b5b-ac84e0ed52da
759 KB
sha256:f945296da6ccfc68e6821c489ee8f72a897551a2779d7e7f73e35fcaf57944bd
coverage-e2e-27400345072-4f6b75f7-78d8-470e-a6d2-bc6693b4802a
701 KB
sha256:6304b84e5c9f6dd569ae50af1bc29f624b6c0e1b559fb7f4b67ad1ff38c85096
coverage-e2e-27400345072-5699268f-4b49-40d9-ba6a-b84f92d98a8a
701 KB
sha256:e110e278d8d96036a9f5d745d967c0322fc98246e776e61461a137ca2ba5b47a
coverage-e2e-27400345072-5897d1d9-5c5e-401a-964f-c60dcb2c4d30
234 KB
sha256:a09c1dc616cd9a62bba29480ddcec6bfad43d6eedddcbab7b0d799e27fc52461
coverage-e2e-27400345072-6155dc4b-c4cf-4e48-bf52-311fee6120e6
233 KB
sha256:8b3ea69f6ecb9c98c44ae796134bc32b308cb0e512fec93792f8fe4819e51853
coverage-e2e-27400345072-62c5ebff-753e-4cfa-a099-8f7f1fe4d26c
236 KB
sha256:70c9b5866f8e7e0326b52d71428d6f0686c2d98a33b868f486c71e198152edbb
coverage-e2e-27400345072-65aa70c3-7a6d-41fe-975d-95fd7d2874c7
702 KB
sha256:1e10c5ad6a0aab36855c4af8a11aeda31483ea069fcc3e637dea8bd89201fc61
coverage-e2e-27400345072-6b47e224-a658-4ce4-813d-da772d91a6f0
720 KB
sha256:b7da065e9b7ab7d0e8904f7a5dd7b02cbe97c82a9e7d4c48fa2c352fa05e43d2
coverage-e2e-27400345072-8379abda-a07c-4871-87dc-e0fdc6a3b8da
315 KB
sha256:112ebdd5217992ec46d87a76a22aee9a0f3316900eca8c4773d8d2aca03d26b0
coverage-e2e-27400345072-8383e491-5acc-4989-b447-1cb1c35ef2c1
271 KB
sha256:7e800460142d3348b269b9a5f926ba2fe2e4d59fc7b292ed162736152e09bc9c
coverage-e2e-27400345072-84e35ad7-f91e-44fc-a3b8-a51988d4abb3
700 KB
sha256:8dfe2c56100f7bf5c51671cfc6584dce9ad87731718ef03d69350638a226019e
coverage-e2e-27400345072-89daa4c1-a306-45e3-9680-9df8302c72a3
349 KB
sha256:20b3466e335cfbd78798524306f5a0d5fd6a4b3902497c11e56d1bdfbf39f91a
coverage-e2e-27400345072-8c66189d-2d5d-41ed-a5a1-d97a1784f093
701 KB
sha256:0a6030a5eb445f6523f9333e168883b027bda31cdd5e2c3f5586d65840c1492f
coverage-e2e-27400345072-8e1d5469-84c4-4945-b78d-d63db3d930e8
314 KB
sha256:972201b2453d21ae408d522946b316117990510d6981aa8d70c30b7b2fdcafb9
coverage-e2e-27400345072-8f2a0217-6571-4290-85db-c6d254c3db6e
231 KB
sha256:2c565004db4f31d8650665dc36cb97381b21a7fb1e0e5ef9895223b7a2b5ee60
coverage-e2e-27400345072-8fb4eb8b-3113-4a68-b87e-0ecf9fa1dfbd
268 KB
sha256:d25b2326f68be11e3d2938c26f15f51ed44232b8b515a4be5513558b38ef9e77
coverage-e2e-27400345072-94a2c2b8-90eb-4c3b-a16b-9ecd234b1964
757 KB
sha256:527d26d582ae6270bead06969ddf914e8892e51056d09587ba8ae26233b6c397
coverage-e2e-27400345072-a3dedca0-e317-4a13-b05f-9907284d1853
254 KB
sha256:2fcebdc872e2a59d311fe87e40050604bd6115c7ef920665300cdf9434434008
coverage-e2e-27400345072-bed94a82-4a28-457e-891e-e618d007b742
762 KB
sha256:ca0e29ba52b5b90d2625753b41b444194ee21bd70e4de90c4de0d8bfad97a31b
coverage-e2e-27400345072-c02ca708-a66d-4d61-b40d-50c0e4b2175f
770 KB
sha256:806d1862752e53583f213efdd4ffa14b9a40f3fc8d25fba792be7f7ced572600
coverage-e2e-27400345072-c0af6102-3fae-49db-86b8-8909c7c1afeb
759 KB
sha256:1238406932c98c07dc37ffa196338a3dd86b0cc024c3596861de24d463759430
coverage-e2e-27400345072-c73a160d-97ca-4d87-96ea-ecf8d33cddc9
265 KB
sha256:e4852a0bb9bd387291e55cf766a33b16df0176740b24de215863aefdb1662fd4
coverage-e2e-27400345072-c90ddfc4-53e0-4bcc-919e-cda96989cc86
701 KB
sha256:d19a730774421476b2a2d634befa4995b7e13e3ceb7d447e291bbb1ed0699a41
coverage-e2e-27400345072-ca6c2d39-67c7-42f4-877e-60db954b5d8e
235 KB
sha256:534d156786766cfc97103a75e395b3d83293398b02ce0cd7d6aa0eddf8dae10e
coverage-e2e-27400345072-cc5ab8b5-d108-405d-92a4-9c4217da7ae2
1.29 KB
sha256:bd9e5e53e9274bdb48555d62ac473bbab582c4c10a01fc35484d38a90e537ad3
coverage-e2e-27400345072-d446f780-cad1-443e-af09-d0b92a4aa195
235 KB
sha256:132ce9ba3955f9106d5b4c7df89d8a01eb54ef1d12adec9d97a7002ea83e7a45
coverage-e2e-27400345072-d53280ef-aabf-483e-9c04-59bed23b5907
347 KB
sha256:7b2244e546325456b205e5176ad059593572809e3c040bb0914e6a97c6bc9dd4
coverage-e2e-27400345072-e0d978eb-6413-4bb1-8de7-3f69890116a9
701 KB
sha256:ed2f60a243e3f3886079b9f58a4f3d31d322d0b15d5d5b61706ae7fd68356a19
coverage-e2e-27400345072-e1ddfc4d-ec57-45c7-9cfe-3ce5e61163c8
764 KB
sha256:5e29e3f68d1ddf88335adeb2fede2671501a756c2f3cc57f787fe5e0b515ceef
coverage-e2e-27400345072-e3372a6e-7c6d-4273-9e86-5fefa6b09d92
701 KB
sha256:140173a19f736b4a051dca175e706dc60589e386f68b52fcf3491f24945a953a
coverage-e2e-27400345072-e89466cb-d22e-4e69-98eb-4914ac893aca
762 KB
sha256:dcfae657a604bf3340a4550d63706873ce3635c169a4e0c9d2059be83df36880
coverage-e2e-27400345072-e92d5505-1900-4a77-a2cb-cf23f6f85096
700 KB
sha256:73494874cf24e55fe3cdb0dbaaea438572dcbdb6ed9ca31dce24c3605c16b8a4
coverage-e2e-27400345072-eb34a9ed-7d0b-4fde-b3bd-c7facb8b403a
295 KB
sha256:b26e36e3b61b190bcbd19f86771e001951f18f03fdec523725bf7ede692e61a2
coverage-e2e-27400345072-f07c88a8-8309-49da-ae1e-89ff98613642
701 KB
sha256:4d0e79050f801ff23e0bb077cc98df96a981ea023834f27ed6efa52d8e3de0c6
coverage-e2e-27400345072-fbffc3dd-9b2b-4ce6-8ca8-d575a57633e4
312 KB
sha256:81f73e6f0ee07503ca81914ace93eb98bfb055ff4be33627a1d9b6dd5742cf1d
coverage-e2e-27400345072-fcfaa176-20f0-443b-8db9-f68ce27d45bc
1.29 KB
sha256:8949d05b343d0fc8b5984127a8f628f6668211f9ee3edc7ad348e54c66dc3097
coverage-e2e-aggregated
143 KB
sha256:386f3c11629d7be258babc09e8ba45dca23d389d04484f639ed1ac6bbb412809
coverage-unit-test-27400345072-3769cde2-e53d-443c-a094-1b42399fa8a1
364 KB
sha256:2c398eff068142bf6d785e6da03a93dbc8c37bfebcd17c44c37cf8f731d4fd05
coverage-unit-test-27400345072-c4198f0e-a666-40f0-ba70-de2a9442ca03
311 KB
sha256:23d7dbdd6e5b2d5b5f3a615ee999e1995a5db7e6a17749b4d6031d586ca9d8b7
coverage-unit-test-aggregated
69.4 KB
sha256:3132d66c3b7691e63ec88c9e27f13d9a3cc295923f5ba8f95fbc414c7d0cae49