Merge branch 'main' into main #16241
cicd-main.yml
on: push
pre-flight
/
pre-flight
17s
Matrix: cicd-container-build
Launch_Unit_Tests_Core
6m 36s
Launch_Unit_Tests_Diffusion
3m 25s
Matrix: cicd-functional-tests-l1
Matrix: cicd-functional-tests-l2
Matrix: cicd-functional-tests-l0
Matrix: cicd-functional-tests-flaky
Matrix: cicd-functional-tests-gb200-l1
Matrix: cicd-functional-tests-gb200-l2
Matrix: cicd-functional-tests-gb200-l0
Matrix: cicd-functional-tests-gb200-flaky
Coverage_Fake
0s
Matrix: Coverage
Deployment protection rules
Reviewers, timers, and other rules protecting deployments in this run
| Event | Environments |
|---|---|
|
svcnvidia-nemo-ci
approved
|
test |
Annotations
2 errors, 4 warnings, and 53 notices
|
cicd-container-build (aws, 766267172432.dkr.ecr.us-east-1.amazonaws.com, nemo-ci-aws-gpu-x2-ephem...
WARNING: The directory '/home/runner/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
|
|
cicd-container-build (aws, 766267172432.dkr.ecr.us-east-1.amazonaws.com, nemo-ci-aws-gpu-x2-ephem...
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
|
|
pre-flight / pre-flight
Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: actions/checkout@v4, step-security/changed-files@v45.0.1. Actions will be forced to run with Node.js 24 by default starting June 16th, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
cicd-container-build (aws, 766267172432.dkr.ecr.us-east-1.amazonaws.com, nemo-ci-aws-gpu-x2-ephem...
Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: docker/build-push-action@v5, docker/setup-buildx-action@v3. Actions will be forced to run with Node.js 24 by default starting June 16th, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
Coverage (unit-test)
Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea. Actions will be forced to run with Node.js 24 by default starting June 16th, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
Coverage (e2e)
Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea. Actions will be forced to run with Node.js 24 by default starting June 16th, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
Result
✅ Launch_Unit_Tests_Diffusion — PASSED
|
|
Result
✅ Launch_Unit_Tests_Core — PASSED
|
|
Result
✅ L1_Launch_models_mistral — PASSED
|
|
Result
✅ L1_Launch_models_glm — PASSED
|
|
Result
✅ L0_Launch_models_llada15 — PASSED
|
|
Result
✅ L0_Launch_models_deepseek — PASSED
|
|
Result
✅ L1_Launch_models_qwen25_vl — PASSED
|
|
Result
✅ L0_Launch_models_minimax_m2 — PASSED
|
|
Result
✅ L0_Launch_models_nemotron_omni — PASSED
|
|
Result
✅ L1_Launch_models_nemotron — PASSED
|
|
Result
✅ L0_Launch_megatron_fsdp — PASSED
|
|
Result
✅ L1_Launch_models_olmoe — PASSED
|
|
Result
✅ L0_Launch_models_glm_vl — PASSED
|
|
Result
✅ L0_Launch_models_gemma4_vl — PASSED
|
|
Result
✅ L0_Launch_converter_fsdp — PASSED
|
|
Result
✅ L0_Launch_models_bailing — PASSED
|
|
Result
✅ L0_Launch_models_kimi_k25_vl — PASSED
|
|
Result
✅ L0_Launch_models_qwen3 — PASSED
|
|
Result
✅ L0_Launch_converter_generate — PASSED
|
|
Result
✅ L1_Launch_models_qwen2 — PASSED
|
|
Result
✅ L1_Launch_models_stepfun — PASSED
|
|
Result
✅ L1_Launch_models_gemma1 — PASSED
|
|
Result
✅ L1_Launch_models_gemma2 — PASSED
|
|
Result
✅ L1_Launch_models_qwen35_vl — PASSED
|
|
Result
✅ L1_Launch_models_ernie_vl — PASSED
|
|
Result
✅ L1_Launch_models_qwen3_vl — PASSED
|
|
Result
✅ L1_Launch_models_ministral3 — PASSED
|
|
Result
✅ L1_Launch_models_gemma3 — PASSED
|
|
Result
✅ L1_Launch_recipes_gpt_oss — PASSED
|
|
Result
✅ L0_Launch_recipes_gemma4_vl — PASSED
|
|
Result
✅ L0_Launch_data — PASSED
|
|
Result
✅ L0_Launch_recipes_deepseek — PASSED
|
|
Result
✅ L0_Launch_models_flux — PASSED
|
|
Result
✅ L1_Launch_models_gpt_oss — PASSED
|
|
Result
✅ L1_Launch_recipes_qwen — PASSED
|
|
Result
✅ L0_Launch_training_megatron_mimo — PASSED
|
|
Result
✅ L0_Launch_models_nemotronh — PASSED
|
|
Result
✅ L1_Launch_recipes_ministral3 — PASSED
|
|
Result
✅ L0_Launch_models_wan — PASSED
|
|
Result
✅ L1_Launch_recipes_llama_3b — PASSED
|
|
Result
✅ L0_Launch_converter — PASSED
|
|
Result
✅ L0_Launch_models_qwen3_omni — PASSED
|
|
Result
✅ L0_Launch_recipes_gemma — PASSED
|
|
Result
✅ L0_Launch_recipes_nemotron_omni — PASSED
|
|
Result
✅ L0_Launch_recipes_llama_1b — PASSED
|
|
Result
✅ L0_Launch_utils — PASSED
|
|
Result
✅ L0_Launch_recipes_perf_config — PASSED
|
|
Result
✅ L0_Launch_recipes_glm_vl — PASSED
|
|
Result
✅ L0_Launch_recipes_nemotronh — PASSED
|
|
Result
✅ L0_Launch_wan_ckpt_conversion — PASSED
|
|
Result
✅ L0_Launch_vlm_inference — PASSED
|
|
Result
✅ L0_Launch_training — PASSED
|
|
Result
✅ L1_Launch_recipes_qwen_vl — PASSED
|
Artifacts
Produced during runtime
| Name | Size | Digest | |
|---|---|---|---|
|
coverage-e2e-27400345072-039eb6f7-8fec-4c32-9ab7-7a4b85268b74
|
234 KB |
sha256:c9471c63fda5c4150855a39c82ffee59ea293df680e708895de0f5b206b5c0eb
|
|
|
coverage-e2e-27400345072-0e959b29-542a-4ef7-b953-c3d6c83db394
|
702 KB |
sha256:6b13daad88203867ba545a2d85e2f2382c04b85b54bbf646318ab2b2f3835c86
|
|
|
coverage-e2e-27400345072-1270cab1-db69-423a-9d99-6715adfd4964
|
702 KB |
sha256:5d7241f45a9db9ccb62758def5a4276b8e2f1862ccd409092a311ace11d46412
|
|
|
coverage-e2e-27400345072-12fe318b-4d12-4220-8071-ca684c7ae04a
|
356 KB |
sha256:3707bf4839409ba86eb0ce7134323abb7702a6fab3fb1c36ed1e2f9c2fba2abc
|
|
|
coverage-e2e-27400345072-16dd3523-06d5-491e-b2b0-c6f7fd20ee38
|
1.29 KB |
sha256:d128bc8f5ba0c5b51c41ecda91909e2119dee54cffc5e0ddd9259d4b263a4f00
|
|
|
coverage-e2e-27400345072-19448d37-f34a-49dc-84a0-092131346b6b
|
357 KB |
sha256:321d322be6913cf2f0729636d34602a70894af33935e1aa21e8d5af1eb6a0713
|
|
|
coverage-e2e-27400345072-19fa167a-059b-4779-b018-114b11517e9d
|
701 KB |
sha256:3a25b85dba6b9c0c4ccf82b0b0d05489d91dc957cc3affb50f0b6f78b335ea8d
|
|
|
coverage-e2e-27400345072-2716d3cd-c3cb-4027-a486-810f7b050c21
|
361 KB |
sha256:245a99e09c2e822ffbed6fc2781966358591004ccd40b7fc893b1d4a0591aa51
|
|
|
coverage-e2e-27400345072-29fe1832-1b45-456f-a265-3d1ecde865c1
|
761 KB |
sha256:491fef53f4a4a3ee201eb639b7ced9831dbc1807fe67d45ac96528713ce8ca89
|
|
|
coverage-e2e-27400345072-2b0b3b09-2c28-49b6-a899-bd49ee7a5f5d
|
269 KB |
sha256:a53f614fede61cb99d7a9bc716d62a70a7bebd040f3899638de23e74f09eea5e
|
|
|
coverage-e2e-27400345072-2b680f85-b825-44c6-8422-14b3413216b8
|
307 KB |
sha256:09c6dc00aae9afb32ce58970dee82e010b791c4ddbc13f72c9168d46f3fcac80
|
|
|
coverage-e2e-27400345072-37d4785e-5b7c-4d08-9459-dd0b037006b4
|
234 KB |
sha256:9b1c5cee1939678df5b8ac5b614a034774170d20106b43fb14f25e37d3bb9fb5
|
|
|
coverage-e2e-27400345072-3bd5de23-5f6b-4bca-9f5e-2f6f209a9933
|
234 KB |
sha256:efa4b0bb00d70ffff6326ab4251c5ea054cbb412d0c3e267bd62f4123f25e765
|
|
|
coverage-e2e-27400345072-3be9f8c2-c813-419d-9224-e17d461dde54
|
346 KB |
sha256:3815e4c68df9b2db14ebc12f0608bba0f2b73e157bb229dfcd5a5ffdc371e66d
|
|
|
coverage-e2e-27400345072-4b536cc1-025b-41cc-bc61-aa39725bac82
|
235 KB |
sha256:28fdbccc40c6c2e80604a8082afcd0fa1fe1d75ee749398d8641cbfbf78746fa
|
|
|
coverage-e2e-27400345072-4d088988-6c79-454e-8b5b-ac84e0ed52da
|
759 KB |
sha256:f945296da6ccfc68e6821c489ee8f72a897551a2779d7e7f73e35fcaf57944bd
|
|
|
coverage-e2e-27400345072-4f6b75f7-78d8-470e-a6d2-bc6693b4802a
|
701 KB |
sha256:6304b84e5c9f6dd569ae50af1bc29f624b6c0e1b559fb7f4b67ad1ff38c85096
|
|
|
coverage-e2e-27400345072-5699268f-4b49-40d9-ba6a-b84f92d98a8a
|
701 KB |
sha256:e110e278d8d96036a9f5d745d967c0322fc98246e776e61461a137ca2ba5b47a
|
|
|
coverage-e2e-27400345072-5897d1d9-5c5e-401a-964f-c60dcb2c4d30
|
234 KB |
sha256:a09c1dc616cd9a62bba29480ddcec6bfad43d6eedddcbab7b0d799e27fc52461
|
|
|
coverage-e2e-27400345072-6155dc4b-c4cf-4e48-bf52-311fee6120e6
|
233 KB |
sha256:8b3ea69f6ecb9c98c44ae796134bc32b308cb0e512fec93792f8fe4819e51853
|
|
|
coverage-e2e-27400345072-62c5ebff-753e-4cfa-a099-8f7f1fe4d26c
|
236 KB |
sha256:70c9b5866f8e7e0326b52d71428d6f0686c2d98a33b868f486c71e198152edbb
|
|
|
coverage-e2e-27400345072-65aa70c3-7a6d-41fe-975d-95fd7d2874c7
|
702 KB |
sha256:1e10c5ad6a0aab36855c4af8a11aeda31483ea069fcc3e637dea8bd89201fc61
|
|
|
coverage-e2e-27400345072-6b47e224-a658-4ce4-813d-da772d91a6f0
|
720 KB |
sha256:b7da065e9b7ab7d0e8904f7a5dd7b02cbe97c82a9e7d4c48fa2c352fa05e43d2
|
|
|
coverage-e2e-27400345072-8379abda-a07c-4871-87dc-e0fdc6a3b8da
|
315 KB |
sha256:112ebdd5217992ec46d87a76a22aee9a0f3316900eca8c4773d8d2aca03d26b0
|
|
|
coverage-e2e-27400345072-8383e491-5acc-4989-b447-1cb1c35ef2c1
|
271 KB |
sha256:7e800460142d3348b269b9a5f926ba2fe2e4d59fc7b292ed162736152e09bc9c
|
|
|
coverage-e2e-27400345072-84e35ad7-f91e-44fc-a3b8-a51988d4abb3
|
700 KB |
sha256:8dfe2c56100f7bf5c51671cfc6584dce9ad87731718ef03d69350638a226019e
|
|
|
coverage-e2e-27400345072-89daa4c1-a306-45e3-9680-9df8302c72a3
|
349 KB |
sha256:20b3466e335cfbd78798524306f5a0d5fd6a4b3902497c11e56d1bdfbf39f91a
|
|
|
coverage-e2e-27400345072-8c66189d-2d5d-41ed-a5a1-d97a1784f093
|
701 KB |
sha256:0a6030a5eb445f6523f9333e168883b027bda31cdd5e2c3f5586d65840c1492f
|
|
|
coverage-e2e-27400345072-8e1d5469-84c4-4945-b78d-d63db3d930e8
|
314 KB |
sha256:972201b2453d21ae408d522946b316117990510d6981aa8d70c30b7b2fdcafb9
|
|
|
coverage-e2e-27400345072-8f2a0217-6571-4290-85db-c6d254c3db6e
|
231 KB |
sha256:2c565004db4f31d8650665dc36cb97381b21a7fb1e0e5ef9895223b7a2b5ee60
|
|
|
coverage-e2e-27400345072-8fb4eb8b-3113-4a68-b87e-0ecf9fa1dfbd
|
268 KB |
sha256:d25b2326f68be11e3d2938c26f15f51ed44232b8b515a4be5513558b38ef9e77
|
|
|
coverage-e2e-27400345072-94a2c2b8-90eb-4c3b-a16b-9ecd234b1964
|
757 KB |
sha256:527d26d582ae6270bead06969ddf914e8892e51056d09587ba8ae26233b6c397
|
|
|
coverage-e2e-27400345072-a3dedca0-e317-4a13-b05f-9907284d1853
|
254 KB |
sha256:2fcebdc872e2a59d311fe87e40050604bd6115c7ef920665300cdf9434434008
|
|
|
coverage-e2e-27400345072-bed94a82-4a28-457e-891e-e618d007b742
|
762 KB |
sha256:ca0e29ba52b5b90d2625753b41b444194ee21bd70e4de90c4de0d8bfad97a31b
|
|
|
coverage-e2e-27400345072-c02ca708-a66d-4d61-b40d-50c0e4b2175f
|
770 KB |
sha256:806d1862752e53583f213efdd4ffa14b9a40f3fc8d25fba792be7f7ced572600
|
|
|
coverage-e2e-27400345072-c0af6102-3fae-49db-86b8-8909c7c1afeb
|
759 KB |
sha256:1238406932c98c07dc37ffa196338a3dd86b0cc024c3596861de24d463759430
|
|
|
coverage-e2e-27400345072-c73a160d-97ca-4d87-96ea-ecf8d33cddc9
|
265 KB |
sha256:e4852a0bb9bd387291e55cf766a33b16df0176740b24de215863aefdb1662fd4
|
|
|
coverage-e2e-27400345072-c90ddfc4-53e0-4bcc-919e-cda96989cc86
|
701 KB |
sha256:d19a730774421476b2a2d634befa4995b7e13e3ceb7d447e291bbb1ed0699a41
|
|
|
coverage-e2e-27400345072-ca6c2d39-67c7-42f4-877e-60db954b5d8e
|
235 KB |
sha256:534d156786766cfc97103a75e395b3d83293398b02ce0cd7d6aa0eddf8dae10e
|
|
|
coverage-e2e-27400345072-cc5ab8b5-d108-405d-92a4-9c4217da7ae2
|
1.29 KB |
sha256:bd9e5e53e9274bdb48555d62ac473bbab582c4c10a01fc35484d38a90e537ad3
|
|
|
coverage-e2e-27400345072-d446f780-cad1-443e-af09-d0b92a4aa195
|
235 KB |
sha256:132ce9ba3955f9106d5b4c7df89d8a01eb54ef1d12adec9d97a7002ea83e7a45
|
|
|
coverage-e2e-27400345072-d53280ef-aabf-483e-9c04-59bed23b5907
|
347 KB |
sha256:7b2244e546325456b205e5176ad059593572809e3c040bb0914e6a97c6bc9dd4
|
|
|
coverage-e2e-27400345072-e0d978eb-6413-4bb1-8de7-3f69890116a9
|
701 KB |
sha256:ed2f60a243e3f3886079b9f58a4f3d31d322d0b15d5d5b61706ae7fd68356a19
|
|
|
coverage-e2e-27400345072-e1ddfc4d-ec57-45c7-9cfe-3ce5e61163c8
|
764 KB |
sha256:5e29e3f68d1ddf88335adeb2fede2671501a756c2f3cc57f787fe5e0b515ceef
|
|
|
coverage-e2e-27400345072-e3372a6e-7c6d-4273-9e86-5fefa6b09d92
|
701 KB |
sha256:140173a19f736b4a051dca175e706dc60589e386f68b52fcf3491f24945a953a
|
|
|
coverage-e2e-27400345072-e89466cb-d22e-4e69-98eb-4914ac893aca
|
762 KB |
sha256:dcfae657a604bf3340a4550d63706873ce3635c169a4e0c9d2059be83df36880
|
|
|
coverage-e2e-27400345072-e92d5505-1900-4a77-a2cb-cf23f6f85096
|
700 KB |
sha256:73494874cf24e55fe3cdb0dbaaea438572dcbdb6ed9ca31dce24c3605c16b8a4
|
|
|
coverage-e2e-27400345072-eb34a9ed-7d0b-4fde-b3bd-c7facb8b403a
|
295 KB |
sha256:b26e36e3b61b190bcbd19f86771e001951f18f03fdec523725bf7ede692e61a2
|
|
|
coverage-e2e-27400345072-f07c88a8-8309-49da-ae1e-89ff98613642
|
701 KB |
sha256:4d0e79050f801ff23e0bb077cc98df96a981ea023834f27ed6efa52d8e3de0c6
|
|
|
coverage-e2e-27400345072-fbffc3dd-9b2b-4ce6-8ca8-d575a57633e4
|
312 KB |
sha256:81f73e6f0ee07503ca81914ace93eb98bfb055ff4be33627a1d9b6dd5742cf1d
|
|
|
coverage-e2e-27400345072-fcfaa176-20f0-443b-8db9-f68ce27d45bc
|
1.29 KB |
sha256:8949d05b343d0fc8b5984127a8f628f6668211f9ee3edc7ad348e54c66dc3097
|
|
|
coverage-e2e-aggregated
|
143 KB |
sha256:386f3c11629d7be258babc09e8ba45dca23d389d04484f639ed1ac6bbb412809
|
|
|
coverage-unit-test-27400345072-3769cde2-e53d-443c-a094-1b42399fa8a1
|
364 KB |
sha256:2c398eff068142bf6d785e6da03a93dbc8c37bfebcd17c44c37cf8f731d4fd05
|
|
|
coverage-unit-test-27400345072-c4198f0e-a666-40f0-ba70-de2a9442ca03
|
311 KB |
sha256:23d7dbdd6e5b2d5b5f3a615ee999e1995a5db7e6a17749b4d6031d586ca9d8b7
|
|
|
coverage-unit-test-aggregated
|
69.4 KB |
sha256:3132d66c3b7691e63ec88c9e27f13d9a3cc295923f5ba8f95fbc414c7d0cae49
|
|