Skip to content

test(deepseek): stabilize Blackwell default recipe tests #16277

test(deepseek): stabilize Blackwell default recipe tests

test(deepseek): stabilize Blackwell default recipe tests #16277

Triggered via push June 12, 2026 21:32
Status Success
Total duration 1h 33m 59s
Artifacts 60

cicd-main.yml

on: push
pre-flight  /  pre-flight
12s
pre-flight / pre-flight
configure
7s
configure
Lint check
17s
Lint check
cicd-wait-in-queue
3s
cicd-wait-in-queue
cicd-compute-build-matrix
3s
cicd-compute-build-matrix
Matrix: cicd-container-build
Launch_Unit_Tests_Core
6m 25s
Launch_Unit_Tests_Core
Launch_Unit_Tests_Diffusion
2m 56s
Launch_Unit_Tests_Diffusion
generate-test-matrix
6s
generate-test-matrix
generate-gb200-test-matrix
6s
generate-gb200-test-matrix
Launch_Import_Check
1m 25s
Launch_Import_Check
Matrix: cicd-functional-tests-l1
Matrix: cicd-functional-tests-l2
Matrix: cicd-functional-tests-l0
Matrix: cicd-functional-tests-flaky
Matrix: cicd-functional-tests-gb200-l1
Matrix: cicd-functional-tests-gb200-l2
Matrix: cicd-functional-tests-gb200-l0
Matrix: cicd-functional-tests-gb200-flaky
Nemo_CICD_Test
7s
Nemo_CICD_Test
Coverage_Fake
0s
Coverage_Fake
Matrix: Coverage
Fit to window
Zoom out
Zoom in

Deployment protection rules

Reviewers, timers, and other rules protecting deployments in this run
Event Environments Comment
svcnvidia-nemo-ci
approved Jun 12, 2026
test

Automatically approved by queue manager

Annotations

3 errors, 5 warnings, and 58 notices
cicd-container-build (gcp, us-east4-docker.pkg.dev/nv-projdgxchipp-20260113193621/megatron-bridge...
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
cicd-container-build (aws, 766267172432.dkr.ecr.us-east-1.amazonaws.com, nemo-ci-aws-gpu-x2)
WARNING: The directory '/home/runner/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
cicd-container-build (aws, 766267172432.dkr.ecr.us-east-1.amazonaws.com, nemo-ci-aws-gpu-x2)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
pre-flight / pre-flight
Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: actions/checkout@v4, step-security/changed-files@v45.0.1. Actions will be forced to run with Node.js 24 by default starting June 16th, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
cicd-container-build (gcp, us-east4-docker.pkg.dev/nv-projdgxchipp-20260113193621/megatron-bridge...
Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: docker/build-push-action@v5, docker/setup-buildx-action@v3. Actions will be forced to run with Node.js 24 by default starting June 16th, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
cicd-container-build (aws, 766267172432.dkr.ecr.us-east-1.amazonaws.com, nemo-ci-aws-gpu-x2)
Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: docker/build-push-action@v5, docker/setup-buildx-action@v3. Actions will be forced to run with Node.js 24 by default starting June 16th, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
Coverage (unit-test)
Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea. Actions will be forced to run with Node.js 24 by default starting June 16th, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
Coverage (e2e)
Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea. Actions will be forced to run with Node.js 24 by default starting June 16th, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
Result
✅ Launch_Unit_Tests_Diffusion — PASSED
Result
✅ Launch_Unit_Tests_Core — PASSED
Result
✅ L0_Launch_models_nemotron_omni — PASSED
Result
✅ L0_Launch_models_nemotron_omni — PASSED
Result
✅ L0_Launch_converter_fsdp — PASSED
Result
✅ L0_Launch_models_llada15 — PASSED
Result
✅ L0_Launch_models_wan — PASSED
Result
✅ L0_Launch_models_kimi_k25_vl — PASSED
Result
✅ L0_Launch_megatron_fsdp — PASSED
Result
✅ L0_Launch_recipes_deepseek_fsdp — PASSED
Result
✅ L0_Launch_models_gemma4_vl — PASSED
Result
✅ L0_Launch_models_minimax_m2 — PASSED
Result
✅ L0_Launch_models_bailing — PASSED
Result
✅ L0_Launch_models_glm_vl — PASSED
Result
✅ L0_Launch_models_deepseek — PASSED
Result
✅ L0_Launch_models_qwen3_omni — PASSED
Result
✅ L0_Launch_models_qwen3_omni — PASSED
Result
✅ L0_Launch_models_bailing — PASSED
Result
✅ L0_Launch_data — PASSED
Result
✅ L0_Launch_recipes_gemma — PASSED
Result
✅ L0_Launch_recipes_deepseek — PASSED
Result
✅ L0_Launch_models_flux — PASSED
Result
✅ L0_Launch_models_minimax_m2 — PASSED
Result
✅ L0_Launch_models_flux — PASSED
Result
✅ L0_Launch_converter_generate — PASSED
Result
✅ L0_Launch_models_wan — PASSED
Result
✅ L0_Launch_models_kimi_k25_vl — PASSED
Result
✅ L0_Launch_models_deepseek — PASSED
Result
✅ L0_Launch_recipes_glm_vl — PASSED
Result
✅ L0_Launch_recipes_glm_vl — PASSED
Result
✅ L0_Launch_recipes_perf_config — PASSED
Result
✅ L0_Launch_models_glm_vl — PASSED
Result
✅ L0_Launch_utils — PASSED
Result
✅ L0_Launch_recipes_gemma4_vl — PASSED
Result
✅ L0_Launch_recipes_deepseek — PASSED
Result
✅ L0_Launch_recipes_gemma4_vl — PASSED
Result
✅ L0_Launch_recipes_nemotron_omni — PASSED
Result
✅ L0_Launch_recipes_llama_1b — PASSED
Result
✅ L0_Launch_data — PASSED
Result
✅ L0_Launch_training_megatron_mimo — PASSED
Result
✅ L0_Launch_vlm_inference — PASSED
Result
✅ L0_Launch_utils — PASSED
Result
✅ L0_Launch_models_nemotronh — PASSED
Result
✅ L0_Launch_recipes_gemma — PASSED
Result
✅ L0_Launch_recipes_llama_1b — PASSED
Result
✅ L0_Launch_wan_ckpt_conversion — PASSED
Result
✅ L0_Launch_recipes_perf_config — PASSED
Result
✅ L0_Launch_models_nemotronh — PASSED
Result
✅ L0_Launch_models_qwen3 — PASSED
Result
✅ L0_Launch_recipes_nemotron_omni — PASSED
Result
✅ L0_Launch_models_qwen3 — PASSED
Result
✅ L0_Launch_training_megatron_mimo — PASSED
Result
✅ L0_Launch_wan_ckpt_conversion — PASSED
Result
✅ L0_Launch_converter — PASSED
Result
✅ L0_Launch_converter — PASSED
Result
✅ L0_Launch_vlm_inference — PASSED
Result
✅ L0_Launch_recipes_nemotronh — PASSED
Result
✅ L0_Launch_training — PASSED

Artifacts

Produced during runtime
Name Size Digest
coverage-e2e-27444275309-1067e12b-5064-43b8-af42-dd56e9803f7d
760 KB
sha256:13b396fe3daa8374a96e629a73e8bd1288379bb12fb35acdaafe0e2acab8dcac
coverage-e2e-27444275309-106c64c7-103e-40d2-95de-8e00d92274a0
233 KB
sha256:7f3ad9cab7d7c94436963c7a7cd18a9deedd88fc3a80aae99e073452f6e6ffdd
coverage-e2e-27444275309-12f62368-f207-421e-b0f4-36de6c0562b9
311 KB
sha256:6820a1075b00d2095587a093356cb2ec78f6a48e5cde33c6ad365fa907ff2366
coverage-e2e-27444275309-1719b106-b91c-4d44-8d5f-730f771dbb99
233 KB
sha256:b2f04cda0fe71ad0dd1687e8d16c99d28a1410c6df87b43a7494e3b893aa3e58
coverage-e2e-27444275309-2a918aa5-f94c-42b9-892f-725692fc47ff
232 KB
sha256:e512922f4fc4a249e1b1babea94a2e8714535f9fb2b267b46a9b65a00b337b13
coverage-e2e-27444275309-30cc7a27-8e06-403a-99c3-4921a5afe7fd
345 KB
sha256:d1709884bfcf64234fedcb3014f90b45080b9eca8954b53cdaf79bdef2abef97
coverage-e2e-27444275309-33865205-32f9-4593-8977-1961328f7be9
305 KB
sha256:519193156a2e72b9845998f203b0679a1a5f1e659aa532865df5fc771f49941d
coverage-e2e-27444275309-397060e8-a5c7-4804-b041-6fdf52de4ae9
1.29 KB
sha256:edff4bfca230d9b753fb41f52a04e324fb382b26f2c639c9ad7d687bc6dfa732
coverage-e2e-27444275309-3a747eb8-39aa-4593-9149-b7842bb0303f
293 KB
sha256:551720f8360221d215ed044273d1efa3cd7b13683948c465e2fda97234d95317
coverage-e2e-27444275309-3c96784d-6348-4662-b4f6-543f5c9e70e6
699 KB
sha256:2926222c55029c1b4424615ff9636c31f1e5c7b1882e800ab79f4c0952e5517f
coverage-e2e-27444275309-3f8d8fca-2595-48cb-85d5-5fa38ee40834
340 KB
sha256:2aed25a1c33ee967107065c45fe4f39012489b08b58399c78e5a566aef96f247
coverage-e2e-27444275309-44a1fd63-9ef7-4412-8fb0-2a33c1913bb3
252 KB
sha256:49a8a7516d30d17ef34a19c78c134545a3c00c414211246b1c0c18a90c4a8039
coverage-e2e-27444275309-49b91a1b-ba65-4cf1-ad92-76a7feb6a337
345 KB
sha256:07dcdd89da6c66ab0f02c8f039ca91c75fa2233328ddb57f73a8f1aa0a5854bc
coverage-e2e-27444275309-4b18ada8-1db3-4c78-ac69-dbdc1a095fba
759 KB
sha256:f8d7b2bec017412c9cfc2fa33e984255caf5ba0eda15bfee393ca3b810521758
coverage-e2e-27444275309-4c153c73-62ef-4f47-85ff-71deff51e8e2
768 KB
sha256:2d3f50165404dc92d3316c8e7d4c741b2f46fd9df69a5a84f5521397e9d4d82c
coverage-e2e-27444275309-4e5d90a3-edea-499e-b04d-92d31290032d
700 KB
sha256:ca01ed31ef36a20d9dbfbbc5ab2b94989e2e4269033eb7ce352ee9348f5edb5b
coverage-e2e-27444275309-4f8dfdde-b336-41c0-bd8d-5ecb0463fec5
700 KB
sha256:9c9194d67c3203e4727a5ae73a7627e598ddb48a72cde28ecdf00821e76e596d
coverage-e2e-27444275309-506a831f-31c7-4132-ba63-15df501acac7
760 KB
sha256:5b1f1ed9d5a75b85279ecec57b0a245814aa85a824fafdee2c4810a0906474af
coverage-e2e-27444275309-660517ee-4f4d-4ae9-8138-772353ce73b0
699 KB
sha256:75c3c7ef818d45633ccce155e2bea678c821fe405e6c40724654359578d47108
coverage-e2e-27444275309-696985f9-fd96-411a-97e1-4a4327a04f0d
698 KB
sha256:5c212116abda7647c21f5ed606e5261af6501f0e32ab0512e9f80ac5547f726e
coverage-e2e-27444275309-6a6223f0-bbe0-47d8-a5c9-290a6f338e33
235 KB
sha256:bdbcaae246e7316c6d6b5442565d0e26999740cdcafa64a1fe111a7b685d05d5
coverage-e2e-27444275309-6c88bf60-9132-4c0d-8e49-36a4f2ec4d21
252 KB
sha256:528c3732ddd515b6c200ae470481d5fafd234af277b54edb57fbe12129fa2931
coverage-e2e-27444275309-7918323d-b5da-455b-af2a-46d6b110b621
354 KB
sha256:eb4a6d5a3b2e7340386aee1a333b7fd4cfc2dbef38c7ba71ff67ebf46f1aac24
coverage-e2e-27444275309-880ef205-c28a-41a6-bd2a-af4d3c79d5a3
757 KB
sha256:f0eabd9f97537ef62c27644d3201f1d2dd1840e199919f7b77b2d5e3137a04e8
coverage-e2e-27444275309-897a8d70-50de-4574-a717-fb04eb91b640
348 KB
sha256:84686442347b0d32156325cf0000e5d4f1cf1854a0e3feca5adcbe8c8a9a6691
coverage-e2e-27444275309-8f55b45f-0599-4135-9dcd-58f34ae82335
346 KB
sha256:56f713532dab83041a09c0c888b802dd81d3a5fce6ce5602121d6a65f6831c05
coverage-e2e-27444275309-9717428f-714e-4bbe-aab6-8eaf66dec8cd
1.29 KB
sha256:ce28b221c4ec76005394b4c51d87020c36c1ddcae42e4a4bbeb2f6257dfad0de
coverage-e2e-27444275309-986f8632-5486-40ff-abb1-fc00544a568b
313 KB
sha256:ddf5894260eeaa0010983afba7dbbb98c21d12473d8740e2a818158cc94b3fcc
coverage-e2e-27444275309-9dadc398-ed5a-4f48-aa76-f67345a5550f
699 KB
sha256:545318e6908fcc577b640a6aae37c6153e427ad4dcb15f6b6ad9adfde2f63548
coverage-e2e-27444275309-a88a3f79-27ee-4f3c-aa08-04cd6445c7d5
233 KB
sha256:22a0153019212838132ce1ab41e82783e84f59b75eda2bf2f61c0701afcf5bff
coverage-e2e-27444275309-a994ece7-54db-4f8a-afa8-7f133c7f7f13
348 KB
sha256:c0b5af3e59514c875c17eb81bc15335b90e59f3e70eb400b37cd7863962a45d6
coverage-e2e-27444275309-ac428ddb-d6dd-4cd1-8ff9-ec004b1d6b2c
759 KB
sha256:5f615f5da11c58345fae3dd45fcfa30a85f7d360fb3b03fdb8a52204b6d55fe2
coverage-e2e-27444275309-b0a63f5c-72aa-47e0-85d6-1431648ec053
264 KB
sha256:077e43a6d9a705fdf813d4ed5c9789da83bdc48c095c69a355247a56ea58561d
coverage-e2e-27444275309-b6caac27-7e8d-4db3-bbd5-cc8308d0abe5
267 KB
sha256:e60fffb45021f0e275f0f913602742d790aa239ed9dabe8da5a619e0aa298531
coverage-e2e-27444275309-bc992967-7eba-4ce1-8b63-c533cf5e63df
719 KB
sha256:1e3b5eac7d56b7dd7eb55a6c0390d11bdd3eb84e5d105c407997ddc3c4f90fba
coverage-e2e-27444275309-beeaec13-b4a3-430a-bf40-8652750799d4
313 KB
sha256:7028eba9d891dddcb7ebd5d0c61ee5c916624dcbe73424d2df99dc67da0821b0
coverage-e2e-27444275309-c2ad6e33-1849-4025-9c65-497ef6503074
264 KB
sha256:db910f82af47257cd16490e482bd8d1385729eb37845c110e06c30c7d7201f8b
coverage-e2e-27444275309-c415ca86-0291-46a3-b820-518f3d297398
305 KB
sha256:38512e72acd7de14c333ace17af848b76ae6735cc1a95a36204dac3ac4296795
coverage-e2e-27444275309-cc81d41a-fb99-4b1a-b0cb-cba8b8462003
269 KB
sha256:aed8013cf900419168ad70592dadf97eefc5cceef7db40817e1b6b396e7c6491
coverage-e2e-27444275309-cd8f7f07-b377-43d8-b7f9-54f2b02b539d
701 KB
sha256:8a20c58abfd045a8824a399eedd11ac15b06f3b1bfad8392d5029c601e72ef38
coverage-e2e-27444275309-d429a804-d7de-4838-b364-70858d08f6ad
346 KB
sha256:fc185ff6a1ae71301bcc5fa5ae34f8bfa4b94794a2d10eb9bb5df7c973e39f9e
coverage-e2e-27444275309-d64821e4-3711-4db6-8358-689139fee12a
269 KB
sha256:e8160d0b9afb1f804e21fb6cf2b95871d2909ac9b17dcd01df4556611fecbf5b
coverage-e2e-27444275309-d97d650a-dcf9-4e58-ad01-be9a75af1dbf
354 KB
sha256:04c83add2df57b6fcc7ee3a7c4fa7d5c914c0380b949ae10880d080496d22de1
coverage-e2e-27444275309-dcbdc61a-2d1a-4baf-a46f-c20c0e1a2b6c
757 KB
sha256:0d73fe52751964f46466f54d7f78af916fccbb1b70356a1e50509554957e3264
coverage-e2e-27444275309-dd189896-8e9f-467c-a100-f893beb10b32
699 KB
sha256:1633032edde63e3013f931a12eeec6f802efdd98fd2292c9a4410d34263c4828
coverage-e2e-27444275309-e5de3cc5-b3b4-4af1-bcbc-c0ff059cab0c
233 KB
sha256:07d4e12dc8a297c687cce20fd4adab9ab45a5734c7672f58066005a7e8fd9947
coverage-e2e-27444275309-e83f9325-3f47-453c-a455-9a731891c27a
1.29 KB
sha256:c0ecb1dce112d40cc73837b90b8ddc0291880550a4dc6e8cc0bfa819a47e57fd
coverage-e2e-27444275309-e8e45dee-06a7-48ca-ad5a-0bca7c9e76d2
233 KB
sha256:490dfe96a4a49e6f7845f229911fe233d283de2de049dbf243ed36043429991b
coverage-e2e-27444275309-e9e2b26c-19f6-4092-b6de-3c9d862e8bc1
313 KB
sha256:7cd04aea37b5d2c682cb6f56902220baa8385174aa3e33cdb7e6129174db0b40
coverage-e2e-27444275309-eb8dbbdd-e216-4ad0-a2d2-42cd65158829
762 KB
sha256:94183129029f4c9760319897a96d530e2eec6ea538478f3b08e080294b66ce52
coverage-e2e-27444275309-ed2f3870-1232-4342-88db-232563d3f554
719 KB
sha256:00513d643261e9c3870d699df2b1932e9c3359d71d0f1275b2295d69a1cf37cf
coverage-e2e-27444275309-ed896fed-938a-4769-b4ca-7c3b1416d330
233 KB
sha256:518d2506dc93ee9cae58caf7b5ffcca6b9579db632aa75f507f4594809dbfdb9
coverage-e2e-27444275309-f4e6158e-48d4-4510-835c-a33253064e2a
313 KB
sha256:5a16003d14a9ac9dbb8f16e3cffb7370e4461482beaf07202bb758b20c7c8cf8
coverage-e2e-27444275309-f6091d8a-4ab4-4f20-bbb7-1a8b0bab7033
269 KB
sha256:2c814d1ba252e480aeeda3910f2eefb8ddd11c106eb70943934259e0b967a7d1
coverage-e2e-27444275309-fe671a00-1740-4b46-9cf6-63cc9f09500e
233 KB
sha256:169374a3b7516f0d4d46062b6580c571ae54630fabcb9d0a6dfaed36ceb4b047
coverage-e2e-27444275309-febdbaee-7cec-4d30-8fee-0cb07b3997e2
311 KB
sha256:afa90a33139f753ea6430feca7aeb0524871e0c797071400781775ba2fc37e85
coverage-e2e-aggregated
142 KB
sha256:9d01098591f940eb1dde61e4c49c52430749c262636759f60d092821e576dee9
coverage-unit-test-27444275309-8a7aa8e3-de4d-40d7-8c2d-5044d78b6c36
310 KB
sha256:415e10be3cd9ce7a94cfcd27d1489292b1140481cea7a97157e26f999aaac1fc
coverage-unit-test-27444275309-b567d6f0-65e3-4705-a4c0-433ee1ecaa25
362 KB
sha256:2ffb5089920a12ac0b92d8bea9e412e775b7f72594490c1746319556ebeaf6ac
coverage-unit-test-aggregated
70.2 KB
sha256:a0ff3dfa2d6b6d8b5674d96829bac1169ee5d1e546f0e536fd532b49c2b4d447