Skip to content

[test] fix: shrink CP packed SFT functional model #16412

[test] fix: shrink CP packed SFT functional model

[test] fix: shrink CP packed SFT functional model #16412

Triggered via push June 16, 2026 02:03
Status Success
Total duration 2h 16m 49s
Artifacts 60

cicd-main.yml

on: push
pre-flight  /  pre-flight
11s
pre-flight / pre-flight
configure
7s
configure
Lint check
19s
Lint check
cicd-wait-in-queue
3s
cicd-wait-in-queue
cicd-compute-build-matrix
2s
cicd-compute-build-matrix
Matrix: cicd-container-build
Launch_Unit_Tests_Core
7m 7s
Launch_Unit_Tests_Core
Launch_Unit_Tests_Diffusion
3m 33s
Launch_Unit_Tests_Diffusion
generate-test-matrix
5s
generate-test-matrix
generate-gb200-test-matrix
7s
generate-gb200-test-matrix
Launch_Import_Check
1m 51s
Launch_Import_Check
Matrix: cicd-functional-tests-l1
Matrix: cicd-functional-tests-l2
Matrix: cicd-functional-tests-l0
Matrix: cicd-functional-tests-flaky
Matrix: cicd-functional-tests-gb200-l1
Matrix: cicd-functional-tests-gb200-l2
Matrix: cicd-functional-tests-gb200-l0
Matrix: cicd-functional-tests-gb200-flaky
Nemo_CICD_Test
8s
Nemo_CICD_Test
Coverage_Fake
0s
Coverage_Fake
Matrix: Coverage
Fit to window
Zoom out
Zoom in

Deployment protection rules

Reviewers, timers, and other rules protecting deployments in this run
Event Environments Comment
svcnvidia-nemo-ci
approved Jun 16, 2026
test

Automatically approved by queue manager

Annotations

3 errors, 5 warnings, and 58 notices
cicd-container-build (gcp, us-east4-docker.pkg.dev/nv-projdgxchipp-20260113193621/megatron-bridge...
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
cicd-container-build (aws, 766267172432.dkr.ecr.us-east-1.amazonaws.com, nemo-ci-aws-gpu-x2)
WARNING: The directory '/home/runner/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
cicd-container-build (aws, 766267172432.dkr.ecr.us-east-1.amazonaws.com, nemo-ci-aws-gpu-x2)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
pre-flight / pre-flight
Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: actions/checkout@v4, step-security/changed-files@v45.0.1. Actions will be forced to run with Node.js 24 by default starting June 16th, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
cicd-container-build (gcp, us-east4-docker.pkg.dev/nv-projdgxchipp-20260113193621/megatron-bridge...
Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: docker/build-push-action@v5, docker/setup-buildx-action@v3. Actions will be forced to run with Node.js 24 by default starting June 16th, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
cicd-container-build (aws, 766267172432.dkr.ecr.us-east-1.amazonaws.com, nemo-ci-aws-gpu-x2)
Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: docker/build-push-action@v5, docker/setup-buildx-action@v3. Actions will be forced to run with Node.js 24 by default starting June 16th, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
Coverage (unit-test)
Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea. Actions will be forced to run with Node.js 24 by default starting June 16th, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
Coverage (e2e)
Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea. Actions will be forced to run with Node.js 24 by default starting June 16th, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
Result
✅ Launch_Unit_Tests_Diffusion — PASSED
Result
✅ Launch_Unit_Tests_Core — PASSED
Result
✅ L0_Launch_models_wan — PASSED
Result
✅ L0_Launch_models_kimi_k25_vl — PASSED
Result
✅ L0_Launch_recipes_gemma — PASSED
Result
✅ L0_Launch_recipes_deepseek — PASSED
Result
✅ L0_Launch_recipes_deepseek_fsdp — PASSED
Result
✅ L0_Launch_models_bailing — PASSED
Result
✅ L0_Launch_models_qwen3_omni — PASSED
Result
✅ L0_Launch_models_flux — PASSED
Result
✅ L0_Launch_data — PASSED
Result
✅ L0_Launch_models_minimax_m2 — PASSED
Result
✅ L0_Launch_models_glm_vl — PASSED
Result
✅ L0_Launch_models_nemotron_omni — PASSED
Result
✅ L0_Launch_recipes_glm_vl — PASSED
Result
✅ L0_Launch_models_deepseek — PASSED
Result
✅ L0_Launch_recipes_gemma4_vl — PASSED
Result
✅ L0_Launch_converter_fsdp — PASSED
Result
✅ L0_Launch_models_qwen3 — PASSED
Result
✅ L0_Launch_recipes_llama_1b — PASSED
Result
✅ L0_Launch_models_nemotronh — PASSED
Result
✅ L0_Launch_recipes_perf_config — PASSED
Result
✅ L0_Launch_recipes_nemotron_omni — PASSED
Result
✅ L0_Launch_utils — PASSED
Result
✅ L0_Launch_converter — PASSED
Result
✅ L0_Launch_vlm_inference — PASSED
Result
✅ L0_Launch_training_megatron_mimo — PASSED
Result
✅ L0_Launch_wan_ckpt_conversion — PASSED
Result
✅ L0_Launch_models_glm_vl — PASSED
Result
✅ L0_Launch_models_nemotron_omni — PASSED
Result
✅ L0_Launch_models_llada15 — PASSED
Result
✅ L0_Launch_megatron_fsdp — PASSED
Result
✅ L0_Launch_converter_generate — PASSED
Result
✅ L0_Launch_models_kimi_k25_vl — PASSED
Result
✅ L0_Launch_models_gemma4_vl — PASSED
Result
✅ L0_Launch_models_minimax_m2 — PASSED
Result
✅ L0_Launch_models_deepseek — PASSED
Result
✅ L0_Launch_data — PASSED
Result
✅ L0_Launch_models_bailing — PASSED
Result
✅ L0_Launch_models_flux — PASSED
Result
✅ L0_Launch_models_nemotronh — PASSED
Result
✅ L0_Launch_models_qwen3 — PASSED
Result
✅ L0_Launch_converter — PASSED
Result
✅ L0_Launch_models_qwen3_omni — PASSED
Result
✅ L0_Launch_models_wan — PASSED
Result
✅ L0_Launch_recipes_deepseek — PASSED
Result
✅ L0_Launch_recipes_gemma — PASSED
Result
✅ L0_Launch_recipes_glm_vl — PASSED
Result
✅ L0_Launch_recipes_gemma4_vl — PASSED
Result
✅ L0_Launch_recipes_perf_config — PASSED
Result
✅ L0_Launch_utils — PASSED
Result
✅ L0_Launch_recipes_llama_1b — PASSED
Result
✅ L0_Launch_vlm_inference — PASSED
Result
✅ L0_Launch_recipes_nemotron_omni — PASSED
Result
✅ L0_Launch_training_megatron_mimo — PASSED
Result
✅ L0_Launch_wan_ckpt_conversion — PASSED
Result
✅ L0_Launch_recipes_nemotronh — PASSED
Result
✅ L0_Launch_training — PASSED

Artifacts

Produced during runtime
Name Size Digest
coverage-e2e-27589119401-07c4766d-1e9a-4da3-9491-5591c3f0dee3
314 KB
sha256:c2144e24053266e000f7d18450958d2b2e924abfb5fda24350d293440a463c41
coverage-e2e-27589119401-0c8e7e99-d2a0-4792-afb8-2b98f6004aa5
719 KB
sha256:b2b267e59631a9bfcf212a4748b6cf32bd9e0fae6afc53bb1c7ff6b86d045638
coverage-e2e-27589119401-12ed2ddf-da4b-4991-9313-f60132c18ca9
760 KB
sha256:6e536723b2122c30ff5327c6068cfdc6e1183bc959cc79a4b0ab310306d2cefd
coverage-e2e-27589119401-1d460254-4a2e-448f-b263-cbd98d438aa6
700 KB
sha256:3b500cfbe3dbd7ea6531656379dcf96253e1f420f193199a03df5cabf7ecf809
coverage-e2e-27589119401-28e65fec-2b6c-4b17-8d08-8697582cd5d8
701 KB
sha256:ce771a63792d5663d60ffc077d75c54773bee257062e9f6b966cd6dfea6ba636
coverage-e2e-27589119401-2a6c77a1-94ec-4026-923e-d2febb36511b
700 KB
sha256:e7f685e1d2e6d661e3d43cbd2595d290acc0c873ed2b09c7542dd8eaf5995518
coverage-e2e-27589119401-3bc815cc-175f-4957-b79d-f2fa38af0869
270 KB
sha256:fcc3f9f25483616583f30df6390b48e4dfd472b004aadd42c646f99a78f05f33
coverage-e2e-27589119401-3f52c87f-7008-4881-a3ee-11f486cc51c9
235 KB
sha256:971f9d2574fb478337d8f24ee7666511b4725e830299c586855bfef89a91c923
coverage-e2e-27589119401-42c2647a-14cb-4da1-8dd6-35221121f37f
1.29 KB
sha256:3fc04f346d4e30f9afc6f70fc703b88f60c8acd0b98637ff0e116075b4776b10
coverage-e2e-27589119401-46df9a0b-d91e-4c5e-b698-89a6334209b2
758 KB
sha256:d50d1c4f89ae9c31e300c84d1c2385660a158f900ae726116cf0edb0c49a8143
coverage-e2e-27589119401-4a4db029-7a37-4ddb-a2f2-075910975170
311 KB
sha256:9d3905934c24a6da468c4ecf73163f864a67d7d2cc227f51114382d195c89c1a
coverage-e2e-27589119401-54a4413e-f182-44b9-9c11-ff08eddd13d7
233 KB
sha256:97e9a512a7818df63cd71a8acae8f26a652304b5e4981aa120605c5cff020fb1
coverage-e2e-27589119401-58dae7ef-a253-4744-b17a-7c5d5bc862a7
340 KB
sha256:e1d6e77c241c6e965d1c3c7adbf8f5328f7fba6342210f4cd518ff67ee868594
coverage-e2e-27589119401-591d7ef5-5b54-4bee-9035-a0b8bb8f2ee1
758 KB
sha256:65a2c5cee70366f98ec4af2ae9dbe6f46bb5e61dbef29e367635db1ff2339ad9
coverage-e2e-27589119401-5a4b73f9-a124-4ce1-adc4-05678af2b0dc
759 KB
sha256:a0a5a992f167c7ffde8c05e976b74fc7ed4b40b5a5394d2170c947897e19c7af
coverage-e2e-27589119401-5b65aa90-0f23-44ab-90a3-2e4cc2682de8
762 KB
sha256:1c89a283677b3ebbc65d0731eac728190880280a9b9380df3f0153478ba882d3
coverage-e2e-27589119401-5b75bade-35bc-465d-99ae-ae98b33c276e
233 KB
sha256:ce1ed3cb276d390a0099d0aa3af4510e25b22b8082444e80b02807ddaa7f1a39
coverage-e2e-27589119401-6bf077f3-2b4f-49f3-8121-875e4867fbab
270 KB
sha256:e3654f19d06d0e087e9f8baf376b041c6a4f2bd3a863da8910f555b6654297e5
coverage-e2e-27589119401-6cb21aad-e659-439a-80f0-0ca6fb6313fa
1.29 KB
sha256:b00915c988bd404b7b0ced6a195029d1491346fd057370533d601f3465d82732
coverage-e2e-27589119401-6e46ba87-9100-4390-bc2a-a7dc71a88f0c
311 KB
sha256:2d87bbde6d2dcd7274ec8886d0a6cd8d195d0a9dc8b40b21f58b6ae0a9a435f2
coverage-e2e-27589119401-738462c7-bfa4-400d-aa56-1a425d8c49e4
234 KB
sha256:7e4930480a89a9cde6aec9da5ae754a72c7425386ad7866ad64c6117047fdf07
coverage-e2e-27589119401-7392e552-e980-4182-9b83-a7e902972057
760 KB
sha256:49a00e7b38bde056551e5cc2a47f8e42397d254794f1dffc3499f4a864a92acb
coverage-e2e-27589119401-75b72178-8b43-411b-9308-3b3f439c6623
267 KB
sha256:6216c89e4043b10f88bc1a7d49f0739cc69fbe018e0c5dec1e10c021260bc315
coverage-e2e-27589119401-76590188-6c6b-4d42-97e7-495e8e8818a3
1.29 KB
sha256:59066f4d3b049a72067b28bdc1a1220404011cd0be5ad0d291081cf8a07c282d
coverage-e2e-27589119401-79ecc5f3-fa25-4848-a7d5-f1c402aabedd
264 KB
sha256:a8818eaed34d66fa256a1ec8430b21bfeabb0cf5d01c96fc0c0dccda079e1e72
coverage-e2e-27589119401-7b5437ac-57da-4642-a147-5e005bc88bd7
769 KB
sha256:1527eea2197459cc0b3ac5d984d1fefb9e96d11647a1600f68e88de6ca2c9f49
coverage-e2e-27589119401-7de39148-f2b9-4740-aa3b-f6018d1a2487
233 KB
sha256:0b9e9b7c63efe6e0374805d8cbc95a07885abd049b83b942d32d7da75435d110
coverage-e2e-27589119401-80561bcd-3d43-488a-b8b4-b28c0787ad86
345 KB
sha256:661ae774a5c88ee3cdc8c518e3457ce70658780fcee647e7635e42fceeef51f8
coverage-e2e-27589119401-8523f122-767d-45b3-b7ba-01ffc27093e6
700 KB
sha256:ae505ed1a63924ffda7bc45a1436577dc03f00d903c92b9a311e3931281a0252
coverage-e2e-27589119401-8afdac23-b5f6-4146-90b0-449cebf93c13
348 KB
sha256:9f129e78177a4bb4e69bdd7c20ebc48a82d69c0c4cc04228f4443a1db2f79d7e
coverage-e2e-27589119401-8c42fdb5-0533-482c-93ef-83437652aa08
346 KB
sha256:4447058e87e7cfa1fc71a04967097720e9c0f6da349020e1d1a0bbfe4a8c6baa
coverage-e2e-27589119401-8dcfeab2-f607-422d-83e8-a7251de18a3f
264 KB
sha256:c0ab9696e0e94a0273eda62107f0f322ee2efd79d16ad5e3daf53b9618d77660
coverage-e2e-27589119401-978d46f0-f73c-47ab-93aa-83eb7e3ca923
699 KB
sha256:35c53c1f2142bbf55f5077b3608d198aa637156210543481f9e39f2246fd1f60
coverage-e2e-27589119401-980d5d6f-e201-4e58-8466-3fb11557685a
699 KB
sha256:82f72f69ec9ba55d08a50622002f4d66956a8639180666898f280e808f53defd
coverage-e2e-27589119401-9b0ce6f0-6949-4c07-9906-68e837fc6d8b
757 KB
sha256:6da25a14826f4a495ad8ea90fb6618322733d0e46df142656620c7c6b149571a
coverage-e2e-27589119401-9c9c8417-d8d5-4da5-bc06-43ebc240c9d5
348 KB
sha256:16003f4ac43ee576925190f53da01ae2f913ada46c076a0901877c3f8dfadd9c
coverage-e2e-27589119401-9f7e4b16-e1af-4c86-af02-76a6f4191d3f
346 KB
sha256:a02f6f809b64b3a17c848a6bcad526b64b3b09a4e6f7fe40f171a83cb64a4432
coverage-e2e-27589119401-a06cef3a-3d85-4015-b6ea-5f54d545ea9f
305 KB
sha256:3077ce3845592029ecfe8365ec85d84db9ae2bd226d40dcea306be072c4030be
coverage-e2e-27589119401-a2c64892-3020-408b-80c0-d8501294a8e8
700 KB
sha256:8b1e3c5cfd54d67ade8d9f63a384cffaccbdfea60ca97eb9e49722a9a00c9dbb
coverage-e2e-27589119401-a3afd54f-cbc2-4b8a-8668-39fe5992f8d5
313 KB
sha256:fb016138b2e5a83212d303e9be3b97fe698610476cacad66fd409c419a08a545
coverage-e2e-27589119401-a6a84c8d-4b4e-4293-bf04-0f24e4e62192
345 KB
sha256:bf56e1dba6905df9c0574f838562bf16a0540c7b99d8baaa32eef34e7a82c776
coverage-e2e-27589119401-a82d339a-d784-4db6-87c9-296ad1d82553
355 KB
sha256:686097d0a6b465f9eb7f66949fa547b5ba2235478f0bdd0ccfb743d2afd0ea17
coverage-e2e-27589119401-adaef5da-362a-4da8-8a2e-2d634a5f29cd
719 KB
sha256:22d3b1e6e92ca8a457ddfaa381f686eefbffa7a6abe707c0ae88c3749db63452
coverage-e2e-27589119401-adfd9d80-967e-44cc-8961-45cc64f803f1
233 KB
sha256:0216c37b71ef280106b1e311e8b89e540b16e7d13704308db0423770a217bca7
coverage-e2e-27589119401-bd47c62d-50d5-42a5-b992-cda0a5eeab29
355 KB
sha256:d7caeaac56cc68d7a273252b1099b274b0c7ee4b03f1591201f07d6097973230
coverage-e2e-27589119401-c50b5c37-0507-4280-b866-75c444e32073
305 KB
sha256:38a0f0eb9f2f7342ee84f705c35b751819aa2ff47bce60e7855a2d209ca98680
coverage-e2e-27589119401-cadb8ed1-0a36-492b-ba09-345f95478878
233 KB
sha256:fa7cdb5e98aa64333d9374bbc42d66f073213805c31376b77f38f9d2b8420d2c
coverage-e2e-27589119401-cb6a58ac-19d7-409d-a059-cc14a813b26a
253 KB
sha256:b5f4249566b97c42bd80c1378164284676e1e12cb2d85ae07fb99d362b53cefc
coverage-e2e-27589119401-ccbd92f4-0f02-489d-9ca5-677f32087ffc
232 KB
sha256:993c3355982bfb37d75dcfb8bf405f65fde323a933c535b9864aeb35a2609e9c
coverage-e2e-27589119401-d0400c8c-df36-4654-bb1f-172292ec159e
270 KB
sha256:fd301aaf314d2100e7adee9037c6623879673497ac6ae5b70d07de340c152b66
coverage-e2e-27589119401-e080cdea-17eb-421f-85af-bd0407e892af
699 KB
sha256:ca057709b8f5fd657f33c15a9eef62a3604ab6020c460cbca5e14bfae8354839
coverage-e2e-27589119401-e1ed8186-9d95-4c8d-bd79-4499a92e8a93
233 KB
sha256:7628b943051ac0d6915d013d2139a4c3dd777c50985f370d5775f5ef5c433913
coverage-e2e-27589119401-eb5abc02-d398-4df3-89f1-329a091e4de7
294 KB
sha256:ec8654d6b145a86c5af45c0b8536b687899e2d9804802ccfa70f375d3bde93c7
coverage-e2e-27589119401-f941d26d-786c-4228-8d8c-e13f0ddf8774
314 KB
sha256:223781799c4506d7a2d89284f5065b6bbaf781e9a789d59f630f06cd1ad9bab9
coverage-e2e-27589119401-fe3b1100-1eb9-4cbe-8df4-f855934abdf0
313 KB
sha256:34b06f93d1ba0d385038214dc5b8caf93e42bf3136faeea5632a059ddeec130f
coverage-e2e-27589119401-ff1407af-e365-40bb-bb81-d3bc0de916bc
253 KB
sha256:a695052c32c900c40b483a9f320d31f2fa2f9b4a3a0686c5f42ef783c2ca63ae
coverage-e2e-aggregated
142 KB
sha256:a0efe17a0e99c0cd67dd5956990039a033bda3706d393538e181d611825d54a0
coverage-unit-test-27589119401-36354e5b-25c6-4b15-8378-4161062d72ef
363 KB
sha256:52f9b0ed50f5971d60e9079207e598520ed0933dd851a8b43f28e4720d9e589f
coverage-unit-test-27589119401-a090f9bc-a6f6-414c-8adf-0ee3bdd1d46e
310 KB
sha256:a02b33add873938cb61a2660b0323c4771e340efcec058597a08a74a0179753d
coverage-unit-test-aggregated
69.6 KB
sha256:7814348227aa5a0cbf804813184ff88d0a7437fcb28cbe47c6c28a3b9837f622