Skip to content

fix(qwen3-asr): preserve audio config for checkpoint export (#4361) #16411

fix(qwen3-asr): preserve audio config for checkpoint export (#4361)

fix(qwen3-asr): preserve audio config for checkpoint export (#4361) #16411

Triggered via push June 16, 2026 02:00
Status Failure
Total duration 2h 41m 33s
Artifacts 54

cicd-main.yml

on: push
pre-flight  /  pre-flight
17s
pre-flight / pre-flight
configure
6s
configure
Lint check
21s
Lint check
cicd-wait-in-queue
3s
cicd-wait-in-queue
cicd-compute-build-matrix
2s
cicd-compute-build-matrix
Matrix: cicd-container-build
Launch_Unit_Tests_Core
7m 3s
Launch_Unit_Tests_Core
Launch_Unit_Tests_Diffusion
3m 19s
Launch_Unit_Tests_Diffusion
generate-test-matrix
5s
generate-test-matrix
generate-gb200-test-matrix
6s
generate-gb200-test-matrix
Launch_Import_Check
2m 3s
Launch_Import_Check
Matrix: cicd-functional-tests-l1
Matrix: cicd-functional-tests-l2
Matrix: cicd-functional-tests-l0
Matrix: cicd-functional-tests-flaky
Matrix: cicd-functional-tests-gb200-l1
Matrix: cicd-functional-tests-gb200-l2
Matrix: cicd-functional-tests-gb200-l0
Matrix: cicd-functional-tests-gb200-flaky
Nemo_CICD_Test
12s
Nemo_CICD_Test
Coverage_Fake
Coverage_Fake
Matrix: Coverage
Fit to window
Zoom out
Zoom in

Deployment protection rules

Reviewers, timers, and other rules protecting deployments in this run
Event Environments Comment
svcnvidia-nemo-ci
approved Jun 16, 2026
test

Automatically approved by queue manager

Annotations

7 errors, 3 warnings, and 54 notices
cicd-container-build (gcp, us-east4-docker.pkg.dev/nv-projdgxchipp-20260113193621/megatron-bridge...
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
cicd-container-build (aws, 766267172432.dkr.ecr.us-east-1.amazonaws.com, nemo-ci-aws-gpu-x2)
WARNING: The directory '/home/runner/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
cicd-container-build (aws, 766267172432.dkr.ecr.us-east-1.amazonaws.com, nemo-ci-aws-gpu-x2)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
L0_Launch_models_qwen3
Process completed with exit code 1.
Result
❌ L0_Launch_models_qwen3 — FAILED (exit 1)
L0_Launch_models_qwen3
Process completed with exit code 1.
Nemo_CICD_Test
Process completed with exit code 1.
pre-flight / pre-flight
Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: actions/checkout@v4, step-security/changed-files@v45.0.1. Actions will be forced to run with Node.js 24 by default starting June 16th, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
cicd-container-build (gcp, us-east4-docker.pkg.dev/nv-projdgxchipp-20260113193621/megatron-bridge...
Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: docker/build-push-action@v5, docker/setup-buildx-action@v3. Actions will be forced to run with Node.js 24 by default starting June 16th, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
cicd-container-build (aws, 766267172432.dkr.ecr.us-east-1.amazonaws.com, nemo-ci-aws-gpu-x2)
Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: docker/build-push-action@v5, docker/setup-buildx-action@v3. Actions will be forced to run with Node.js 24 by default starting June 16th, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
Result
✅ Launch_Unit_Tests_Diffusion — PASSED
Result
✅ Launch_Unit_Tests_Core — PASSED
Result
✅ L0_Launch_recipes_deepseek_fsdp — PASSED
Result
✅ L0_Launch_models_deepseek — PASSED
Result
✅ L0_Launch_converter_fsdp — PASSED
Result
✅ L0_Launch_models_wan — PASSED
Result
✅ L0_Launch_models_bailing — PASSED
Result
✅ L0_Launch_models_minimax_m2 — PASSED
Result
✅ L0_Launch_models_bailing — PASSED
Result
✅ L0_Launch_models_glm_vl — PASSED
Result
✅ L0_Launch_models_kimi_k25_vl — PASSED
Result
✅ L0_Launch_models_nemotron_omni — PASSED
Result
✅ L0_Launch_data — PASSED
Result
✅ L0_Launch_models_wan — PASSED
Result
✅ L0_Launch_models_flux — PASSED
Result
✅ L0_Launch_models_kimi_k25_vl — PASSED
Result
✅ L0_Launch_recipes_glm_vl — PASSED
Result
✅ L0_Launch_models_qwen3_omni — PASSED
Result
✅ L0_Launch_recipes_deepseek — PASSED
Result
✅ L0_Launch_recipes_gemma4_vl — PASSED
Result
✅ L0_Launch_recipes_gemma — PASSED
Result
✅ L0_Launch_recipes_llama_1b — PASSED
Result
✅ L0_Launch_vlm_inference — PASSED
Result
✅ L0_Launch_recipes_perf_config — PASSED
Result
✅ L0_Launch_utils — PASSED
Result
✅ L0_Launch_training_megatron_mimo — PASSED
Result
✅ L0_Launch_recipes_nemotron_omni — PASSED
Result
✅ L0_Launch_wan_ckpt_conversion — PASSED
Result
✅ L0_Launch_models_qwen3_omni — PASSED
Result
✅ L0_Launch_models_qwen3 — PASSED
Result
✅ L0_Launch_megatron_fsdp — PASSED
Result
✅ L0_Launch_models_gemma4_vl — PASSED
Result
✅ L0_Launch_models_flux — PASSED
Result
✅ L0_Launch_models_minimax_m2 — PASSED
Result
✅ L0_Launch_models_glm_vl — PASSED
Result
✅ L0_Launch_models_deepseek — PASSED
Result
✅ L0_Launch_converter — PASSED
Result
✅ L0_Launch_data — PASSED
Result
✅ L0_Launch_converter_generate — PASSED
Result
✅ L0_Launch_recipes_glm_vl — PASSED
Result
✅ L0_Launch_recipes_deepseek — PASSED
Result
✅ L0_Launch_recipes_gemma4_vl — PASSED
Result
✅ L0_Launch_recipes_gemma — PASSED
Result
✅ L0_Launch_recipes_llama_1b — PASSED
Result
✅ L0_Launch_converter — PASSED
Result
✅ L0_Launch_recipes_perf_config — PASSED
Result
✅ L0_Launch_vlm_inference — PASSED
Result
✅ L0_Launch_utils — PASSED
Result
✅ L0_Launch_recipes_nemotron_omni — PASSED
Result
✅ L0_Launch_training_megatron_mimo — PASSED
Result
✅ L0_Launch_wan_ckpt_conversion — PASSED
Result
✅ L0_Launch_training — PASSED
Result
✅ L0_Launch_recipes_nemotronh — PASSED
Result
✅ L0_Launch_models_nemotron_omni — PASSED

Artifacts

Produced during runtime
Name Size Digest
coverage-e2e-27589013134-048c10c3-8beb-4ec9-96a8-11a01e564647
229 KB
sha256:179b7f98a9d68ee526779d827a74cf89dc6ee0622b7fd843eec6b163be791db1
coverage-e2e-27589013134-067da8eb-6d5e-411a-a456-66a4ad145fa9
249 KB
sha256:bebdeb320c80b892b6738fe2fc01388383caa640c7ba82d1cd04d33aefec6ea3
coverage-e2e-27589013134-1c0fd150-a781-4724-9314-5896ce5cbb71
753 KB
sha256:db9f63302d0fa08bdfca69468369f698d6c560a59a4b7ac76f75d53bbd22e38f
coverage-e2e-27589013134-1c97b1e4-ca7c-4625-bd31-89f743e2b9f4
308 KB
sha256:0e12b34777fd9089591fb3eb93c70aed7dfbc94dea620b694702ebe968880c16
coverage-e2e-27589013134-1cf9256e-bffe-4384-9c26-86982f8e6943
694 KB
sha256:6479d2a61ba6bf3178eb0c14699f4d5399632fbd09c938c784a606fa978de24c
coverage-e2e-27589013134-21df8ce0-99f1-4344-9faa-33eb8b56a6bc
341 KB
sha256:4025fa554326cfec6630246b1c873c7e19635a3619bb8662fcdcf7e4c05f5585
coverage-e2e-27589013134-295a5b76-0279-40cc-95a5-bce2572a7330
259 KB
sha256:76b249f36d1086ba32a27cebdb9bef30b6300d7312f71fbb7280d130fdf802b0
coverage-e2e-27589013134-32029228-8451-41c3-913d-f3ff37ea95f9
231 KB
sha256:88bb3f25f95899add24b9e310e0aa91fe5446430d9075b6701e6c3eb1b33d77c
coverage-e2e-27589013134-32ebd88d-e412-4aee-9114-0cba06f6a361
229 KB
sha256:a3bd4a3435f80a055581191827e58d0e6f9d88f12279ffe1f2a91d564f316c2a
coverage-e2e-27589013134-35f91b9d-a250-49e2-a686-900e348770ca
300 KB
sha256:0c9355066f88c984459b0a96e613451ab2b91b6b0a4556da38842fb073ce57c3
coverage-e2e-27589013134-3d5ef7bb-2e5f-4018-91ac-0061e7dd55d2
265 KB
sha256:f804b8ebfb42dc1ea77150557f887c65afb0a614f4f5bebda327ac38989b268d
coverage-e2e-27589013134-3ffd93e0-9ee8-4851-aff1-022ae94f64da
289 KB
sha256:263a6e9e795314e5152a81cb1157b40429cc93eea519b25e5ea7fe5b8206bae6
coverage-e2e-27589013134-452f4220-56f9-4fc7-baf2-733ed2ddf903
339 KB
sha256:2090375d0afecfaa1adcfea4bd08850d491dbed9bf994714d3a59494c788e9f1
coverage-e2e-27589013134-4a71e6c3-aa57-4d8a-b048-9049339b23e7
229 KB
sha256:ac63c65e1e03211ba514f3c0e5dd90e625b9328d8603ee5ab73d5170412447db
coverage-e2e-27589013134-4ac256dd-6922-4866-9580-4805f17b8235
694 KB
sha256:050b96e82246cede62b7ec932a782f19b97f21bae97e3a448449b4503be23bfb
coverage-e2e-27589013134-4f322a7f-dacd-40af-845a-f736ec70d224
350 KB
sha256:d6ffe106c43f2ae2934802060fdd50ceceb72bb8f417ccdc90a62ee9f66ca0c4
coverage-e2e-27589013134-528157f6-ff65-45fa-972c-299915914107
263 KB
sha256:aea6f1f0b6cc5844a260861ac26ba81bf8e7320de068703ef2611fb2353b460c
coverage-e2e-27589013134-54447f66-8d72-42e8-913b-7f9d67540433
306 KB
sha256:950815eeede88dcf12e2da35886757f3db1efab988ffaa6ddc51de0b3dc65070
coverage-e2e-27589013134-59a82bda-1c4e-47b1-bedf-526e467fa01e
259 KB
sha256:fbfde82520dbb1e2634ca10d8014315f8aea6503043126a0581f376903dd05b8
coverage-e2e-27589013134-5b839c23-c20e-48d5-bbc7-630dd4748dac
306 KB
sha256:3ff87b29e23047b195f78d9f6e0168fc7d91ba1fa54958b4052834ecc85f326c
coverage-e2e-27589013134-5d2eeb2e-7d26-4bf6-9799-398767c742f1
229 KB
sha256:f1f83e22acff7ad1e01ba6e1e071807bf2b0caa8eb35980155ae173813457232
coverage-e2e-27589013134-648b98cd-a7f0-4556-b7cc-bf5f2d0c625e
1.29 KB
sha256:9f8736ced278803de04bdf929c4ba2fb2b037e7f69ca97e0a0f097a5b7989749
coverage-e2e-27589013134-6844f103-9caf-4ca5-81d7-01ac98306cd2
343 KB
sha256:e7c6a4c6507f0e825de1fd3a61c23e8faf3fe95ad479ca4c34f255ee2af7fc73
coverage-e2e-27589013134-6853f455-a952-4bf3-9e24-dbf7b849d02e
230 KB
sha256:c4b4063e120577e53177cf9828d2a70f69f733b4b753af55dea190102a2cd9e7
coverage-e2e-27589013134-7121a770-7140-454b-82a3-957d31c1c592
230 KB
sha256:57b29b770a40ae1354a3d90619062254e83df469380ae1535d2ca2f9dfa5d364
coverage-e2e-27589013134-744b57d8-e776-4cfa-b193-a49d8df27e5f
308 KB
sha256:cf479f6bac2618de4b61efc27f51c403ec87a485cef228bf23f6110700171726
coverage-e2e-27589013134-765b31af-13ad-4db9-99cf-ae46e95d93ca
694 KB
sha256:785e316824efabc8a5da1d2bdf2cf1e5a53ee31826add7be8cf30af038c3c80d
coverage-e2e-27589013134-77ded9f0-deb6-48cd-9896-fb4e223534a4
249 KB
sha256:1c8437416187815d2741cd0affb9eafe1c51c378709d020a0892e9808428f015
coverage-e2e-27589013134-78d4ebd2-0534-424b-a343-78ec73e118c0
695 KB
sha256:388abd30a417d6b5b88def321b587bffcdbcaad3639743a9fe357ebc7aaec352
coverage-e2e-27589013134-7a6fd616-451e-404c-8cf9-4ba8f8b31832
753 KB
sha256:98d47de652474726b4ec66e3282e80c6c1d582a553fefd0ea942f9b0aee35c1a
coverage-e2e-27589013134-7e71ae8a-3fe7-4605-b8d9-24948fd73724
694 KB
sha256:5f150a6ae2d4a1dd353fa8fb2de46bba98126b2c4911ff56ea6345c4759bc2dd
coverage-e2e-27589013134-7e7282eb-c6ed-4a81-860f-49040a188528
335 KB
sha256:76f0dc09d6669d08bce27f8654f2508f8e2269a1617127ea968c8d63ed5a8d3b
coverage-e2e-27589013134-806f6431-a093-4491-b52d-9bfa874f1cd1
755 KB
sha256:6bf4bafb59003c943b118b7fdf7e121aaf095f69f953355d18402b618d5ae201
coverage-e2e-27589013134-8597f7ee-8bf0-448d-8206-04356c04cf7c
265 KB
sha256:9c4000407ebf37e457e06db43786dfa500d2d3c5684a6d7f9d8f763a3b8d9a99
coverage-e2e-27589013134-8e4ed053-dafb-42a6-8081-887713ed4949
339 KB
sha256:2a55886e7bdae30b4b4b635c19a55587ab7a6f7e672a9acd5bd91049c065207b
coverage-e2e-27589013134-8e937346-b97b-4c87-a708-3e20e8c06cfe
754 KB
sha256:6752eec18c667a6f4c7e0e00d0efb2b1877603ea848b253880b5b493439dfa1a
coverage-e2e-27589013134-99c35fc7-6b9f-4a56-957d-be931dd46ecd
763 KB
sha256:3f4be91a2620ecb2acefd9bcf45efe3b270e400bb8e4fb9400c61be45ae7eb8b
coverage-e2e-27589013134-a2f6a6d3-6f55-4e8d-9d45-00a31f282341
300 KB
sha256:2c0b2bfd0f66b8c25628f2fdc17b93c65d04b35ecd1327f44f0d866bf226cf3e
coverage-e2e-27589013134-ae5be22d-f69f-4d96-93c9-743762b99624
343 KB
sha256:cbfc561781947a315ad4553859e24cb85dfbb47f557ea5f10d1b506fb0100f79
coverage-e2e-27589013134-b2ee2285-de58-499e-94c1-311da71e2621
265 KB
sha256:ba1ed2ee95907ce6938c2d263b3844d349840403c95002098b863cb7a7ee732d
coverage-e2e-27589013134-b2f9d476-af37-4141-b4e3-751af3b4d348
308 KB
sha256:3f74e2e857edc36c5b89f5ad6f32b40b527d1cee4f0075977e12da861f5f391d
coverage-e2e-27589013134-b51e32bd-5ee0-4ac6-8500-424518395754
350 KB
sha256:00b921324d953da1c829c618fcec572be8482a59423ec4bfaba476994db3bf98
coverage-e2e-27589013134-c289afe9-a987-4680-a6c0-2276e8ad1327
714 KB
sha256:c9a6081fee4048e8206e5e5851468333c7d9ce165fcfbf5d016ca1e7e160c237
coverage-e2e-27589013134-cddfe98d-6032-472e-919f-196c29785f26
714 KB
sha256:5f33eef35aac9a39e0289ebe3def740d816929f10f7b69279ed0ef67fcb6fdcc
coverage-e2e-27589013134-d373252f-8f46-4fea-bfed-eec3b353f302
229 KB
sha256:788a2cb3a9e0284ec746c2c6ef466eab57b4fe8fc718c7fbb1e452489d2c3fb7
coverage-e2e-27589013134-d8605268-9d88-48f6-9e42-454ace0492ac
341 KB
sha256:d710494912c3f783ea1b65ef0fae36bf02123f9a1da3dfe86456d163298f6f1e
coverage-e2e-27589013134-dc0c3212-6266-42ff-8e61-001ca20c88c5
308 KB
sha256:71c409504dc821616194a88796af4b3b70b800e120c60a5773f85a6c9bd91c4d
coverage-e2e-27589013134-e838b08f-c04e-434b-8c20-df8df8081196
229 KB
sha256:74d807b5022fce61e55d53a4de221a67dc6f48276962d53d6635956a9aece225
coverage-e2e-27589013134-e8f719c8-3aec-4d1b-81dc-e8d0a4015268
755 KB
sha256:3273c9d6764ec1271ad710651528ee89976b7fb695c2b247ba019e93a89f8b7c
coverage-e2e-27589013134-f8aa0540-6837-4123-95e7-be20b0c330f2
757 KB
sha256:933da071e09622743d32d590aa7a6c8d5ede18fb3bd9516fa52227f1cd45cdf7
coverage-e2e-27589013134-f93a5f0d-f12b-4d4e-a2fe-a780d5d04a20
754 KB
sha256:5667103ec1c36655d326c690858ccec89c9a06690312b9348f2a1c3f22b88545
coverage-e2e-27589013134-fbe76d37-b291-4cd8-bc3f-0102fbf793a6
230 KB
sha256:5b45d50617fc5380e3f34e8170a3272852520d4bd9f55a7592ac32dbc662281a
coverage-unit-test-27589013134-1344682d-f02d-4cf4-b94f-081959afc841
356 KB
sha256:bdd475d304e5dcf39c748b3bf21ffdad932f5d6ebc47172209a51e474a02bee2
coverage-unit-test-27589013134-cd2a2306-16e5-4ec3-8917-a380ccbf765f
304 KB
sha256:62709f11c1aea1c26a5f31d2325b628f219ce1b27102273df8789f71190192e6