[AutoDeploy]: Pre-merge L0 cost-reduction

[galagam]

🚀 The feature, motivation and pitch

1. Move perf tests to post-merge. Currently perf/test_perf.py::test_perf[deepseek_r1_distill_qwen_32b-bench-_autodeploy-float16-kv_frac:0.8-input_output_len:1024,1024-reqs:512] is in pre-merge and accounts for a bug chunk of our budget. Consider if checkpoint is one of the crucial perf tests, or should we focus on models we care more about.
2. Reduce test variants, specifically in Nemotron tests. Focus on testing the model registry config in pre-merge, and move other variants to post-merge.

Full report link: https://gitlab-master.nvidia.com/junq/my-notes/-/blob/main/l0-pre-merge-4wk-2026-05-13/l0-pre-merge-duration-analysis.md

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

[suyoggupta]

Let's add this as a task as part of the broader epic to improve CI for AutoDeploy. @hnover-nv

[MrGeva]

AutoDeploy tests in test_llm_api_autodeploy.py — pre_merge vs post_merge

Scope: all entries under tests/integration/test_lists/test-db/l0_*.yml that reference accuracy/test_llm_api_autodeploy.py. AD tests only appear in: l0_l40s.yml, l0_h100.yml, l0_b200.yml, l0_dgx_h100.yml, l0_dgx_b200.yml. No other test-db YAML (gb200/gb300/h200/dgx_b300/etc.) references this file.

Totals

Stage	Entries
pre_merge	54
post_merge	11
Total	65

Note: TestNemotronUltraV3::test_accuracy[nvfp4-4] is listed in both the 4-GPU pre_merge and 4-GPU post_merge sections of l0_dgx_b200.yml — this is a duplicate.

---

Pre-merge

l0_l40s.yml — 1× L40S

• TestNemotronNanoV3::test_accuracy[fp8-1-flashinfer]

l0_h100.yml — 1× H100 (backend=autodeploy)

• TestLlama3_1_8B::test_auto_dtype[trtllm-False-1]
• TestLlama3_1_8B::test_auto_dtype[trtllm-True-1]
• TestLlama3_1_8B::test_auto_dtype[triton_paged-False-1]
• TestLlama3_1_8B_Instruct_Eagle3::test_eagle3_one_model[flashinfer]
• TestLlama3_1_8B_Instruct_Eagle3::test_eagle3_one_model[trtllm]
• TestNemotronH::test_auto_dtype[trtllm-triton_ssm-False]
• TestNemotronH::test_auto_dtype[trtllm-flashinfer_ssm-False]
• TestNemotronH::test_auto_dtype[trtllm-triton_ssm-True]
• TestGLM4Flash::test_auto_dtype[trtllm-False]
• TestGLM4Flash::test_auto_dtype[trtllm-True]
• TestNemotronV2::test_auto_dtype[False]
• TestNemotronV2::test_auto_dtype[True]
• TestNemotronV2::test_fp8[True]
• TestNemotronNanoV3::test_accuracy[bf16-1-trtllm]
• TestNemotronNanoV3::test_accuracy[fp8-1-trtllm]

l0_b200.yml — 1× B100/B200 (backend=autodeploy)

• TestLlama3_1_8B::test_auto_dtype[trtllm-False-1]
• TestLlama3_1_8B::test_auto_dtype[triton_paged-False-1]
• TestLlama3_1_8B::test_auto_dtype[torch-True-1]
• TestGLM4Flash::test_auto_dtype[trtllm-False]
• TestGLM4Flash::test_nvfp4[False]
• TestNemotronV2::test_auto_dtype[False]
• TestNemotronV2::test_auto_dtype[True]
• TestNemotronV2::test_fp8[True]
• TestNemotronV2::test_nvfp4[True]
• TestNemotronNanoV3::test_accuracy[fp8-1-trtllm]
• TestNemotronNanoV3::test_accuracy[nvfp4-1-trtllm]
• TestNemotronSuperV3::test_accuracy[nvfp4-1-attn_dp_off-trtllm]
• TestNemotronSuperV3::test_functional_small[bf16]
• TestNemotronSuperV3::test_functional_small[fp8]

l0_dgx_h100.yml — 4× H100 (backend=autodeploy)

• TestLlama3_1_8B::test_auto_dtype[trtllm-False-4]
• TestLlama3_1_8B_Instruct_Eagle3::test_eagle3_one_model[flashinfer]
• TestLlama3_1_8B_Instruct_Eagle3::test_eagle3_one_model[trtllm]
• TestNemotronNanoV3::test_accuracy[bf16-4-trtllm]
• TestNemotronNanoV3::test_accuracy[fp8-4-trtllm]
• TestNemotronSuperV3::test_accuracy[bf16-4-attn_dp_off-trtllm]
• TestNemotronSuperV3::test_accuracy[bf16-4-attn_dp_on-trtllm]
• TestNemotronSuperV3::test_accuracy[fp8-4-attn_dp_off-trtllm]
• TestNemotronSuperV3::test_accuracy[fp8-4-attn_dp_on-trtllm]
• TestLlama3_1_8B::test_attention_dp[4]
• TestQwen3_5_397B_MoE::test_bf16_small[4]
• TestGemma4MoE::test_bf16
• TestMiniMaxM2::test_finegrained_fp8

l0_dgx_b200.yml — 4× B200 (backend=autodeploy)

• TestLlama3_1_8B::test_auto_dtype[trtllm-False-4]
• TestNemotronNanoV3::test_accuracy[fp8-4-trtllm]
• TestNemotronNanoV3::test_accuracy[nvfp4-4-trtllm]
• TestNemotronSuperV3::test_accuracy[bf16-4-attn_dp_off-trtllm]
• TestNemotronSuperV3::test_accuracy[fp8-4-attn_dp_off-trtllm]
• TestNemotronSuperV3::test_accuracy[fp8-4-attn_dp_on-trtllm]
• TestNemotronSuperV3::test_accuracy[nvfp4-4-attn_dp_on-trtllm]
• TestNemotronSuperV3::test_mtp[ws4_180gb-flashinfer]
• TestNemotronSuperV3::test_mtp[ws4_180gb-trtllm]
• TestNemotronUltraV3::test_accuracy[nvfp4-4]
• TestModelRegistryAccuracy::test_autodeploy_from_registry[nvidia_Llama-3.1-8B-Instruct-NVFP4-True]

---

Post-merge

l0_dgx_h100.yml — 4× H100 (backend=autodeploy)

• TestModelRegistryAccuracy::test_autodeploy_from_registry[google_gemma-3-1b-it-False]
• TestModelRegistryAccuracy::test_autodeploy_from_registry[meta-llama_Llama-3.1-8B-Instruct-False]
• TestModelRegistryAccuracy::test_autodeploy_from_registry[mistralai_Ministral-8B-Instruct-2410-False]
• TestModelRegistryAccuracy::test_autodeploy_from_registry[mistralai_Codestral-22B-v0.1-False]
• TestModelRegistryAccuracy::test_autodeploy_from_registry[Qwen_QwQ-32B-False]
• TestModelRegistryAccuracy::test_autodeploy_from_registry[meta-llama_Llama-3.3-70B-Instruct-False]
• TestModelRegistryAccuracy::test_autodeploy_from_registry[nvidia_Llama-3.1-8B-Instruct-FP8-True]

l0_dgx_b200.yml — 4× B200 (backend=autodeploy)

• TestNemotronUltraV3::test_accuracy[nvfp4-4] (also in pre_merge — duplicate)

l0_dgx_b200.yml — 8× B200 (backend=autodeploy)

• TestModelRegistryAccuracy::test_autodeploy_from_registry[deepseek-ai_DeepSeek-R1-0528-True]
• TestQwen3_5_397B_MoE::test_nvfp4[8]
• TestNemotronUltraV3::test_accuracy[nvfp4-8]

---

Per-class breakdown

Test class	Pre-merge	Post-merge
TestLlama3_1_8B	7	0
TestLlama3_1_8B_Instruct_Eagle3	4	0
TestNemotronH	3	0
TestNemotronV2	7	0
TestNemotronNanoV3	8	0
TestNemotronSuperV3	11	0
TestNemotronUltraV3	1	3 (incl. 1 duplicate of the pre-merge entry)
TestGLM4Flash	4	0
TestQwen3_5_397B_MoE	1	1
TestGemma4MoE	1	0
TestMiniMaxM2	1	0
TestModelRegistryAccuracy	1	8

Classes with no CI coverage in any test-db YAML

These classes are defined in test_llm_api_autodeploy.py but are not listed in any pre_merge or post_merge section:

• TestQwen3NextInstruct
• TestKimiK2_5
• TestGPTOSS
• TestNemotronSuperV3_IR
• TestQwen3_5_MoE_IR

Notable observations

1. TestModelRegistryAccuracy is almost entirely post-merge — except for the single Blackwell-NVFP4 entry (nvidia_Llama-3.1-8B-Instruct-NVFP4-True) in 4× B200 pre_merge.
2. The Nemotron Super V3 sweep dominates pre_merge on dgx_h100/dgx_b200 (8 of the 24 dgx-tier pre-merge entries).
3. One duplication exists — TestNemotronUltraV3::test_accuracy[nvfp4-4] in l0_dgx_b200.yml is listed under both the 4-GPU pre_merge section (line 353) and the 4-GPU post_merge section (line 372). Worth deduping.
4. Five test classes have no L0 coverage at all: the two IR-sharding classes and three model-specific classes (Qwen3-Next, Kimi-K2.5, GPT-OSS). They likely exist for ad-hoc/QA runs only.

[MrGeva]

Removed:

• perf/test_perf.py::test_perf[deepseek_r1_distill_qwen_32b-bench-_autodeploy-float16-kv_frac:0.8-input_output_len:1024,1024-reqs:512]
• perf/test_perf.py::test_perf[nemotron_nano_3_30b_fp8-bench-_autodeploy-float16-input_output_len:1024,1024-reqs:512]
  [#14173][chore] AutoDeploy: Removed perf tests from L0 #14220

[MrGeva]

@galagam Re:

Reduce test variants, specifically in Nemotron tests. Focus on testing the model registry config in pre-merge, and move other variants to post-merge.

Nemotron is not being tested currently using the model registry tests. please clarify

[galagam]

@galagam Re:

Reduce test variants, specifically in Nemotron tests. Focus on testing the model registry config in pre-merge, and move other variants to post-merge.

Nemotron is not being tested currently using the model registry tests. please clarify

@MrGeva
Constraints:

1. We Need to refactor somehow and reduce the load on pre-merge.
2. The most important config to validate for each model is the config listed in the registry

How we do it is open for debate. I suggest to run the registry config as-is in pre-merge and the other variations post-merge. This doesn't have to involve TestModelRegistryAccuracy, but it can - what ever makes more sense.
Another option, is to keep the Nemotron test classes, and have a single non-parameterized test that runs the registry config as-is.
We can probably omit the bf16 checkpoints from the pre-merge because they are not meant for deployment.
e.g. for pre-merge nano:
TestNemotronNanoV3::test_accuracy_registry[nvfp4-1]
TestNemotronNanoV3::test_accuracy_registry[fp8-1]
TestNemotronNanoV3::test_accuracy_registry[nvfp4-4]
TestNemotronNanoV3::test_accuracy_registry[fp8-4]

and in post merge we can run more variants, e.g.
TestNemotronUltraV3::test_accuracy[nvfp4-4-trtllm-attn_dp]

[galagam]

@MrGeva Maybe discuss with @hnover-nv before making big refactors - see #14173 (comment)