[TRTLLM-14473][chore] Remove legacy TensorRT-backend tests, examples, and CI plumbing#16610
Open
Wanli-Jiang wants to merge 1 commit into
Open
[TRTLLM-14473][chore] Remove legacy TensorRT-backend tests, examples, and CI plumbing#16610Wanli-Jiang wants to merge 1 commit into
Wanli-Jiang wants to merge 1 commit into
Conversation
Wanli-Jiang
commented
Jul 20, 2026
| } | ||
|
|
||
|
|
||
| def runPackageSanityCheck(pipeline, wheel_path, reinstall_dependencies=false, cpver="cp312") |
Collaborator
Author
There was a problem hiding this comment.
I am not sure whether we need to keep this function or not.
If need to keep, we might need to update the example runs.
Wanli-Jiang
force-pushed
the
user/williamj/trtllm-14027-tests-examples
branch
from
July 20, 2026 09:01
d3a833e to
6669d31
Compare
Wanli-Jiang
marked this pull request as ready for review
July 20, 2026 09:03
Wanli-Jiang
requested review from
BowenFu,
LarryXFly,
QiJune,
kaiyux,
laikhtewari,
taylor-yb-lee,
xinhe-nv,
yiqingy0 and
yuanjingx87
July 20, 2026 09:03
Collaborator
Author
|
/bot run --disable-fail-fast |
Collaborator
|
@coderabbitai check tests/unittest/llmapi/apps, if there is any test that still uses tensorrt backend |
xinhe-nv
approved these changes
Jul 20, 2026
Collaborator
|
PR_Github #60335 [ run ] triggered by Bot. Commit: |
Contributor
📝 WalkthroughWalkthroughThis PR removes numerous legacy examples, plugins, model workflows, dependencies, and integration helpers. It updates CI backend selection, shifts test matrices toward PyTorch and LLM API coverage, adds a skipped TP2 accuracy test, and replaces selected documentation and waiver entries. ChangesLegacy workflow and documentation cleanup
CI and runtime changes
Integration and test configuration
Estimated code review effort: 4 (Complex) | ~60 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
BowenFu
approved these changes
Jul 20, 2026
BowenFu
left a comment
BowenFu
left a comment
There was a problem hiding this comment.
LGTM — reviewed for over-deletion. Nothing the live PyTorch/AutoDeploy path or any scheduled test/CI stage depends on is removed: _utils.py's supports_inflight_batching is a TRT-engine-only helper with zero remaining importers (former callers all deleted here or in #16369), and the 42 dropped waives all target deleted example defs / de-scheduled TRT perf configs — no active pytorch/llmapi/disagg/moe waive was dropped, so the validator invariant holds.
Two non-blocking notes (dead code, not breakage):
runPackageSanityCheckis not actually deleted despite the PR description — it still lives inL0_Test.groovy(~:4373) and references deletedgenerate_checkpoint_config.py/run.py. Its call site is commented out so it's dead/unreachable, but the description is inaccurate.- The retained
CliFlowAccuracyTestHarnessnow dangles on deletedsummarize.py/eval_long_context.py(it's unscheduled —test_cli_flowis in no active list — so nothing breaks, but the harness is non-functional). Minor doc link rot in accuracy README + speculative-decoding.md too.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
tests/integration/defs/examples/test_phi.py (1)
54-56: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winProfiler key doesn't match the test name.
ci_profiler.start/stopis keyed on"test_llm_torch_multi_lora_support"(the shared helper's name) rather than this test's own name, unlike the analogous new test intest_gpt.pywhich profiles under its own test name. If another Phi (or future) test reuses this helper, profiling data will collide under the same key.🔧 Proposed fix
- defs.ci_profiler.start("test_llm_torch_multi_lora_support") + defs.ci_profiler.start("test_phi_4_mini_instruct_with_bf16_lora_torch") test_llm_torch_multi_lora_support(hf_model_dir=llm_phi_model_root, llm_venv=llm_venv, num_loras=2, lora_rank=8, target_hf_modules=["qkv_proj"], target_trtllm_modules=["attn_qkv"], zero_lora_weights=True, tensor_parallel_size=1) - defs.ci_profiler.stop("test_llm_torch_multi_lora_support") + defs.ci_profiler.stop("test_phi_4_mini_instruct_with_bf16_lora_torch")🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/integration/defs/examples/test_phi.py` around lines 54 - 56, Update the profiler key passed to ci_profiler.start in the Phi test to use that test's own function name rather than the shared test_llm_torch_multi_lora_support helper name, and ensure the corresponding stop call uses the same key.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/source/features/auto_deploy/advanced/testing_strategy.md`:
- Around line 120-121: Update the testing strategy example around the documented
registration format so the modeling_llama entry is either a concrete fully
qualified test node ID or is explicitly identified as a command-style selector
supported by CI YAML. Keep the existing individual test-case example unchanged
and ensure the examples match the format described in the surrounding
documentation.
In `@tests/integration/defs/accuracy/test_llm_api_pytorch.py`:
- Around line 4385-4396: Remove the unconditional pytest.skip from test_tp2, or
replace it with a topology-aware skip that allows execution on validated NVLink
platforms while excluding unsupported PCIe-only topologies. Also remove the
test_tp2 entry from tests/integration/test_lists/qa/llm_function_core.txt until
the test is executable in QA.
---
Outside diff comments:
In `@tests/integration/defs/examples/test_phi.py`:
- Around line 54-56: Update the profiler key passed to ci_profiler.start in the
Phi test to use that test's own function name rather than the shared
test_llm_torch_multi_lora_support helper name, and ensure the corresponding stop
call uses the same key.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: b0e0665b-32db-490b-9b84-1f1c9fa2816c
⛔ Files ignored due to path filters (3)
examples/models/core/qwen2audio/audio/glass-breaking-151256.mp3is excluded by!**/*.mp3examples/models/core/qwenvl/pics/1.pngis excluded by!**/*.pngexamples/models/core/qwenvl/pics/demo.jpegis excluded by!**/*.jpeg
📒 Files selected for processing (149)
docs/source/developer-guide/ci-overview.mddocs/source/features/auto_deploy/advanced/testing_strategy.mdexamples/cpp_library/CMakeLists.txtexamples/cpp_library/build.shexamples/cpp_library/main.cppexamples/cpp_library/tensorrt_llm_libutils.hexamples/dora/README.mdexamples/dora/normalize_weights.pyexamples/eval_long_context.pyexamples/generate_checkpoint_config.pyexamples/generate_xgrammar_tokenizer_info.pyexamples/hf_lora_convert.pyexamples/language_adapter/README.mdexamples/llm-eval/lm-eval-harness/README.mdexamples/llm-eval/lm-eval-harness/lm_eval_tensorrt_llm.pyexamples/llm-eval/lm-eval-harness/requirements.txtexamples/mmlu.pyexamples/models/contrib/arctic/README.mdexamples/models/contrib/blip2/README.mdexamples/models/contrib/chatglm-6b/README.mdexamples/models/contrib/chatglm-6b/requirements.txtexamples/models/contrib/chatglm-6b/tokenization_chatglm.pyexamples/models/contrib/chatglm2-6b/README.mdexamples/models/contrib/chatglm2-6b/requirements.txtexamples/models/contrib/chatglm2-6b/tokenization_chatglm.pyexamples/models/contrib/chatglm3-6b-32k/README.mdexamples/models/contrib/chatglm3-6b-32k/requirements.txtexamples/models/contrib/chatglm3-6b-32k/tokenization_chatglm.pyexamples/models/contrib/internlm/.gitignoreexamples/models/contrib/internlm/README.mdexamples/models/contrib/internlm/requirements.txtexamples/models/contrib/jais/README.mdexamples/models/contrib/jais/requirements.txtexamples/models/contrib/sdxl/README.mdexamples/models/contrib/sdxl/build_sdxl_unet.pyexamples/models/contrib/sdxl/pipeline_stable_diffusion_xl.pyexamples/models/contrib/sdxl/run_sdxl.pyexamples/models/contrib/skywork/README.mdexamples/models/contrib/skywork/requirements.txtexamples/models/contrib/smaug/README.mdexamples/models/contrib/smaug/requirements.txtexamples/models/core/granite/README.mdexamples/models/core/mixtral/README.mdexamples/models/core/mixtral/requirements.txtexamples/models/core/multimodal/README.mdexamples/models/core/multimodal/__init__.pyexamples/models/core/multimodal/build_multimodal_engine.pyexamples/models/core/multimodal/eval.pyexamples/models/core/multimodal/requirements-eclair.txtexamples/models/core/multimodal/requirements-internlm-xcomposer2.txtexamples/models/core/multimodal/requirements-llava_onevision.txtexamples/models/core/multimodal/requirements-qwen2vl.txtexamples/models/core/multimodal/requirements-vila.txtexamples/models/core/multimodal/run.pyexamples/models/core/multimodal/utils.pyexamples/models/core/nemotron/README_nemotron-3.mdexamples/models/core/nemotron/requirements.txtexamples/models/core/qwen2audio/README.mdexamples/models/core/qwen2audio/requirements.txtexamples/models/core/qwen2audio/run.pyexamples/models/core/qwen2audio/run_chat.pyexamples/models/core/qwen2audio/utils.pyexamples/models/core/qwenvl/README.mdexamples/models/core/qwenvl/requirements.txtexamples/models/core/qwenvl/run.pyexamples/models/core/qwenvl/run_chat.pyexamples/models/core/qwenvl/show_pic.pyexamples/models/core/qwenvl/vit_onnx_trt.pyexamples/ngram/README.mdexamples/ngram/run_dtm_ngram.pyexamples/openai_triton/README.mdexamples/openai_triton/manual_plugin/CMakeLists.txtexamples/openai_triton/manual_plugin/README.mdexamples/openai_triton/manual_plugin/TritonFlashAttentionPlugin.cppexamples/openai_triton/manual_plugin/TritonFlashAttentionPlugin.hexamples/openai_triton/manual_plugin/build.pyexamples/openai_triton/manual_plugin/fmha_triton.pyexamples/openai_triton/manual_plugin/plugin.pyexamples/openai_triton/manual_plugin/run.pyexamples/openai_triton/manual_plugin/tritonPlugins.cppexamples/openai_triton/plugin_autogen/README.mdexamples/openai_triton/plugin_autogen/build_engine.pyexamples/openai_triton/plugin_autogen/kernel_config.pyexamples/openai_triton/plugin_autogen/run_engine.pyexamples/python_plugin/README.mdexamples/python_plugin/build_lookup.pyexamples/python_plugin/plugin_lib/__init__.pyexamples/python_plugin/plugin_lib/lookup_kernel.pyexamples/python_plugin/plugin_lib/lookup_plugin.pyexamples/python_plugin/run_lookup.pyexamples/run.pyexamples/sample_weight_stripping/README.mdexamples/summarize.pyexamples/utils.pyjenkins/L0_Test.groovyjenkins/scripts/cbts/blocks.pytensorrt_llm/_utils.pytensorrt_llm/models/automodel.pytests/integration/defs/accuracy/test_llm_api_pytorch.pytests/integration/defs/common.pytests/integration/defs/conftest.pytests/integration/defs/examples/run_llm_fp8_quant_llama_70b.pytests/integration/defs/examples/test_bert.pytests/integration/defs/examples/test_bindings.pytests/integration/defs/examples/test_chatglm.pytests/integration/defs/examples/test_commandr.pytests/integration/defs/examples/test_draft_target_model.pytests/integration/defs/examples/test_eagle.pytests/integration/defs/examples/test_enc_dec.pytests/integration/defs/examples/test_flux.pytests/integration/defs/examples/test_gpt.pytests/integration/defs/examples/test_gptj.pytests/integration/defs/examples/test_granite.pytests/integration/defs/examples/test_internlm.pytests/integration/defs/examples/test_llama.pytests/integration/defs/examples/test_mamba.pytests/integration/defs/examples/test_medusa.pytests/integration/defs/examples/test_mistral.pytests/integration/defs/examples/test_mixtral.pytests/integration/defs/examples/test_multimodal.pytests/integration/defs/examples/test_nemotron.pytests/integration/defs/examples/test_nemotron_nas.pytests/integration/defs/examples/test_ngram.pytests/integration/defs/examples/test_openai.pytests/integration/defs/examples/test_phi.pytests/integration/defs/examples/test_qwen.pytests/integration/defs/examples/test_qwen2audio.pytests/integration/defs/examples/test_qwenvl.pytests/integration/defs/examples/test_redrafter.pytests/integration/defs/examples/test_whisper.pytests/integration/defs/examples/validate_whisper_beam_logits.pytests/integration/defs/examples/validate_whisper_log_probs_determinism.pytests/integration/defs/perf/build.pytests/integration/defs/perf/test_perf.pytests/integration/defs/test_e2e.pytests/integration/defs/triton_server/conftest.pytests/integration/test_lists/qa/README.mdtests/integration/test_lists/qa/llm_function_core.txttests/integration/test_lists/test-db/README.mdtests/integration/test_lists/test-db/l0_a10.ymltests/integration/test_lists/test-db/l0_a100.ymltests/integration/test_lists/test-db/l0_a30.ymltests/integration/test_lists/test-db/l0_b200.ymltests/integration/test_lists/test-db/l0_dgx_h200.ymltests/integration/test_lists/test-db/l0_gb203.ymltests/integration/test_lists/test-db/l0_gh200.ymltests/integration/test_lists/test-db/l0_h100.ymltests/integration/test_lists/test-db/l0_l40s.ymltests/integration/test_lists/waives.txt
💤 Files with no reviewable changes (111)
- examples/openai_triton/README.md
- examples/llm-eval/lm-eval-harness/requirements.txt
- examples/openai_triton/manual_plugin/README.md
- examples/llm-eval/lm-eval-harness/README.md
- examples/models/contrib/chatglm3-6b-32k/requirements.txt
- tests/integration/defs/examples/test_bindings.py
- examples/models/contrib/internlm/requirements.txt
- tests/integration/defs/examples/test_gptj.py
- examples/models/contrib/skywork/README.md
- examples/python_plugin/plugin_lib/lookup_kernel.py
- examples/openai_triton/plugin_autogen/README.md
- examples/models/core/multimodal/build_multimodal_engine.py
- examples/models/contrib/jais/README.md
- examples/models/contrib/jais/requirements.txt
- examples/models/core/mixtral/README.md
- examples/models/core/qwenvl/requirements.txt
- examples/cpp_library/CMakeLists.txt
- examples/models/contrib/chatglm-6b/requirements.txt
- examples/models/core/mixtral/requirements.txt
- examples/models/contrib/chatglm2-6b/requirements.txt
- examples/models/core/multimodal/requirements-vila.txt
- examples/models/contrib/chatglm2-6b/README.md
- examples/cpp_library/build.sh
- examples/models/core/multimodal/requirements-eclair.txt
- examples/models/core/nemotron/README_nemotron-3.md
- examples/models/contrib/chatglm3-6b-32k/README.md
- examples/openai_triton/plugin_autogen/kernel_config.py
- examples/models/contrib/smaug/README.md
- examples/dora/README.md
- examples/models/contrib/internlm/README.md
- examples/models/contrib/chatglm2-6b/tokenization_chatglm.py
- examples/python_plugin/plugin_lib/init.py
- examples/models/core/qwenvl/README.md
- examples/models/core/multimodal/requirements-internlm-xcomposer2.txt
- examples/models/contrib/chatglm-6b/README.md
- tests/integration/defs/examples/test_bert.py
- examples/generate_xgrammar_tokenizer_info.py
- examples/models/core/qwen2audio/README.md
- examples/models/core/granite/README.md
- examples/sample_weight_stripping/README.md
- examples/models/core/multimodal/utils.py
- tests/integration/defs/examples/test_flux.py
- examples/models/core/qwen2audio/requirements.txt
- tests/integration/defs/examples/run_llm_fp8_quant_llama_70b.py
- examples/models/contrib/sdxl/build_sdxl_unet.py
- examples/openai_triton/manual_plugin/build.py
- examples/models/core/qwen2audio/run_chat.py
- examples/openai_triton/manual_plugin/CMakeLists.txt
- examples/models/contrib/internlm/.gitignore
- tests/integration/defs/examples/test_medusa.py
- examples/models/core/qwenvl/vit_onnx_trt.py
- examples/python_plugin/run_lookup.py
- examples/openai_triton/manual_plugin/tritonPlugins.cpp
- examples/models/contrib/arctic/README.md
- examples/models/contrib/sdxl/README.md
- tests/integration/defs/examples/test_chatglm.py
- examples/models/contrib/sdxl/run_sdxl.py
- examples/language_adapter/README.md
- tests/integration/defs/examples/test_eagle.py
- examples/models/contrib/skywork/requirements.txt
- examples/models/contrib/chatglm-6b/tokenization_chatglm.py
- examples/ngram/run_dtm_ngram.py
- examples/models/core/qwenvl/show_pic.py
- examples/models/core/nemotron/requirements.txt
- examples/models/core/multimodal/requirements-llava_onevision.txt
- examples/models/core/qwenvl/run_chat.py
- examples/cpp_library/tensorrt_llm_libutils.h
- examples/python_plugin/build_lookup.py
- examples/models/core/qwen2audio/utils.py
- examples/openai_triton/plugin_autogen/run_engine.py
- examples/python_plugin/README.md
- examples/generate_checkpoint_config.py
- tests/integration/defs/examples/test_commandr.py
- examples/run.py
- examples/models/core/multimodal/run.py
- examples/openai_triton/manual_plugin/fmha_triton.py
- examples/hf_lora_convert.py
- examples/models/contrib/chatglm3-6b-32k/tokenization_chatglm.py
- examples/models/core/multimodal/eval.py
- tests/integration/defs/examples/test_mamba.py
- examples/mmlu.py
- examples/cpp_library/main.cpp
- examples/eval_long_context.py
- examples/models/core/multimodal/requirements-qwen2vl.txt
- examples/models/contrib/sdxl/pipeline_stable_diffusion_xl.py
- tests/integration/defs/examples/test_granite.py
- examples/models/contrib/blip2/README.md
- tests/integration/defs/examples/test_internlm.py
- examples/openai_triton/manual_plugin/TritonFlashAttentionPlugin.cpp
- examples/openai_triton/manual_plugin/run.py
- examples/dora/normalize_weights.py
- examples/models/contrib/smaug/requirements.txt
- tests/integration/defs/test_e2e.py
- tests/integration/defs/examples/test_enc_dec.py
- examples/openai_triton/manual_plugin/plugin.py
- examples/utils.py
- examples/openai_triton/plugin_autogen/build_engine.py
- tests/integration/defs/examples/test_llama.py
- examples/summarize.py
- examples/llm-eval/lm-eval-harness/lm_eval_tensorrt_llm.py
- examples/models/core/qwen2audio/run.py
- tests/integration/test_lists/test-db/l0_a30.yml
- examples/openai_triton/manual_plugin/TritonFlashAttentionPlugin.h
- jenkins/scripts/cbts/blocks.py
- tests/integration/defs/examples/test_draft_target_model.py
- examples/models/core/qwenvl/run.py
- examples/python_plugin/plugin_lib/lookup_plugin.py
- tests/integration/test_lists/test-db/l0_dgx_h200.yml
- tests/integration/test_lists/waives.txt
- tests/integration/defs/conftest.py
- tests/integration/defs/triton_server/conftest.py
Comment on lines
+120
to
+121
| - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8B::test_auto_dtype | ||
| - unittest/_torch/modeling -k "modeling_llama" |
Contributor
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Align the example with the documented registration format.
Line 121 is a directory-level -k selector, not an individual fully qualified test case as described in Lines 113-115. Either use a concrete node ID or explicitly document that CI YAML also accepts command-style selectors.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/source/features/auto_deploy/advanced/testing_strategy.md` around lines
120 - 121, Update the testing strategy example around the documented
registration format so the modeling_llama entry is either a concrete fully
qualified test node ID or is explicitly identified as a command-style selector
supported by CI YAML. Keep the existing individual test-case example unchanged
and ensure the examples match the format described in the surrounding
documentation.
Comment on lines
+4385
to
+4396
| @pytest.mark.skip( | ||
| reason="TP2 hangs in the AUTO custom allreduce on PCIe-only (all-SYS " | ||
| "topology) nodes; the LMHead AllReduce ignores allreduce_strategy and " | ||
| "always takes the AUTO path (tunable_allreduce), so the strategy knob " | ||
| "cannot work around it. Unskip once validated on an NVLink platform " | ||
| "or the lm_head strategy plumbing is fixed.") | ||
| @pytest.mark.skip_less_device(2) | ||
| def test_tp2(self): | ||
| with LLM(self.MODEL_PATH, tensor_parallel_size=2) as llm: | ||
| task = CnnDailymail(self.MODEL_NAME) | ||
| task.evaluate(llm, | ||
| extra_evaluator_kwargs=self.EXTRA_EVALUATOR_KWARGS) |
Contributor
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Do not schedule an unconditionally skipped TP2 test. pytest.mark.skip prevents this test from providing any QA coverage, despite its new QA-list entry. Remove it from the list until it can run, or replace the unconditional skip with a topology-aware condition and validate it on a supported NVLink platform.
tests/integration/defs/accuracy/test_llm_api_pytorch.py#L4385-L4396: make the skip conditional on unsupported topology, or retain this as an unlisted placeholder.tests/integration/test_lists/qa/llm_function_core.txt#L697-L697: remove the entry until the test is executable in QA.
📍 Affects 2 files
tests/integration/defs/accuracy/test_llm_api_pytorch.py#L4385-L4396(this comment)tests/integration/test_lists/qa/llm_function_core.txt#L697-L697
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/integration/defs/accuracy/test_llm_api_pytorch.py` around lines 4385 -
4396, Remove the unconditional pytest.skip from test_tp2, or replace it with a
topology-aware skip that allows execution on validated NVLink platforms while
excluding unsupported PCIe-only topologies. Also remove the test_tp2 entry from
tests/integration/test_lists/qa/llm_function_core.txt until the test is
executable in QA.
Source: Path instructions
Contributor
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
scope="tests/unittest/llmapi/apps"
echo "== Files in scope =="
fd --type f . "$scope" | sort
echo
echo "== Structural outline =="
ast-grep outline "$scope" --items all --type class,function --view compact || true
echo
echo "== TensorRT/backend-related occurrences with context =="
rg -n -i -C 4 \
-e '\btensorrt\b' \
-e '\btrtllm\b' \
-e '\btrt_llm\b' \
-e '\bbackend\s*=' \
-e '\bbackend\s*:' \
-e '\bbackend\b' \
"$scope" || true
echo
echo "== Python imports/calls that may configure backend =="
ast-grep run --lang python --pattern 'LLM($$$)' "$scope" || true
ast-grep run --lang python --pattern '$OBJ = LLM($$$)' "$scope" || trueLength of output: 66941 🏁 Script executed: #!/bin/bash
set -euo pipefail
scope="tests/unittest/llmapi/apps"
echo "== Exact/near-exact legacy backend tokens in Python source =="
python3 - <<'PY'
import ast
from pathlib import Path
root = Path("tests/unittest/llmapi/apps")
for path in sorted(root.rglob("*.py")):
try:
tree = ast.parse(path.read_text())
except SyntaxError as exc:
print(f"{path}:{exc.lineno}: syntax error: {exc}")
continue
for node in ast.walk(tree):
if isinstance(node, ast.Constant) and isinstance(node.value, str):
value = node.value.lower()
if "tensorrt" in value or value in {"trt", "trtllm"}:
print(f"{path}:{node.lineno}: {node.value!r}")
PY
echo
echo "== All backend fixture parameterizations and server CLI construction =="
rg -n -i -C 3 \
-e '`@pytest`\.fixture\([^)]*params\s*=' \
-e 'pytest\.mark\.parametrize\([^)]*backend' \
-e '["'\'']--backend["'\'']' \
-e 'backend\s*==|backend\s*!=' \
"$scope" -g '*.py' || true
echo
echo "== Parent-level pytest fixtures/configuration that could supply a backend =="
fd --type f --glob 'conftest.py' tests/unittest/llmapi | sort | while read -r file; do
echo "--- $file"
rg -n -i -C 3 -e '\btensorrt\b' -e '\bbackend\b' "$file" || true
done
echo
echo "== Focused declarations for backend fixture parameters =="
rg -n -i -U -C 2 \
'`@pytest`\.fixture\([\s\S]{0,250}?params\s*=\s*\[[^\]]+\][\s\S]{0,150}?def backend\b' \
"$scope" -g '*.py' || trueLength of output: 34786
All local The only stale backend-related item is a comment in |
StanleySun639
approved these changes
Jul 20, 2026
Collaborator
|
PR_Github #60335 [ run ] completed with state
|
… and CI plumbing Trailing test/example/CI cleanup after the legacy TensorRT backend removal (TRTLLM-14026, PR NVIDIA#16369). CI schedules / jenkins: - Move backend-neutral tests off the retired -TensorRT- stages and delete all dead backend=tensorrt test-db buckets; flip remaining neutral buckets to pytorch so parked coverage runs again. - NEW stages A100X-PyTorch-Post-Merge-1 and L40S-PyTorch-Post-Merge-1 schedule the restored a100/l40s post-merge coverage (CI-owner review). - Retire the -TensorRT- plumbing in L0_Test.groovy (commented stage rows, backend=tensorrt mako branch, TEST_BACKEND changeMap entry, excludedBackends) and the -TensorRT- pattern in jenkins/scripts/cbts/blocks.py. - waives.txt: drop 42 waives for deleted/unscheduled tests (the validator requires waived tests to be in an active list). Legacy engine-flow example tests: - Delete the defs/examples wrappers whose every test drove the removed convert->trtllm-build->run flow: test_{bert,chatglm,commandr, draft_target_model,eagle,enc_dec,gptj,granite,internlm,llama,mamba, medusa,mistral,mixtral,multimodal,nemotron,nemotron_nas,ngram, openai,qwen,qwen2audio,qwenvl,redrafter,whisper,bindings,flux}.py, the whisper validators, and run_llm_fp8_quant_llama_70b.py. - test_gpt.py rewritten keeping only the live test_gpt_oss_20b_lora_torch; test_phi.py drops its quantization engine test; test_e2e.py drops test_gpt3_175b_1layers_build_only. - Remove the dead helper cascade in defs/common.py (convert_weights, quantize_data, prune_checkpoint, refit_model, find_tensorrt), defs/perf/build.py and test_perf.py's trtllm-build branches, and 19 orphaned example_root fixtures across defs/conftest.py and triton_server/conftest.py (verified: no dynamic getfixturevalue resolution exists for any of them). Dormant example scripts (imported deleted ModelRunner*/builder surfaces or drove engine flows; unreachable from CI): - examples/{run,summarize,utils,mmlu,eval_long_context}.py, examples/openai_triton/**, examples/python_plugin/**, examples/models/contrib/sdxl/**, examples/llm-eval/lm-eval-harness, the engine-flow multimodal/qwenvl/qwen2audio scripts, and examples/ngram/run_dtm_ngram.py (READMEs rewritten for the PyTorch flow where a live section remains). examples/ reference sweep (checked against tests, published docs, jenkins, and code): - contrib: delete arctic, blip2, chatglm-6b/2-6b/3-6b-32k, internlm, jais, skywork, smaug (README/requirements-only legacy-flow dirs; hyperclovax stays - live PyTorch README linked from release notes). - core: delete granite and mixtral; multimodal keeps the live Qwen-Image-Bench stub + eval script but drops utils.py, __init__.py and the per-model requirements-*.txt; nemotron drops the Nemo-flow README_nemotron-3.md + requirements.txt (PyTorch READMEs stay). - top level: delete generate_checkpoint_config.py, generate_xgrammar_tokenizer_info.py, hf_lora_convert.py, sample_weight_stripping/, cpp_library/, dora/, language_adapter/. - Kept deliberately: infinitebench (dependency of the retained CLI-flow suite) and longbench/opentelemetry/sparse_attention (unreferenced but document current features). Accuracy suite: - accuracy/test_cli_flow.py and CliFlowAccuracyTestHarness are KEPT (currently unscheduled; retained for possible future reuse). - Port TestQwen2_7BInstruct::test_tp2 to the PyTorch accuracy suite (skipped: TP2 hangs in the AUTO custom allreduce on PCIe-only nodes; the LMHead AllReduce ignores allreduce_strategy, so the strategy knob cannot work around it). Misc: - automodel.py isMedusa/isEagle branch and _utils.py::supports_inflight_batching (mapped to deleted TRT model classes); test-db README example refreshed to a real test id; docs fixes in ci-overview.md and the AutoDeploy testing strategy. legacy-files.txt/ruff baseline entries for the deleted files are left to a mechanical 'legacy_utils.py prune-files' pass after this and the python-cleanup PR merge, to avoid cross-PR conflicts. Verified: defs collection passes (4206 tests), pre-commit hooks incl. the test-list AST validator pass, no broken example links. Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
Wanli-Jiang
force-pushed
the
user/williamj/trtllm-14027-tests-examples
branch
from
July 21, 2026 03:31
6669d31 to
1913f7a
Compare
Collaborator
Author
|
/bot run --disable-fail-fast |
Collaborator
|
PR_Github #60561 [ run ] triggered by Bot. Commit: |
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Features
Why this PR exists
Phase C removed the TensorRT engine backend itself (
trtllm-build,convert_checkpoint.pyflows,ModelRunner*,tensorrt_llm.builder).This PR removes the test wrappers, example scripts, and CI plumbing that
existed only to exercise that removed backend. Everything deleted here
falls into one of two buckets:
no longer exists (
trtllm-build,examples/run.py, deleted modelclasses), or
backend: tensorrttest-db buckets /-TensorRT-CI stages retiredhere, or referenced by nothing at all (verified by reference sweep).
Change groups
1. CI schedules / jenkins (CI-owner review requested)
-TensorRT-stages-TensorRT-Post-Mergestage rows, thebackend=tensorrtmako branch, theTEST_BACKENDchangeMap entry, andexcludedBackendsmentions removed fromjenkins/L0_Test.groovy; the("-TensorRT-", "tensorrt")pattern dropped fromjenkins/scripts/cbts/blocks.py(kept in sync with the groovy map, hence in this PR, not the infra PR)backend: tensorrtbucketsl0_a30.yml(−33),l0_dgx_h200.yml(−21), plus legacy subsets inl0_a10/h100/l40spytorchso parked coverage runs again (l0_a100,l0_b200,l0_dgx_h2004-GPU,l0_l40s)A100X-PyTorch-Post-Merge-1andL40S-PyTorch-Post-Merge-1schedule the restored a100/l40s post-merge coverage — this is the main thing CI owners should eyeballNote: the dead
runPackageSanityCheckwheel-check removal is not inthis PR — it moved to the infra PR (TRTLLM-14027) since it is package
infra, uncoupled from test schedules.
2. Legacy engine-flow integration tests (~14k lines)
tests/integration/defs/examples/wrappers — every test inthem drove the removed convert →
trtllm-build→ run flow:test_{bert, chatglm, commandr, draft_target_model, eagle, enc_dec, gptj, granite, internlm, llama, mamba, medusa, mistral, mixtral, multimodal, nemotron, nemotron_nas, ngram, openai, qwen, qwen2audio, qwenvl, redrafter, whisper, bindings, flux}.py, the two whispervalidators, and
run_llm_fp8_quant_llama_70b.py.test_gpt.py(740→84 lines, keeps the livetest_gpt_oss_20b_lora_torch),test_phi.py(drops the quantizationengine test),
test_e2e.py(dropstest_gpt3_175b_1layers_build_only—unscheduled and invokes
trtllm-build).defs/common.py:convert_weights,quantize_data,prune_checkpoint,refit_model,find_tensorrt—last consumers were the wrappers above.
defs/perf/build.py(1,524lines) and
test_perf.py'strtllm-build/build.pybranches removed;the non-
trtllm-benchbuild-script legs now raiseRuntimeErrorso afuture misconfiguration fails loudly.
*_example_rootfixtures removed acrossdefs/conftest.pyandtriton_server/conftest.py. Verified: zerousers, and no dynamic
request.getfixturevalue(f"...")resolutionexists anywhere in defs that could reach them.
3. Dormant example scripts (~6k lines)
All imported deleted
ModelRunner*/builder surfaces or drove engineflows, and were unreachable from CI:
examples/{run, summarize, utils, mmlu, eval_long_context}.pyexamples/openai_triton/**(TRT plugin demos),examples/python_plugin/**examples/models/contrib/sdxl/**,examples/llm-eval/lm-eval-harness/**build_multimodal_engine.py,run.py,eval.py),qwenvl/**,qwen2audio/**examples/ngram/run_dtm_ngram.py(READMEs rewritten for the PyTorchflow where a live section remains)
4. examples/ reference sweep (~4.5k lines)
Every path checked against tests, published docs, jenkins, and code
(three independent scan agents + manual verification; the parked CLI-flow
suite deliberately not counted as a consumer):
arctic,blip2(redirect stub),chatglm-6b/2-6b/3-6b-32k,internlm,jais,skywork,smaug—README/requirements-only legacy-flow dirs, zero references.
hyperclovaxstays (live PyTorch README linked from release notes).granite/andmixtral/deleted (legacy-flow READMEs only);multimodal/keeps the live Qwen-Image-Bench stub +qwen_image_bench_eval.py(used by live unittests) but dropsutils.py,__init__.py, and 6 per-modelrequirements-*.txt;nemotron/drops the Nemo-flowREADME_nemotron-3.md+requirements.txt(the three current PyTorch READMEs stay).generate_checkpoint_config.py(fedtrtllm-build),generate_xgrammar_tokenizer_info.py(unreferenced; the live pathcalls
_xgrammar_tokenizer_info()directly),hf_lora_convert.py(produced the removed
.npyLoRA format),sample_weight_stripping/,cpp_library/,dora/,language_adapter/.5. Small code prunes
tensorrt_llm/models/automodel.py: thedecoding_modeisMedusa/isEagle branch — it mapped to
MedusaForCausalLM/EagleForCausalLMTRT model classes deleted in Phase B, so the branchcould only raise; no caller passes
decoding_mode. PyTorch spec-decdoes not use this path.
tensorrt_llm/_utils.py::supports_inflight_batching— same story.ci-overview.md, AutoDeploytesting_strategy.md,test-db README example refreshed to a real test id.
6. Additions (the +110)
TestQwen2_7BInstruct::test_tp2ported to the PyTorch accuracy suite(
test_llm_api_pytorch.py) + itsqa/llm_function_core.txtentry.Currently skipped with reason: TP2 hangs in the AUTO custom
allreduce on PCIe-only (all-SYS topology) nodes, and the LMHead
AllReduce ignores
allreduce_strategy, so the knob cannot work aroundit. Unskip once validated on NVLink or the lm_head strategy plumbing is
fixed. (Two nvbug filings pending for the underlying defects.)
Deliberately KEPT
accuracy/test_cli_flow.py+CliFlowAccuracyTestHarnessexamples/infinitebench/examples/{longbench, opentelemetry, sparse_attention}examples/models/contrib/hyperclovax/benchmarks/prepare_dataset.pyhyperclovax,exaone)Validation
tests/integration/defscollects 4,225 tests on therebased base, zero errors (was 4,207 pre-sweep; −1 deleted dead test,
+19 upstream additions).
validator and the test-db config checks.
(TRTLLM-14027), python-cleanup (TRTLLM-14474), and cpp-cleanup
(TRTLLM-14475) branches onto
origin/mainis conflict-free and wasverified to produce the intended combined tree.
origin/mainb8604c46ce; thewaives.txtconflict was resolved as upstream'sversion minus exactly our 42 deletions (our change verified to be
pure deletion).
Known follow-ups (not in this PR)
legacy-files.txt/ ruff-baseline entries for files deleted here gostale — a mechanical
scripts/legacy_utils.py prune-files+gen-configspass runs after this and the python-cleanup PR merge(kept out of both to avoid cross-PR conflicts).
test_tp2investigation: (1) AUTO customallreduce deadlock on all-SYS PCIe topology; (2) LMHead/Linear
AllReduce constructed without
model_config.allreduce_strategy.remaining
trtllm-buildREADME sections.Summary by CodeRabbit
Documentation
Removed Features
Tests
Description
Test Coverage
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.