[https://nvbugs/5989920][test] Unwaive DeepSeekV3 nvfp4 mtp3_fp8kv_chunked test by yizhang-nv · Pull Request #12533 · NVIDIA/TensorRT-LLM

yizhang-nv · 2026-03-25T09:08:07Z

Summary by CodeRabbit

Tests
- Re-enabled a previously skipped test case, indicating the underlying issue has been resolved and test coverage is being expanded.

Description

Unwaive TestDeepSeekV32::test_nvfp4_multi_gpus_piecewise_cuda_graph[mtp3_fp8kv_chunked] (nvbugs/5989920) to re-enable it in CI.

Test Coverage

The unwaived test itself: accuracy/test_llm_api_pytorch.py::TestDeepSeekV32::test_nvfp4_multi_gpus_piecewise_cuda_graph[mtp3_fp8kv_chunked]

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

yizhang-nv · 2026-03-25T09:08:15Z

/bot run

tensorrt-cicd · 2026-03-25T09:14:03Z

PR_Github #40306 [ run ] triggered by Bot. Commit: dc1cf29 Link to invocation

tensorrt-cicd · 2026-03-25T16:50:31Z

PR_Github #40306 [ run ] completed with state SUCCESS. Commit: dc1cf29
/LLM/main/L0_MergeRequest_PR pipeline #31417 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

yizhang-nv · 2026-03-30T02:40:01Z

/bot run --only-multi-gpu-test --disable-fail-fast

coderabbitai · 2026-03-30T02:41:18Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 51de01b8-cd20-4dfa-a1eb-81716f3c6eb2

📥 Commits

Reviewing files that changed from the base of the PR and between a34c981 and a397535.

📒 Files selected for processing (1)

tests/integration/test_lists/waives.txt

💤 Files with no reviewable changes (1)

tests/integration/test_lists/waives.txt

📝 Walkthrough

Walkthrough

A skip/waive entry for a specific parametrized test in tests/integration/test_lists/waives.txt was removed, meaning the test accuracy/test_llm_api_pytorch.py::TestDeepSeekV32::test_nvfp4_multi_gpus_piecewise_cuda_graph[mtp3_fp8kv_chunked] is no longer listed to be skipped.

Changes

Cohort / File(s)	Summary
Test Configuration `tests/integration/test_lists/waives.txt`	Removed a waive entry for a parametrized test variant, allowing it to run instead of being skipped.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~2 minutes

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly references the specific test being unwaived and the associated bug number, directly matching the changeset.
Description check	✅ Passed	The description follows the template, includes the bug reference, explains what is being done and why, specifies test coverage, and completes the PR checklist.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

tensorrt-cicd · 2026-03-30T02:45:38Z

PR_Github #40632 [ run ] triggered by Bot. Commit: a397535 Link to invocation

tensorrt-cicd · 2026-03-30T06:18:30Z

PR_Github #40632 [ run ] completed with state SUCCESS. Commit: a397535
/LLM/main/L0_MergeRequest_PR pipeline #31671 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

yizhang-nv · 2026-03-30T06:47:44Z

/bot run --only-multi-gpu-test --disable-fail-fast

tensorrt-cicd · 2026-03-30T06:53:14Z

PR_Github #40683 [ run ] triggered by Bot. Commit: a397535 Link to invocation

tensorrt-cicd · 2026-03-30T11:21:23Z

PR_Github #40683 [ run ] completed with state SUCCESS. Commit: a397535
/LLM/main/L0_MergeRequest_PR pipeline #31713 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

liji-nv · 2026-04-03T13:41:27Z

The PR is currently blocked by #12659

yizhang-nv · 2026-04-16T07:12:41Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-04-16T07:18:23Z

PR_Github #43690 [ run ] triggered by Bot. Commit: ac31507 Link to invocation

tensorrt-cicd · 2026-04-16T20:34:13Z

PR_Github #43690 [ run ] completed with state FAILURE. Commit: ac31507
/LLM/main/L0_MergeRequest_PR pipeline #34174 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

yizhang-nv · 2026-04-17T09:18:14Z

/bot run --only-multi-gpu-test --disable-fail-fast

tensorrt-cicd · 2026-04-17T09:24:38Z

PR_Github #44011 [ run ] triggered by Bot. Commit: 0a2c4f0 Link to invocation

tensorrt-cicd · 2026-04-18T09:08:31Z

PR_Github #44011 [ run ] completed with state FAILURE. Commit: 0a2c4f0
/LLM/main/L0_MergeRequest_PR pipeline #34450 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

yizhang-nv · 2026-04-20T05:15:38Z

/bot run --only-multi-gpu-test --disable-fail-fast

tensorrt-cicd · 2026-04-20T05:21:37Z

PR_Github #44306 [ run ] triggered by Bot. Commit: a337977 Link to invocation

tensorrt-cicd · 2026-04-20T05:50:35Z

PR_Github #44306 [ run ] completed with state FAILURE. Commit: a337977
/LLM/main/L0_MergeRequest_PR pipeline #34727 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

yizhang-nv · 2026-04-20T09:03:32Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-04-20T09:10:54Z

PR_Github #44422 [ run ] triggered by Bot. Commit: 85ab435 Link to invocation

yizhang-nv · 2026-04-20T13:37:51Z

/bot run --only-multi-gpu-test --disable-fail-fast

tensorrt-cicd · 2026-04-20T13:43:52Z

PR_Github #44461 [ run ] triggered by Bot. Commit: 85ab435 Link to invocation

tensorrt-cicd · 2026-04-20T13:44:00Z

PR_Github #44422 [ run ] completed with state ABORTED. Commit: 85ab435

Link to invocation

tensorrt-cicd · 2026-04-20T18:18:54Z

PR_Github #44461 [ run ] completed with state SUCCESS. Commit: 85ab435
/LLM/main/L0_MergeRequest_PR pipeline #34864 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

yizhang-nv · 2026-04-21T02:32:28Z

/bot run --only-multi-gpu-test --disable-fail-fast

tensorrt-cicd · 2026-04-21T02:37:57Z

PR_Github #44587 [ run ] triggered by Bot. Commit: 85ab435 Link to invocation

tensorrt-cicd · 2026-04-21T05:09:00Z

PR_Github #44587 [ run ] completed with state SUCCESS. Commit: 85ab435
/LLM/main/L0_MergeRequest_PR pipeline #34972 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

yizhang-nv · 2026-04-21T10:05:56Z

/bot run --only-multi-gpu-test --disable-fail-fast

tensorrt-cicd · 2026-04-21T10:13:37Z

PR_Github #44702 [ run ] triggered by Bot. Commit: 9473334 Link to invocation

tensorrt-cicd · 2026-04-21T20:58:25Z

PR_Github #44702 [ run ] completed with state SUCCESS. Commit: 9473334
/LLM/main/L0_MergeRequest_PR pipeline #35066 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

yizhang-nv · 2026-04-22T01:45:48Z

/bot run --only-multi-gpu-test --disable-fail-fast

Unwaive TestDeepSeekV32::test_nvfp4_multi_gpus_piecewise_cuda_graph[mtp3_fp8kv_chunked] (nvbugs/5989920) to re-enable it in CI. Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>

yizhang-nv · 2026-04-22T07:47:55Z

/bot run --only-multi-gpu-test --disable-fail-fast

tensorrt-cicd · 2026-04-22T07:53:31Z

PR_Github #44934 [ run ] triggered by Bot. Commit: 41a4405 Link to invocation

yizhang-nv · 2026-04-23T02:27:47Z

/bot run --only-multi-gpu-test --disable-fail-fast

tensorrt-cicd · 2026-04-23T02:33:52Z

PR_Github #45079 [ run ] triggered by Bot. Commit: 41a4405 Link to invocation

tensorrt-cicd · 2026-04-23T20:02:45Z

PR_Github #45079 [ run ] completed with state SUCCESS. Commit: 41a4405
/LLM/main/L0_MergeRequest_PR pipeline #35376 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

yizhang-nv · 2026-04-25T03:48:03Z

/bot run --only-multi-gpu-test --disable-fail-fast

Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>

yizhang-nv · 2026-04-27T02:37:42Z

/bot run --only-multi-gpu-test --disable-fail-fast

tensorrt-cicd · 2026-04-27T02:44:42Z

PR_Github #45627 [ run ] triggered by Bot. Commit: d277abb Link to invocation

tensorrt-cicd · 2026-04-27T07:45:57Z

PR_Github #45627 [ run ] completed with state SUCCESS. Commit: d277abb
/LLM/main/L0_MergeRequest_PR pipeline #35840 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>

yizhang-nv · 2026-05-25T10:20:58Z

/bot run --only-multi-gpu-test --disable-fail-fast

yizhang-nv · 2026-05-25T10:27:03Z

/bot run --only-multi-gpu-test --disable-fail-fast

tensorrt-cicd · 2026-05-25T10:27:04Z

PR_Github #50207 [ run ] triggered by Bot. Commit: e3cf242 Link to invocation

tensorrt-cicd · 2026-05-25T10:33:21Z

PR_Github #50208 [ run ] triggered by Bot. Commit: e3cf242 Link to invocation

tensorrt-cicd · 2026-05-25T15:07:52Z

PR_Github #50208 [ run ] completed with state SUCCESS. Commit: e3cf242
/LLM/main/L0_MergeRequest_PR pipeline #39746 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

github-actions Bot assigned yizhang-nv Mar 25, 2026

yizhang-nv changed the title ~~[None][test] Unwaive DeepSeekV3 nvfp4 mtp3_fp8kv_chunked test~~ [https://nvbugs/5989920][test] Unwaive DeepSeekV3 nvfp4 mtp3_fp8kv_chunked test Mar 25, 2026

yizhang-nv force-pushed the unwaive-deepseekv3-nvfp4-mtp3-fp8kv-chunked branch from dc1cf29 to a397535 Compare March 30, 2026 02:39

liji-nv mentioned this pull request Apr 3, 2026

[https://nvbugs/6029220][fix] Disable multi-stream in maybe_execute_i… #12659

Merged

1 task

yizhang-nv force-pushed the unwaive-deepseekv3-nvfp4-mtp3-fp8kv-chunked branch from a2ecf4a to ac31507 Compare April 16, 2026 07:11

yizhang-nv force-pushed the unwaive-deepseekv3-nvfp4-mtp3-fp8kv-chunked branch from ac31507 to 0a2c4f0 Compare April 17, 2026 09:18

yizhang-nv force-pushed the unwaive-deepseekv3-nvfp4-mtp3-fp8kv-chunked branch from 0a2c4f0 to a337977 Compare April 20, 2026 05:15

yizhang-nv force-pushed the unwaive-deepseekv3-nvfp4-mtp3-fp8kv-chunked branch from a337977 to 85ab435 Compare April 20, 2026 09:03

yizhang-nv force-pushed the unwaive-deepseekv3-nvfp4-mtp3-fp8kv-chunked branch from 85ab435 to 9473334 Compare April 21, 2026 10:05

[None][test] Unwaive DeepSeekV3 nvfp4 mtp3_fp8kv_chunked test

41a4405

Unwaive TestDeepSeekV32::test_nvfp4_multi_gpus_piecewise_cuda_graph[mtp3_fp8kv_chunked] (nvbugs/5989920) to re-enable it in CI. Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>

yizhang-nv force-pushed the unwaive-deepseekv3-nvfp4-mtp3-fp8kv-chunked branch from 9473334 to 41a4405 Compare April 22, 2026 07:47

Merge branch 'main' into unwaive-deepseekv3-nvfp4-mtp3-fp8kv-chunked

d277abb

Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>

Merge branch 'main' into unwaive-deepseekv3-nvfp4-mtp3-fp8kv-chunked

e3cf242

Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>

Conversation

yizhang-nv commented Mar 25, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

yizhang-nv commented Mar 25, 2026

Uh oh!

tensorrt-cicd commented Mar 25, 2026

Uh oh!

tensorrt-cicd commented Mar 25, 2026

Uh oh!

yizhang-nv commented Mar 30, 2026

Uh oh!

coderabbitai Bot commented Mar 30, 2026

Walkthrough

Changes

Estimated code review effort

Uh oh!

tensorrt-cicd commented Mar 30, 2026

Uh oh!

tensorrt-cicd commented Mar 30, 2026

Uh oh!

yizhang-nv commented Mar 30, 2026

Uh oh!

tensorrt-cicd commented Mar 30, 2026

Uh oh!

tensorrt-cicd commented Mar 30, 2026

Uh oh!

liji-nv commented Apr 3, 2026

Uh oh!

yizhang-nv commented Apr 16, 2026

Uh oh!

tensorrt-cicd commented Apr 16, 2026

Uh oh!

tensorrt-cicd commented Apr 16, 2026

Uh oh!

yizhang-nv commented Apr 17, 2026

Uh oh!

tensorrt-cicd commented Apr 17, 2026

Uh oh!

tensorrt-cicd commented Apr 18, 2026

Uh oh!

yizhang-nv commented Apr 20, 2026

Uh oh!

tensorrt-cicd commented Apr 20, 2026

Uh oh!

tensorrt-cicd commented Apr 20, 2026

Uh oh!

yizhang-nv commented Apr 20, 2026

Uh oh!

tensorrt-cicd commented Apr 20, 2026

Uh oh!

yizhang-nv commented Apr 20, 2026

Uh oh!

tensorrt-cicd commented Apr 20, 2026

Uh oh!

tensorrt-cicd commented Apr 20, 2026

Uh oh!

tensorrt-cicd commented Apr 20, 2026

Uh oh!

yizhang-nv commented Apr 21, 2026

Uh oh!

tensorrt-cicd commented Apr 21, 2026

Uh oh!

tensorrt-cicd commented Apr 21, 2026

Uh oh!

yizhang-nv commented Apr 21, 2026

Uh oh!

tensorrt-cicd commented Apr 21, 2026

Uh oh!

tensorrt-cicd commented Apr 21, 2026

Uh oh!

yizhang-nv commented Apr 22, 2026

Uh oh!

yizhang-nv commented Apr 22, 2026

Uh oh!

yizhang-nv commented Mar 25, 2026 •

edited by coderabbitai Bot

Loading

yizhang-nv commented May 25, 2026 •

edited

Loading