[None][feat] Add silu to trtllm-gen MoE by IwakuraRein · Pull Request #11663 · NVIDIA/TensorRT-LLM

IwakuraRein · 2026-02-23T23:21:18Z

Description

This PR aims to allow Trtllm-gen MoE support the silu activation.

The change in the tolerance of MoE unit test is to ensure the internal CI/CD passes. Before, the unittest/_torch/thop/serial/test_moe.py::TestMoeFp4::test_autotune[RoutingDS_SuperV3-silu-768-1024-1024] failed with mismatched percentage 8.1762%.

Test Coverage

pytest tests/unittest/_torch/thop/serial/test_moe.py -k silu

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...

Provide a user friendly way for developers to interact with a Jenkins server.

Run /bot [-h|--help] to print this help message.

See details below for each supported subcommand.

Details

run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental)]

Launch build/test pipelines. All previously running jobs will be killed.

--reuse-test (optional)pipeline-id (OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.

--disable-reuse-test (OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.

--disable-fail-fast (OPTIONAL) : Disable fail fast on build/tests/infra failures.

--skip-test (OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.

--stage-list "A10-PyTorch-1, xxx" (OPTIONAL) : Only run the specified test stages. Examples: "A10-PyTorch-1, xxx". Note: Does NOT update GitHub check status.

--gpu-type "A30, H100_PCIe" (OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.

--test-backend "pytorch, cpp" (OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.

--only-multi-gpu-test (OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.

--disable-multi-gpu-test (OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.

--add-multi-gpu-test (OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.

--post-merge (OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.

--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" (OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx".

--detailed-log (OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.

--debug (OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in the stage-list parameter to access the appropriate container environment. Note: Does NOT update GitHub check status.

For guidance on mapping tests to stage names, see docs/source/reference/ci-overview.md
and the scripts/test_to_stage_mapping.py helper.

kill

kill

Kill all running builds associated with pull request.

skip

skip --comment COMMENT

Skip testing for latest commit on pull request. --comment "Reason for skipping build/test" is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

reuse-pipeline

reuse-pipeline

Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

IwakuraRein · 2026-02-25T17:53:04Z

/bot run

nvxuanyuc · 2026-02-25T19:14:57Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-02-25T19:28:04Z

PR_Github #36824 [ run ] triggered by Bot. Commit: 08cdadc Link to invocation

tensorrt-cicd · 2026-02-26T02:42:49Z

PR_Github #36824 [ run ] completed with state SUCCESS. Commit: 08cdadc
/LLM/main/L0_MergeRequest_PR pipeline #28515 completed with status: 'FAILURE'

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

nvxuanyuc · 2026-02-26T18:46:28Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-02-26T18:57:13Z

PR_Github #36958 [ run ] triggered by Bot. Commit: 155ba91 Link to invocation

tensorrt-cicd · 2026-02-26T19:00:53Z

PR_Github #36958 [ run ] completed with state FAILURE. Commit: 155ba91

Link to invocation

nvxuanyuc · 2026-02-26T19:02:29Z

/bot run

tensorrt-cicd · 2026-02-26T19:09:20Z

PR_Github #36960 [ run ] triggered by Bot. Commit: 155ba91 Link to invocation

tensorrt-cicd · 2026-02-26T19:12:03Z

PR_Github #36960 [ run ] completed with state FAILURE. Commit: 155ba91

Link to invocation

nvxuanyuc · 2026-02-26T21:39:56Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-02-26T21:45:56Z

PR_Github #36969 [ run ] triggered by Bot. Commit: 155ba91 Link to invocation

tensorrt-cicd · 2026-02-26T21:50:46Z

PR_Github #36969 [ run ] completed with state FAILURE. Commit: 155ba91

Link to invocation

IwakuraRein · 2026-02-27T04:28:43Z

/bot run --disable-fail-fast

nvxuanyuc · 2026-02-27T04:33:43Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-02-27T04:40:05Z

PR_Github #37021 [ run ] triggered by Bot. Commit: 235ac57 Link to invocation

tensorrt-cicd · 2026-02-27T11:29:03Z

PR_Github #37021 [ run ] completed with state SUCCESS. Commit: 235ac57
/LLM/main/L0_MergeRequest_PR pipeline #28665 completed with status: 'FAILURE'

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

nvxuanyuc · 2026-02-28T01:07:47Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-02-28T01:20:32Z

PR_Github #37135 [ run ] triggered by Bot. Commit: efffefb Link to invocation

nvxuanyuc · 2026-03-02T17:44:04Z

/bot run

xxi-nv · 2026-03-04T05:38:07Z

FYI, test_moe.py is deprecating, please check https://github.com/NVIDIA/TensorRT-LLM/pull/11817/changes#diff-4eacc2997b498ca9e12ae0c0848da6844d546fbc24e5182cc6b8c4ffc4e303e3.
Please add the test case in the test_moe_backend.py or test_moe_module.py.
You can do it in another PR.

rosenrodt

Looks good to me on the runner part of the change.

nvxuanyuc · 2026-03-04T20:44:17Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-03-04T20:50:43Z

PR_Github #37757 [ run ] triggered by Bot. Commit: cc841cc Link to invocation

tensorrt-cicd · 2026-03-05T01:36:08Z

PR_Github #37757 [ run ] completed with state SUCCESS. Commit: cc841cc
/LLM/main/L0_MergeRequest_PR pipeline #29227 completed with status: 'FAILURE'

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

jiahanc · 2026-03-05T04:41:57Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-03-05T04:47:55Z

PR_Github #37807 [ run ] triggered by Bot. Commit: cc841cc Link to invocation

tensorrt-cicd · 2026-03-05T09:32:40Z

PR_Github #37807 [ run ] completed with state SUCCESS. Commit: cc841cc
/LLM/main/L0_MergeRequest_PR pipeline #29270 completed with status: 'FAILURE'

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

Signed-off-by: Siyuan Fu <siyuanf@nvidia.com> update config.json Signed-off-by: Siyuan Fu <siyuanf@nvidia.com> add silu Signed-off-by: Siyuan Fu <siyuanf@nvidia.com> update to main branch headers Signed-off-by: Siyuan Fu <siyuanf@nvidia.com> update cubins Signed-off-by: Siyuan Fu <siyuanf@nvidia.com> remove script Signed-off-by: Siyuan Fu <siyuanf@nvidia.com> trailing-whitespace Signed-off-by: siyuanf <siyuanf@nvidia.com> increase tolerance Signed-off-by: Siyuan Fu <siyuanf@nvidia.com>

jiahanc · 2026-03-06T02:47:59Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-03-06T02:53:43Z

PR_Github #37948 [ run ] triggered by Bot. Commit: e2f355c Link to invocation

xxi-nv

The MOE python LGTM.

tensorrt_llm/_torch/utils.py

tensorrt-cicd · 2026-03-06T10:28:51Z

PR_Github #37948 [ run ] completed with state SUCCESS. Commit: e2f355c
/LLM/main/L0_MergeRequest_PR pipeline #29391 completed with status: 'FAILURE'

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

IwakuraRein · 2026-03-06T18:10:08Z

/bot run

hchings

There's very few llm-api changes in this MR. Approve to unblock. The core MoE part needs Anthony and Xingfei's review if have not.

brb-nv · 2026-03-06T19:14:59Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-03-06T19:21:51Z

PR_Github #38060 [ run ] triggered by Bot. Commit: e2f355c Link to invocation

IwakuraRein · 2026-03-06T20:24:05Z

FYI, test_moe.py is deprecating, please check https://github.com/NVIDIA/TensorRT-LLM/pull/11817/changes#diff-4eacc2997b498ca9e12ae0c0848da6844d546fbc24e5182cc6b8c4ffc4e303e3. Please add the test case in the test_moe_backend.py or test_moe_module.py. You can do it in another PR.

Hi @xxi-nv, if I'm understanding correctly, the test_moe_backend.py and test_moe_module.py test the fused moe modules, not the kernels directly. However, looks like the fused moe modules aren't compatible with non-gated activation yet? For instance, there is an assertion that the w3 cannot be None:

TensorRT-LLM/tensorrt_llm/_torch/modules/fused_moe/quantization.py

Line 557 in 5918348

assert w1_weight is not None and w3_weight is not None

tensorrt-cicd · 2026-03-06T20:52:07Z

PR_Github #38060 [ run ] completed with state SUCCESS. Commit: e2f355c
/LLM/main/L0_MergeRequest_PR pipeline #29487 completed with status: 'SUCCESS'

Link to invocation

Signed-off-by: Siyuan Fu <siyuanf@nvidia.com>

IwakuraRein marked this pull request as ready for review February 25, 2026 17:52

IwakuraRein requested review from a team as code owners February 25, 2026 17:52

IwakuraRein requested review from galagam, mikeiovine and yilin-void February 25, 2026 17:52

IwakuraRein force-pushed the silu-support branch 2 times, most recently from eb40fb2 to 235ac57 Compare February 27, 2026 02:58

IwakuraRein force-pushed the silu-support branch from 235ac57 to efffefb Compare February 28, 2026 00:06

rosenrodt approved these changes Mar 4, 2026

View reviewed changes

IwakuraRein force-pushed the silu-support branch 2 times, most recently from 0194ad2 to cc841cc Compare March 4, 2026 19:51

IwakuraRein requested a review from a team as a code owner March 4, 2026 19:51

IwakuraRein requested a review from hchings March 4, 2026 19:51

IwakuraRein force-pushed the silu-support branch from cc841cc to e2f355c Compare March 6, 2026 01:48

xxi-nv approved these changes Mar 6, 2026

View reviewed changes

tensorrt_llm/_torch/utils.py Show resolved Hide resolved

hchings approved these changes Mar 6, 2026

View reviewed changes

lucaslie approved these changes Mar 6, 2026

View reviewed changes

LarryXFly approved these changes Mar 7, 2026

View reviewed changes

niukuo approved these changes Mar 9, 2026

View reviewed changes

jhaotingc merged commit ae2dc3d into NVIDIA:main Mar 9, 2026
5 checks passed

tianyuz-nv pushed a commit to wanqian-nv/TensorRT-LLM that referenced this pull request Mar 19, 2026

[None][feat] Add silu to trtllm-gen MoE (NVIDIA#11663)

2caec44

Signed-off-by: Siyuan Fu <siyuanf@nvidia.com>

limin2021 pushed a commit to limin2021/TensorRT-LLM that referenced this pull request Mar 19, 2026

[None][feat] Add silu to trtllm-gen MoE (NVIDIA#11663)

d766364

Signed-off-by: Siyuan Fu <siyuanf@nvidia.com>

Conversation

IwakuraRein commented Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Test Coverage

PR Checklist

GitHub Bot Help

kill

skip

reuse-pipeline

Uh oh!

IwakuraRein commented Feb 25, 2026

Uh oh!

nvxuanyuc commented Feb 25, 2026

Uh oh!

tensorrt-cicd commented Feb 25, 2026

Uh oh!

tensorrt-cicd commented Feb 26, 2026

Uh oh!

nvxuanyuc commented Feb 26, 2026

Uh oh!

tensorrt-cicd commented Feb 26, 2026

Uh oh!

tensorrt-cicd commented Feb 26, 2026

Uh oh!

nvxuanyuc commented Feb 26, 2026

Uh oh!

tensorrt-cicd commented Feb 26, 2026

Uh oh!

tensorrt-cicd commented Feb 26, 2026

Uh oh!

nvxuanyuc commented Feb 26, 2026

Uh oh!

tensorrt-cicd commented Feb 26, 2026

Uh oh!

tensorrt-cicd commented Feb 26, 2026

Uh oh!

IwakuraRein commented Feb 27, 2026

Uh oh!

nvxuanyuc commented Feb 27, 2026

Uh oh!

tensorrt-cicd commented Feb 27, 2026

Uh oh!

tensorrt-cicd commented Feb 27, 2026

Uh oh!

nvxuanyuc commented Feb 28, 2026

Uh oh!

tensorrt-cicd commented Feb 28, 2026

Uh oh!

nvxuanyuc commented Mar 2, 2026

Uh oh!

xxi-nv commented Mar 4, 2026

Uh oh!

rosenrodt left a comment

Choose a reason for hiding this comment

Uh oh!

nvxuanyuc commented Mar 4, 2026

Uh oh!

tensorrt-cicd commented Mar 4, 2026

Uh oh!

tensorrt-cicd commented Mar 5, 2026

Uh oh!

jiahanc commented Mar 5, 2026

Uh oh!

tensorrt-cicd commented Mar 5, 2026

Uh oh!

tensorrt-cicd commented Mar 5, 2026

Uh oh!

jiahanc commented Mar 6, 2026

Uh oh!

tensorrt-cicd commented Mar 6, 2026

Uh oh!

xxi-nv left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tensorrt-cicd commented Mar 6, 2026

Uh oh!

IwakuraRein commented Mar 6, 2026

Uh oh!

IwakuraRein commented Feb 23, 2026 •

edited

Loading