perf(moe): speed up grouped MoE routing - #20
Merged
Conversation
Signed-off-by: Shivanjan Chakravorty <shivanjanc@nvidia.com>
Glitchfix
marked this pull request as ready for review
June 4, 2026 16:35
Collaborator
|
overall LGTM |
lfengad
approved these changes
Jun 5, 2026
foreverlms
approved these changes
Jun 8, 2026
lfengad
enabled auto-merge (squash)
June 8, 2026 12:00
rahul-steiger-nv
pushed a commit
to rahul-steiger-nv/cosmos-framework
that referenced
this pull request
Jun 15, 2026
## Summary This PR tightens the routing/unrouting path in `Qwen3VLMoeTextExpertsGroupedMm`: - keep routed token indices as a 1D vector instead of expanding them across the hidden dimension - use `index_select` / `index_add_` for route and combine instead of hidden-size-expanded `gather` / `scatter_add_` - update the standalone MoE comparison harness so it calls the current expert API and checks grouped output against the PyTorch reference The old path was doing a lot of unnecessary index traffic. That is mostly invisible for smaller routing configs, but it becomes very noticeable when `top_k` and the expert count go up. ## Benchmarks Environment: RTX 6000 Ada, `torch==2.10.0+cu128`, BF16 unless noted. Focused MoE harness: ```text python -m cosmos_framework.model.vfm.vlm.qwen3_vl_moe.moe_test naive: 13.81 ms optimized: 3.92 ms speedup: 3.53x relative diff: 0.00531 ``` Synthetic decoder stack throughput, including attention + norms + MoE, old grouped routing vs this PR: ```text E128_K8_I768, 8 layers, seq=1024 old: 134.58 ms, 7,608.7 tok/s new: 42.16 ms, 24,289.3 tok/s speedup: 3.19x, +219.2% E128_K8_I768, 12 layers, seq=1024 old: 136.89 ms, 7,480.3 tok/s new: 63.33 ms, 16,170.2 tok/s speedup: 2.16x, +116.2% E128_K8_I1408, 8 layers, seq=1024 old: 103.66 ms, 9,878.7 tok/s new: 52.81 ms, 19,391.2 tok/s speedup: 1.96x, +96.3% ``` For the default-ish `E=60, top_k=4` shape, I did not see a meaningful total-throughput improvement. The gain scales with routing pressure; high-expert / high-`top_k` configurations are where the old expanded-index route/combine path becomes expensive. ## Tests ```text LD_LIBRARY_PATH= uv run --no-sync python -m ruff check \ cosmos_framework/model/vfm/vlm/qwen3_vl_moe/moe.py \ cosmos_framework/model/vfm/vlm/qwen3_vl_moe/moe_test.py LD_LIBRARY_PATH= uv run --no-sync python -m cosmos_framework.model.vfm.vlm.qwen3_vl_moe.moe_test LD_LIBRARY_PATH= uv run --no-sync python -m cosmos_framework.model.vfm.vlm.qwen3_vl_moe.moe_bench \ --compare --num-tokens 256 --num-iters 10 --num-warmup 3 --no-compile --dtype fp32 LD_LIBRARY_PATH= uv run --no-sync python -m cosmos_framework.model.vfm.vlm.qwen3_vl_moe.moe_bench \ --compare --num-tokens 2048 --num-iters 20 --num-warmup 5 --no-compile --dtype bf16 LD_LIBRARY_PATH= uv run --no-sync python -m cosmos_framework.model.vfm.vlm.qwen3_vl_moe.moe_bench \ --num-tokens 1024 --num-iters 5 --num-warmup 2 --dtype bf16 --backward ``` Signed-off-by: Shivanjan Chakravorty <shivanjanc@nvidia.com> Co-authored-by: lfengad <liangf@nvidia.com>
wxwy
added a commit
to wxwy/cosmos-framework
that referenced
this pull request
Aug 27, 2026
按 2026-08-27 用户要求"smoke 跑完后的计划要自动触发":
- eval_libero_4in1_acceptance_4090.sh:末尾追加 auto-post-smoke hook
调用 tools/g0/auto_post_smoke.sh
AUTO_POST_SMOKE_HOOK=0 可关;本次 driver 实例已跑完不生效,未来重跑自动走
锁由 auto_post_smoke.sh 自身 sentinel 保证幂等
- launch_action_server_libero_edge_all.sh:转发 GUIDANCE 给 server
--guidance "${GUIDANCE:-1.0}",CFG 调试必需
- launch_closed_loop_eval_libero_task0.sh:SAVE_GIFS/SAVE_PRED_MP4 用 ${VAR:+...}
set -u 下 unset 不触发 unbound,task NVIDIA#20 防御性变量
Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR tightens the routing/unrouting path in
Qwen3VLMoeTextExpertsGroupedMm:index_select/index_add_for route and combine instead of hidden-size-expandedgather/scatter_add_The old path was doing a lot of unnecessary index traffic. That is mostly invisible for smaller routing configs, but it becomes very noticeable when
top_kand the expert count go up.Benchmarks
Environment: RTX 6000 Ada,
torch==2.10.0+cu128, BF16 unless noted.Focused MoE harness:
Synthetic decoder stack throughput, including attention + norms + MoE, old grouped routing vs this PR:
For the default-ish
E=60, top_k=4shape, I did not see a meaningful total-throughput improvement. The gain scales with routing pressure; high-expert / high-top_kconfigurations are where the old expanded-index route/combine path becomes expensive.Tests