Skip to content

perf(moe): speed up grouped MoE routing - #20

Merged
lfengad merged 2 commits into
NVIDIA:mainfrom
Glitchfix:perf/qwen3-vl-moe-routing
Jun 8, 2026
Merged

perf(moe): speed up grouped MoE routing#20
lfengad merged 2 commits into
NVIDIA:mainfrom
Glitchfix:perf/qwen3-vl-moe-routing

Conversation

@Glitchfix

Copy link
Copy Markdown
Contributor

Summary

This PR tightens the routing/unrouting path in Qwen3VLMoeTextExpertsGroupedMm:

  • keep routed token indices as a 1D vector instead of expanding them across the hidden dimension
  • use index_select / index_add_ for route and combine instead of hidden-size-expanded gather / scatter_add_
  • update the standalone MoE comparison harness so it calls the current expert API and checks grouped output against the PyTorch reference

The old path was doing a lot of unnecessary index traffic. That is mostly invisible for smaller routing configs, but it becomes very noticeable when top_k and the expert count go up.

Benchmarks

Environment: RTX 6000 Ada, torch==2.10.0+cu128, BF16 unless noted.

Focused MoE harness:

python -m cosmos_framework.model.vfm.vlm.qwen3_vl_moe.moe_test

naive:      13.81 ms
optimized:   3.92 ms
speedup:     3.53x
relative diff: 0.00531

Synthetic decoder stack throughput, including attention + norms + MoE, old grouped routing vs this PR:

E128_K8_I768, 8 layers, seq=1024
old: 134.58 ms,  7,608.7 tok/s
new:  42.16 ms, 24,289.3 tok/s
speedup: 3.19x, +219.2%

E128_K8_I768, 12 layers, seq=1024
old: 136.89 ms,  7,480.3 tok/s
new:  63.33 ms, 16,170.2 tok/s
speedup: 2.16x, +116.2%

E128_K8_I1408, 8 layers, seq=1024
old: 103.66 ms,  9,878.7 tok/s
new:  52.81 ms, 19,391.2 tok/s
speedup: 1.96x, +96.3%

For the default-ish E=60, top_k=4 shape, I did not see a meaningful total-throughput improvement. The gain scales with routing pressure; high-expert / high-top_k configurations are where the old expanded-index route/combine path becomes expensive.

Tests

LD_LIBRARY_PATH= uv run --no-sync python -m ruff check \
  cosmos_framework/model/vfm/vlm/qwen3_vl_moe/moe.py \
  cosmos_framework/model/vfm/vlm/qwen3_vl_moe/moe_test.py

LD_LIBRARY_PATH= uv run --no-sync python -m cosmos_framework.model.vfm.vlm.qwen3_vl_moe.moe_test

LD_LIBRARY_PATH= uv run --no-sync python -m cosmos_framework.model.vfm.vlm.qwen3_vl_moe.moe_bench \
  --compare --num-tokens 256 --num-iters 10 --num-warmup 3 --no-compile --dtype fp32

LD_LIBRARY_PATH= uv run --no-sync python -m cosmos_framework.model.vfm.vlm.qwen3_vl_moe.moe_bench \
  --compare --num-tokens 2048 --num-iters 20 --num-warmup 5 --no-compile --dtype bf16

LD_LIBRARY_PATH= uv run --no-sync python -m cosmos_framework.model.vfm.vlm.qwen3_vl_moe.moe_bench \
  --num-tokens 1024 --num-iters 5 --num-warmup 2 --dtype bf16 --backward

Signed-off-by: Shivanjan Chakravorty <shivanjanc@nvidia.com>
@Glitchfix
Glitchfix marked this pull request as ready for review June 4, 2026 16:35
@foreverlms
foreverlms requested a review from yy-code-nv June 5, 2026 04:11
@lfengad

lfengad commented Jun 5, 2026

Copy link
Copy Markdown
Collaborator

overall LGTM

@lfengad
lfengad enabled auto-merge (squash) June 8, 2026 12:00
@lfengad
lfengad merged commit fa63e99 into NVIDIA:main Jun 8, 2026
7 checks passed
rahul-steiger-nv pushed a commit to rahul-steiger-nv/cosmos-framework that referenced this pull request Jun 15, 2026
## Summary

This PR tightens the routing/unrouting path in
`Qwen3VLMoeTextExpertsGroupedMm`:

- keep routed token indices as a 1D vector instead of expanding them
across the hidden dimension
- use `index_select` / `index_add_` for route and combine instead of
hidden-size-expanded `gather` / `scatter_add_`
- update the standalone MoE comparison harness so it calls the current
expert API and checks grouped output against the PyTorch reference

The old path was doing a lot of unnecessary index traffic. That is
mostly invisible for smaller routing configs, but it becomes very
noticeable when `top_k` and the expert count go up.

## Benchmarks

Environment: RTX 6000 Ada, `torch==2.10.0+cu128`, BF16 unless noted.

Focused MoE harness:

```text
python -m cosmos_framework.model.vfm.vlm.qwen3_vl_moe.moe_test

naive:      13.81 ms
optimized:   3.92 ms
speedup:     3.53x
relative diff: 0.00531
```

Synthetic decoder stack throughput, including attention + norms + MoE,
old grouped routing vs this PR:

```text
E128_K8_I768, 8 layers, seq=1024
old: 134.58 ms,  7,608.7 tok/s
new:  42.16 ms, 24,289.3 tok/s
speedup: 3.19x, +219.2%

E128_K8_I768, 12 layers, seq=1024
old: 136.89 ms,  7,480.3 tok/s
new:  63.33 ms, 16,170.2 tok/s
speedup: 2.16x, +116.2%

E128_K8_I1408, 8 layers, seq=1024
old: 103.66 ms,  9,878.7 tok/s
new:  52.81 ms, 19,391.2 tok/s
speedup: 1.96x, +96.3%
```

For the default-ish `E=60, top_k=4` shape, I did not see a meaningful
total-throughput improvement. The gain scales with routing pressure;
high-expert / high-`top_k` configurations are where the old
expanded-index route/combine path becomes expensive.

## Tests

```text
LD_LIBRARY_PATH= uv run --no-sync python -m ruff check \
  cosmos_framework/model/vfm/vlm/qwen3_vl_moe/moe.py \
  cosmos_framework/model/vfm/vlm/qwen3_vl_moe/moe_test.py

LD_LIBRARY_PATH= uv run --no-sync python -m cosmos_framework.model.vfm.vlm.qwen3_vl_moe.moe_test

LD_LIBRARY_PATH= uv run --no-sync python -m cosmos_framework.model.vfm.vlm.qwen3_vl_moe.moe_bench \
  --compare --num-tokens 256 --num-iters 10 --num-warmup 3 --no-compile --dtype fp32

LD_LIBRARY_PATH= uv run --no-sync python -m cosmos_framework.model.vfm.vlm.qwen3_vl_moe.moe_bench \
  --compare --num-tokens 2048 --num-iters 20 --num-warmup 5 --no-compile --dtype bf16

LD_LIBRARY_PATH= uv run --no-sync python -m cosmos_framework.model.vfm.vlm.qwen3_vl_moe.moe_bench \
  --num-tokens 1024 --num-iters 5 --num-warmup 2 --dtype bf16 --backward
```

Signed-off-by: Shivanjan Chakravorty <shivanjanc@nvidia.com>
Co-authored-by: lfengad <liangf@nvidia.com>
wxwy added a commit to wxwy/cosmos-framework that referenced this pull request Aug 27, 2026
按 2026-08-27 用户要求"smoke 跑完后的计划要自动触发":

- eval_libero_4in1_acceptance_4090.sh:末尾追加 auto-post-smoke hook
  调用 tools/g0/auto_post_smoke.sh
  AUTO_POST_SMOKE_HOOK=0 可关;本次 driver 实例已跑完不生效,未来重跑自动走
  锁由 auto_post_smoke.sh 自身 sentinel 保证幂等

- launch_action_server_libero_edge_all.sh:转发 GUIDANCE 给 server
  --guidance "${GUIDANCE:-1.0}",CFG 调试必需

- launch_closed_loop_eval_libero_task0.sh:SAVE_GIFS/SAVE_PRED_MP4 用 ${VAR:+...}
  set -u 下 unset 不触发 unbound,task NVIDIA#20 防御性变量

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants