Skip to content

Clarify packed MoE and QMoE semantics - #32447

Draft
Sayan Shaw (sayanshaw24) wants to merge 1 commit into
mainfrom
sayanshaw/qmoe
Draft

Clarify packed MoE and QMoE semantics#32447
Sayan Shaw (sayanshaw24) wants to merge 1 commit into
mainfrom
sayanshaw/qmoe

Conversation

@sayanshaw24

Copy link
Copy Markdown
Contributor

Clarify packed MoE/QMoE semantics and cover WebGPU QMoE

Description

Clarifies that the existing 2D MoE and QMoE inputs are already the packed token-major representation required by continuous-batching engines. Tokens from variable-length requests can be concatenated into (num_tokens, hidden_size), with matching router rows in (num_tokens, num_experts).

Unlike sequence-dependent operators such as NGramHashMapping, MoE routing and expert evaluation are token-local. Request boundaries cannot affect a token's result, so a separate VarlenMoE/VarlenQMoE schema and cumulative_sequence_length input would be redundant.

The existing 3D (batch_size, sequence_length, hidden_size) form remains a dense convenience layout and produces an output of the same shape.

Changes

  • Documents packed 2D and dense 3D semantics in the authoritative MoE and QMoE schemas.
  • Clarifies that router inputs contain one row per token.
  • Updates generated contrib-op documentation.
  • Adds a WebGPU QMoE test representing three ragged requests with lengths [2, 1, 2] packed into one 2D input.
  • Uses expert-specific outputs in the test to verify routing order and token independence.

Why no cumulative sequence lengths?

VarlenNGramHashMapping needs explicit request boundaries because each output reads preceding token IDs. MoE/QMoE do not read neighboring tokens: each input row is routed and evaluated independently. Flattening multiple requests is therefore equivalent to evaluating those requests separately and concatenating their outputs.

Qwen 3.8 Flash / WebGPU scope

The Qwen MoE builder already emits the QMoE configuration supported by WebGPU for the normal INT4 path: packed/dense token rows, symmetric quantization, fused SwiGLU, and no FC3 input. This PR establishes the packed contract and adds explicit WebGPU coverage before pursuing unrelated CUDA-parity features in separate EP-specific PRs.

Remaining WebGPU QMoE gaps include explicit zero points, separate router_weights, FC3 gating, additional activations, and CUDA-only FP4/FP8 modes. Those are feature-parity work rather than prerequisites for the packed input layout.

Validation

  • clang-format --dry-run --Werror passes for changed C++ files.
  • git diff --check passes.
  • Editor diagnostics report no errors.
  • WebGPU runtime execution was not available in the local build; CI will execute the new provider test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant