Clarify packed MoE and QMoE semantics - #32447
Draft
Sayan Shaw (sayanshaw24) wants to merge 1 commit into
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Clarify packed MoE/QMoE semantics and cover WebGPU QMoE
Description
Clarifies that the existing 2D
MoEandQMoEinputs are already the packed token-major representation required by continuous-batching engines. Tokens from variable-length requests can be concatenated into(num_tokens, hidden_size), with matching router rows in(num_tokens, num_experts).Unlike sequence-dependent operators such as
NGramHashMapping, MoE routing and expert evaluation are token-local. Request boundaries cannot affect a token's result, so a separateVarlenMoE/VarlenQMoEschema andcumulative_sequence_lengthinput would be redundant.The existing 3D
(batch_size, sequence_length, hidden_size)form remains a dense convenience layout and produces an output of the same shape.Changes
MoEandQMoEschemas.[2, 1, 2]packed into one 2D input.Why no cumulative sequence lengths?
VarlenNGramHashMappingneeds explicit request boundaries because each output reads preceding token IDs. MoE/QMoE do not read neighboring tokens: each input row is routed and evaluated independently. Flattening multiple requests is therefore equivalent to evaluating those requests separately and concatenating their outputs.Qwen 3.8 Flash / WebGPU scope
The Qwen MoE builder already emits the QMoE configuration supported by WebGPU for the normal INT4 path: packed/dense token rows, symmetric quantization, fused SwiGLU, and no FC3 input. This PR establishes the packed contract and adds explicit WebGPU coverage before pursuing unrelated CUDA-parity features in separate EP-specific PRs.
Remaining WebGPU QMoE gaps include explicit zero points, separate
router_weights, FC3 gating, additional activations, and CUDA-only FP4/FP8 modes. Those are feature-parity work rather than prerequisites for the packed input layout.Validation
clang-format --dry-run --Werrorpasses for changed C++ files.git diff --checkpasses.