Skip to content

[Model] Add Sarvam MoE model support - #3519

Open
dsing-qual wants to merge 1 commit into
mlc-ai:mainfrom
dsing-qual:sarvam_moe_clean
Open

[Model] Add Sarvam MoE model support#3519
dsing-qual wants to merge 1 commit into
mlc-ai:mainfrom
dsing-qual:sarvam_moe_clean

Conversation

@dsing-qual

@dsing-qual dsing-qual commented Jul 22, 2026

Copy link
Copy Markdown

This PR adds support for the Sarvam MoE architecture in MLC LLM.

Changes include:

  • Add sarvam_moe model implementation with:
    • Sarvam MoE config parsing
    • Qwen3-style attention with Q/K norm
    • Dense early MLP layers
    • Routed MoE expert layers
    • Shared expert branch
    • Sigmoid router with expert bias and routed scaling
  • Add HuggingFace weight loading support for Sarvam checkpoints, including:
    • Embedding/name remapping
    • Packed QKV attention mapping
    • Packed gate/up MLP mapping
    • Routed expert weight stacking
    • Shared expert mapping
    • Router expert-bias mapping
  • Register sarvam_moe in the model registry.
  • This PR adds support for the Sarvam MoE architecture in MLC LLM.

@dsing-qual
dsing-qual force-pushed the sarvam_moe_clean branch 2 times, most recently from 02e1629 to 1723aa2 Compare July 22, 2026 10:02
@dsing-qual dsing-qual changed the title Add Sarvam MoE model support [Model] Add Sarvam MoE model support Jul 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant