When can Deepseek be supported

[jiasu-hezhip]

I just want to know if it's feasible to remove some experts?

[BaiTheBest]

Thank you for your interest in our work!

Because DeepSeek v3 and R1 are so recent, we’ll need a bit of time to explore how to adapt our SparseLLM methodology for pruning its MoE experts. However, I can definitely provide some personal insights on how our global pruning framework can, in principle, be extended to the MoE setting in DeepSeek:

1. In our paper, SparseLLM views each feed‐forward block as a whole for pruning. Crucially, SparseLLM is global—it uses auxiliary variables to "connect" all sub-layers across the whole FFN during pruning, rather than making local, layer‐by‐layer decisions.

2. Extending to MoE: In DeepSeekMoE, each “MoE layer” is not a single FFN block; instead, it’s a collection of experts (i.e., multiple parallel FFNs) plus a router. To do a “global” approach, you’d want to treat all experts across an MoE layer as pruning candidates—so you can figure out whether, for example, removing Expert 2 is more beneficial than removing Expert 7, and so on.

3. SparseLLM decomposes pruning into manageable subproblems via auxiliary variables. In MoE, you can still adopt a similar “soft constraint” approach, but now each expert is effectively a block to be zeroed out or kept. This might require a slight reformulation of the original objective in SparseLLM.