[Question] What is MODEL1?

[gary-wjc]

I found the term "MODEL1" is mentioned several times in this repository:

FlashMLA/csrc/sm100/decode/head64/kernel.cuh

Line 554 in 48c6dc4

// MODEL1: RoPE is the last 64 dims within the full 512 dim, which couples with the last 64 dim from the NoPE part when performing dual GEMM. i.e.

FlashMLA/csrc/sm90/decode/sparse_fp8/splitkv_mla.cuh

Line 161 in 48c6dc4

// The following fields are only valid for MODEL1

I have some questions about this novel term. @interestingLSY Could you please take a look?

1. Are there some publicly available models belonging to the category of "MODEL1"?

2. What are the major differences of attention mechanisms between "MODEL1" and V3.2 ?

Update: My preliminary guess is the following for the 2nd question:

1. The topk length becomes dynamic in MODEL1 while constant(2048) in V3.2
2. MODEL1 introduces additional KV states which are not generated from history tokens in the current sequence
3. The number of features in KV cache without position encoding for MODEL1 is 448, which is reduced from 512 as in V3.2
4. Position encoding is also included when recovering V cache

[Cipheeer]

MODEL1 might be the engineering version of V4, because they released R1 on this day a year ago

[DeepSpaceLLM]

656B (for V32) or 576B (for MODEL1) different parameters means use a new architecture? The changes do seem huge, so it should really be called v4. Looking forward to the release of the new model more and more.

[ayokings202]

You and I know its V4

[junjundaidai]

@interestingLSY 你们自己提交上来的代码，我出于好奇问了问题，你们不解释就算了，还要放一个"踩"的表情，什么意思？以为自己很了不起吗？

估计作者点错了？

[Cipheeer]

@interestingLSY 你们自己提交上来的代码，我出于好奇问了问题，你们不解释就算了，还要放一个"踩"的表情，什么意思？以为自己很了不起吗？

这不是大拇哥吗

[junjundaidai]

@interestingLSY 你们自己提交上来的代码，我出于好奇问了问题，你们不解释就算了，还要放一个"踩"的表情，什么意思？以为自己很了不起吗？

这不是大拇哥吗

刚变成大拇指了，果然是点错了（狗头）

[SamCheng0717]

Perhaps this is the R2 model~👿