about MLA kv cache

[FL77N]

Hello, I wanted to kindly ask for clarification: In the DeepseekV2AttentionMLA implementation within sglang, is it correct that the prefill phase does not utilize KV cache, and the decode phase is when KV cache is used? Additionally, when employing MLA, would it be advisable to avoid enabling the mix chunked feature? Thank you for your guidance

[ispobock]

In the DeepseekV2AttentionMLA implementation within sglang, is it correct that the prefill phase does not utilize KV cache, and the decode phase is when KV cache is used?

Prefill will not use KV cache. Extend and decode will use KV cache.

Additionally, when employing MLA, would it be advisable to avoid enabling the mix chunked feature?

Yes. Prefill and decode have different computation/memory access characteristics. We optimize it with different forward logic. It's better to use separate batch.

[FL77N]

thank for you reply!

[imoisture]

what is the difference between Extend and Prefill mode?🦾

[wejoncy]

Hi @ispobock
Kindly asking, Is it possible to go normal branch in prefill/extend phase? I am trying to enable it for a 32k seq prefill, and the output is not correct. However, the profiling shows forward_normal is faster than forward_absorb().

Would you point me how to enable the forward_normal with extend support in subsequent chunks? Much thanks.