The current _append_paged_kv_cache_kernel does not support k_head_dim being different from v_head_dim

[qiyuxinlin]

[yzh119]

Hi @qiyuxinlin yes that's true, but I wonder what's the use case of this?

For MLA, there should be only one cache with head_dim 576 or two cache, with head_dim_ckv=512 and head_dim_kpe=64, I suppose you are talking about the later case?

[qiyuxinlin]

Hi @yzh119. I want to keep two mla caches with head_dim of 512 and 64 respectively.