Future model support (Qwen3 dens/moe)

[Chen-DaYi]

Qwen3 is a relatively popular model in the open-source community. I hope it can be adapted.

https://huggingface.co/collections/Qwen/qwen3-67dd247413f0e2e4f653967f

[xidulu]

Thanks for the suggestions, I do agree that qwen3 is popular and I use it a lot.

However, currently all the authors of this project don't seem to have bandwidth for this project as they either have left the team or busy with other projects...

But I am happy to provide help if you are interested in implementing this yourself.

[Chen-DaYi]

Thanks for the suggestions, I do agree that qwen3 is popular and I use it a lot.

However, currently all the authors of this project don't seem to have bandwidth for this project as they either have left the team or busy with other projects...

But I am happy to provide help if you are interested in implementing this yourself.

Please help me fix this bug, thx!
I don't know what the reason. get kb_embedding or GQA?🥲

add the kb key values to the key and value states
class KBLaMQwen3Attention - forward
code:

elif len(kb_keys.shape) == 3:  # Has a batch dim
    kb_len = kb_keys.shape[1]
    kb_keys = kb_keys.view(
        bsz,
        kb_len,
        1 + self.config.num_hidden_layers // kb_layer_frequency,
        -1,
    )[:, :, kb_idx]
    kb_values = kb_values.view(
        bsz,
        kb_len,
        1 + self.config.num_hidden_layers // kb_layer_frequency,
        -1,
    )[:, :, kb_idx]
    kb_keys = kb_keys.view(
        bsz, kb_len, self.config.num_attention_heads, self.config.head_dim
    ).transpose(1, 2)
    kb_values = kb_values.view(
        bsz, kb_len, self.config.num_attention_heads, self.config.head_dim
    ).transpose(1, 2)

error:
[rank0]: RuntimeError: shape '[10, 6, 32, 128]' is invalid for input of size 153600