Add Qwen3 and Qwen3-MoE

[Blaizzy]

No description provided.

[awni]

@Goekdeniz-Guelmez can you combine efforts with @Blaizzy here and merge in #42. The two PRs were back to back so let's make you both co-authors. But I think since this one came in first unless there is a good reason we can use it as the working draft.

[Blaizzy]

Awesome, lets collaborate once again @Goekdeniz-Guelmez!

[Blaizzy]

If you are interested you can check the @Goekdeniz-Guelmez moe.

still needs some work

[Goekdeniz-Guelmez]

Wow, that was actually amazing timing :D, yes your PR should merge singer you was faster then me, I'll just look at your code here and give you some comments, and delete my PR.

[Goekdeniz-Guelmez]

#44 Here is the draft PR, im now trying inference with it, I'll tell you how it goes.

[Blaizzy]

Awesome!
Point your PR to my branch since it will depend on it.

[Blaizzy]

Hey @awni

Fixed the bug on the MoE, unlike previous v2, the v3 now normalizes the experts topk probs.

We are ready to review, merge and wait for the official release!

[awni]

Thanks!

[Blaizzy]

My pleasure!