[TEST][GPU]qwen3 moe support #30448

riverlijunjie · 2025-05-07T01:18:11Z

Support Qwen3 MoE model running with GPU plugin

Details:

Fuse moe subgraph into single moe_expert op to decrease total ops number and improve compile_model and inference performance.
moe_expert primitive execution stage:
- First token adopts onednn gemm kernels pipeline and optimized opencl kernel(gatther, scatter) to do moe execution, each expert is executed in serial.
- Second token adopts optimized opencl kernels(mlp_gate_up, mlp_down, softmax_topk, reduce) to do multiple-experts parallel execution.
Moe weight of each layer is allocated in a single usm memory and create submemory from it for each expert's weights/scale/zp memory, which is helpful for second token's expert kernels parallel execution.
Optimize key_cache and value_cache input.
Only support moe: u4 weight, f16 scale, u4 zp and group_size=128, which is required by qwen3 MoE 30B model.
Only support systolic gpu (A770/B580/ARL/LNL), doesn't support MTL, because first token need call onednn gemm kernel.

Moe fusion result

Original moe(contains 128 experts) exec graph:

With this PR, it will become one single moe_expert op:

TODO:

Support more MoE patterns, current only verify and support qwen3 moe pattern.
Integrate optimized cm kernel for second token moe
Align cm kernel to use the same scale/zp layout with opencl kernel.
Support more moe data type: u8 weight
Support other subgroup size: 32, 64,256...

Tickets:

https://jira.devtools.intel.com/browse/CVS-166011

… when expert_no == 0 for gpu

* Build Subgraph in parallel to improve compile_model performance * SharedOpOptimization optimizes attribute visit --------- Co-authored-by: Tingqian Li <[email protected]>

…g time.

itikhono · 2025-05-12T13:40:58Z

@praasz @mitruska do we plan to introduce some "internal" specification for ov internal ops? maybe not so formal as for official opset
PagedAttention is covered by some presentation, for Rope we only have some comments in the code (as far as I know)

src/common/transformations/src/transformations/common_optimizations/fuse_moe_expert.cpp

itikhono · 2025-05-12T13:46:58Z

src/common/transformations/src/transformations/common_optimizations/fuse_moe_expert.cpp

+ov::pass::FuseMoeExpert::FuseMoeExpert() {
+    MATCHER_SCOPE(FuseMoeExpert);
+
+    auto expert_mask = makePattern(ov::Rank(3));


@CuriousPanCake could you provide a short transition guide from makePattern to our new symbolic infrastructure? does it cover all presented cases?
should we refine it now or handle it later?

mitruska · 2025-05-12T16:40:33Z

@praasz @mitruska do we plan to introduce some "internal" specification for ov internal ops? maybe not so formal as for official opset PagedAttention is covered by some presentation, for Rope we only have some comments in the code (as far as I know)

@itikhono Directory for internal ops specifications exists here: https://github.com/openvinotoolkit/openvino/tree/master/docs/articles_en/documentation/openvino-ir-format/operation-sets/operation-specs/internal
Feel free to contribute.

src/plugins/intel_gpu/src/graph/impls/ocl_v2/moe_expert_down_cm.cm

src/common/transformations/include/ov_ops/moe_expert.hpp

src/common/transformations/src/ov_ops/moe_expert.cpp

sshlyapn · 2025-05-19T07:52:45Z

src/plugins/intel_gpu/src/graph/impls/ocl_v2/moe_expert_opt.cpp

+    std::unordered_map<std::pair<int, int>, onednn_kernel, PairHash> _kernels;
+
+    onednn_kernel& get_kernel(int n_token, int expert_no, typed_primitive_inst<moe_expert>& instance) {
+        auto key = std::make_pair(n_token, expert_no);


What is the difference between kernels export_no=N and export_no=N+1? Can they be reused?

sshlyapn · 2025-05-19T07:56:59Z

src/plugins/intel_gpu/src/graph/impls/ocl_v2/moe_expert_opt.cpp

+            return std::hash<T1>()(p.first) ^ std::hash<T2>()(p.second);
+        }
+    };
+    std::unordered_map<std::pair<int, int>, onednn_kernel, PairHash> _kernels;


This kernels cache doesn't have any size limitations, and could potentially affect memory consumption after long execution with various cases. I'd suggest reusing existing LRU cache implementation src/plugins/intel_gpu/include/intel_gpu/runtime/lru_cache.hpp instead

sshlyapn · 2025-05-19T07:59:22Z

src/plugins/intel_gpu/src/plugin/ops/moe_expert.cpp

+    auto size = get_weights_size(op);
+    auto layout = cldnn::layout({1, 1, 1, static_cast<ov::Dimension::value_type>(size)}, ov::element::i8, cldnn::format::bfyx);
+    auto alloc_type = p.get_engine().get_preferred_memory_allocation_type(false);
+    auto mem = p.get_engine().allocate_memory(layout, alloc_type, false);


As I understand, this weights will be directly used in the kernels as is, so it would be better to use usm_device isntead of usm_host allocation

I remember get_preferred_memory_allocation_type will return usm_device, will double confirm.

@riverlijunjie , you are right, sorry for the confusion! I mistakenly thought it was the get_lockable_preferred_memory_allocation_type call

sshlyapn · 2025-05-19T08:01:25Z

src/plugins/intel_gpu/src/graph/impls/ocl_v2/moe_expert_opt.cpp

+        for (int i = 0; i < expert_num; i++) {
+            internal_buffers.emplace_back(index_layout, true);  // batch
+            internal_buffers.emplace_back(index_layout, true);  // topk
+        }


I think it would be better to use kernel's scalar arguments for this data instead of internal_buffers, as scalars should have better performance than usm_host allocations

internal_buffers is a vector of cldnn::BufferDescriptor, it will not be sent to kernel.

For sure, my point is that it allocates single-element buffers for the batch and top_k values. I think we shouldn't use buffers for passing these values at all - we can just pass scalar values to the kernel instead

sshlyapn · 2025-05-19T08:03:58Z

src/plugins/intel_gpu/src/graph/impls/ocl_v2/moe_expert_opt.cpp

+                auto expert_no = tok_p[t];
+                OPENVINO_ASSERT(expert_no < max_expert_num);
+                expert_mask.batch[expert_no].push_back(b);
+                expert_mask.topk[expert_no].push_back(t + b * max_topk);


Can we have single larger preallocated buffer for this instead of pushing_back data to reduce runtime memory allocations?

It is used to convert from map[token, top_expert] to map[expert, top_token], it will become complex if we use data/offset to access it.

sshlyapn · 2025-05-19T08:08:17Z

src/plugins/intel_gpu/src/graph/impls/ocl_v2/moe_expert_opt.cpp

+
+        int max_expert_num = static_cast<int>(config.expert_num), max_topk = static_cast<int>(config.topk), max_tokens = static_cast<int>(shape[0]);
+
+        expert_mask.pred_flag.resize(max_expert_num, 0);


I think we can store these masks in the implementation to avoid reallocating them on every invocation

It will bring memory reset to avoid data corruption.

sshlyapn · 2025-05-19T08:13:25Z

src/plugins/intel_gpu/src/graph/impls/ocl_v2/moe_expert_opt.cpp

+        auto final_hidden_states_layout = instance.get_output_layout(0);
+
+        // onednn path will accumulate to the output
+        final_hidden_states_mem_ptr->fill(stream, false);


In case of out of order queue, if the output memory is reused between MOE and some previous operations, it might affect the accuracy of the previous operations, as this call could start filling the memory while the previous operation is still executing, so we should add barrier for OOO queue synchronization here

So we need consider both OOO and In Order? Currently, it should go In Order for systolic platform, do filling here will be safe. If we need support OOO as well, it is needed to add synchronization here, but it will bring some performance issue.

sshlyapn · 2025-05-19T08:15:09Z

src/plugins/intel_gpu/src/graph/impls/ocl_v2/moe_expert_opt.cpp

+
+        expert_mask_cpu expert_mask;
+        // Wait for topk is ready
+        stream.finish();


It would be better to use softmax_topk output event for this synchronization to reduce finish() calls number

sshlyapn · 2025-05-19T08:17:44Z

src/plugins/intel_gpu/src/graph/impls/ocl_v2/moe_expert_opt.cpp

+    dnnl::memory::data_type m_a_type;
+    int bin_post_id;
+
+    static onednn_linear create(dnnl::engine eng,


I think we should reuse existing oneDNN FC/MatMul implementations instead of using oneDNN interfaces here explicitly

It means to use fully_connected_onednn? If so, we need enforce fully_connected_onednn to make it meet moe_expert_impl requirement, such as post-ops.

sshlyapn · 2025-05-19T08:20:05Z

src/common/transformations/include/ov_ops/moe_expert.hpp

+
+private:
+    Config m_config{};
+    std::vector<ConstsPerExpert> m_consts;


Are there any reasons why these constants are not exposed as nodes in the graph, but instead encapsulated within this operation in that way?

The major reason is that there will many nodes in the graph, each expert will have 3 fc(up/gate/down), each fc has 3 consts(weight/scale/zp), each layer will have 128 experts, and the model have 48 layers. The total number of these const node is about 55,000. Moving them inside the op will be both beneficial for common transformations in core part and building stage in plugin.

src/common/transformations/include/ov_ops/moe_expert.hpp

praasz · 2025-05-20T09:57:19Z

src/common/transformations/include/ov_ops/moe_expert.hpp

+    const std::vector<ConstsPerExpert>& get_consts() const {
+        return m_attrs.consts;
+    }


It can be removed just get attributes and const member.

The const node and config are frequently used in separate stages and different helper function will make the semantics more clear.

src/common/transformations/include/ov_ops/moe_expert.hpp

praasz · 2025-05-20T10:37:07Z

src/common/transformations/include/ov_ops/moe_expert.hpp

+    }
+
+    void add_consts(int expert_no, const ConstsPerExpert& consts) {
+        OPENVINO_ASSERT(expert_no == static_cast<int>(m_attrs.consts.size()));


Some explanation can be useful why add constant is not possible

Do you mean add constant one by one? The op may have 128 * 3 * 3 constant nodes and this will make it very inefficent.

luo-cheng2021 and others added 30 commits April 8, 2025 09:29

add perf

aa10cbe

perf for sdpa/pa

3a80081

add git ignore

d209f03

insert if for moe expert

45bf16e

add moeexpert support

795e323

fix moexpert precision is always f32

d7f2602

add moeexpert support for gpu

7f4b901

opt: 1, simplify subgraph inside moeexpert; 2, only compute skip flag…

76a7d5b

… when expert_no == 0 for gpu

opt: remove nonzero->split from subgraph into moeexpert for gpu

6079269

Support Qwen3 rms kernel for input with dynamic padding

c385c8f

Add test case

ada754a

WA: moe_expert wait all inputs ready

38ded44

fix incorrect output shape computation

f84303e

add fast path for expert mask computation if no padding

df0ca20

qwen3 moe compile model opt, from 150s to 70s in LNL (#66)

00e7d9a

* Build Subgraph in parallel to improve compile_model performance * SharedOpOptimization optimizes attribute visit --------- Co-authored-by: Tingqian Li <[email protected]>

Move FuseMoeExpert2 ahead of CommonOptimizations to decrease compilin…

019262b

…g time.

not use subgraph for moeexpert

994c094

fix scale/zp layout; first expert should not be inplace

22c93ee

merge all experts into one op

7c872a0

Optimize gather and index_add performance

75b9683

fix out_of_resource error on lunarlake

2d8eb4e

Move weigts from usm_host to usm_device memory

06e436c

Add ITT for MoE

02f2331

Optimize BMG first token due to index_add kernel

b6b5f1d

opt: merge all experts into one for batch1

9383141

opt: cl code for mlp_*

c7ef4ea

change weight back to ba

76e6ed7

small tune for lunarlake

8471f6b

fuse onehot into moe

818ba1b

not wait gpu for batch1

eed40eb

fix ci error

a011264

riverlijunjie marked this pull request as ready for review May 12, 2025 13:20

riverlijunjie requested review from a team as code owners May 12, 2025 13:20

riverlijunjie requested review from itikhono and removed request for a team May 12, 2025 13:20

itikhono requested review from CuriousPanCake, praasz and mitruska May 12, 2025 13:35

itikhono reviewed May 12, 2025

View reviewed changes

src/common/transformations/src/transformations/common_optimizations/fuse_moe_expert.cpp Outdated Show resolved Hide resolved

itikhono reviewed May 12, 2025

View reviewed changes

fix CI failure

0d9354f

peterchen-intel requested review from usstq and sshlyapn May 14, 2025 08:22

p-durandin reviewed May 15, 2025

View reviewed changes

src/plugins/intel_gpu/src/graph/impls/ocl_v2/moe_expert_down_cm.cm Outdated Show resolved Hide resolved

praasz reviewed May 15, 2025

View reviewed changes

sshlyapn reviewed May 19, 2025

View reviewed changes

Move cm kernel to cm directory

8d1f344

github-actions bot removed the category: build OpenVINO cmake script / infra label May 19, 2025

riverlijunjie and others added 2 commits May 20, 2025 09:36

Merge branch 'master' into gpu/qwen3_moe_cm

6df53a1

apply review comments

ff0c74f

praasz reviewed May 20, 2025

View reviewed changes

riverlijunjie and others added 3 commits May 21, 2025 00:40

Some reviewer comments

18312f2

Remove CM kernel and move to the following PR

e06f8de

apply review comments

f43d457


		int max_expert_num = static_cast<int>(config.expert_num), max_topk = static_cast<int>(config.topk), max_tokens = static_cast<int>(shape[0]);

		expert_mask.pred_flag.resize(max_expert_num, 0);

[TEST][GPU]qwen3 moe support #30448

Are you sure you want to change the base?

[TEST][GPU]qwen3 moe support #30448

Conversation

riverlijunjie commented May 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Details:

Moe fusion result

TODO:

Tickets:

Uh oh!

itikhono commented May 12, 2025

Uh oh!

Uh oh!

itikhono May 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mitruska commented May 12, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sshlyapn May 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

riverlijunjie commented May 7, 2025 •

edited

Loading

itikhono May 12, 2025 •

edited

Loading

sshlyapn May 19, 2025 •

edited

Loading

luo-cheng2021 May 21, 2025 •

edited

Loading