Eval bug: Request for Qwen3-Next-80B-A3B Vulkan Inference Optimization

[engrtipusultan]

Name and Version

Details

llama-server --version
load_backend: loaded RPC backend from /home/tipu/Applications/llamacpp/libggml-rpc.so
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV RENOIR) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
load_backend: loaded Vulkan backend from /home/tipu/Applications/llamacpp/libggml-vulkan.so
load_backend: loaded CPU backend from /home/tipu/Applications/llamacpp/libggml-cpu-haswell.so
version: 7243 (13628d8)
built with cc (Ubuntu 11.4.0-1ubuntu1~22.04.2) 11.4.0 for x86_64-linux-gnu

Operating systems

Linux

GGML backends

Vulkan

Hardware

Details

Device Name AMD Radeon Graphics
PCI (domain:bus:dev.func) 0000:03:00.0
DeviceID:RevID 0x15E7.0xC1
OpenGL Driver Version Mesa 25.3.0 - kisak-mesa PPA
gfx_target_version gfx90c

GPU Type APU
Family Raven (RV)
ASIC Name Renoir
Chip Class GFX9
Shader Engine (SE) 1
Shader Array (SA/SH) per SE 1
CU per SA 8
Total CU 8
RenderBackendPlus (RB+) 2 (16 ROPs)
Peak Pixel Fill-Rate 32 GP/s
GPU Clock 200-2000 MHz
Peak FP32 2048 GFLOPS

VRAM Type DDR4
VRAM Bit Width 128-bit
VRAM Vendor Unknown
VRAM Size 16384 MiB
Memory Clock 400-1333 MHz
ResizableBAR Enabled
ECC Memory Not Supported

L1 Cache (per CU) 16 KiB
L2 Cache 1024 KiB (4 Banks)

Supported Power Profiles[
"3D_FULL_SCREEN",
"VIDEO",
"VR",
"COMPUTE",
"CUSTOM",
]

Models

Qwen3-Next-80B-A3B

Problem description & steps to reproduce

Qwen3-Next-80B-A3B current implementation is not optimized. It is much slower as compared to other A3B Qwen models. In coming weeks/months as per your free time, please support to make it optimized.

First Bad Commit

N/A

Relevant log output

Qwen3-Next-80B-A3B-Instruct llama-bench

Details

bash  llama-bench -m /home/tipu/AI/models/unsloth/Qwen3-Next/Qwen3-Next-80B-A3B-Instruct-UD-Q4_K_XL.gguf -ngl 99 --ubatch-size 128,512 --batch-size 2048 --mmap 0 -fa 0,1 --prio 3
load_backend: loaded RPC backend from /home/tipu/Applications/llamacpp/libggml-rpc.so
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV RENOIR) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
load_backend: loaded Vulkan backend from /home/tipu/Applications/llamacpp/libggml-vulkan.so
load_backend: loaded CPU backend from /home/tipu/Applications/llamacpp/libggml-cpu-haswell.so
| model                          |       size |     params | backend    | ngl | n_ubatch | fa | mmap |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | -: | ---: | --------------: | -------------------: |
| qwen3next ?B Q4_K - Medium     |  42.01 GiB |    79.67 B | Vulkan     |  99 |      128 |  0 |    0 |           pp512 |         32.24 ± 0.45 |
| qwen3next ?B Q4_K - Medium     |  42.01 GiB |    79.67 B | Vulkan     |  99 |      128 |  0 |    0 |           tg128 |          8.79 ± 0.00 |
| qwen3next ?B Q4_K - Medium     |  42.01 GiB |    79.67 B | Vulkan     |  99 |      128 |  1 |    0 |           pp512 |         32.28 ± 0.51 |
| qwen3next ?B Q4_K - Medium     |  42.01 GiB |    79.67 B | Vulkan     |  99 |      128 |  1 |    0 |           tg128 |          8.82 ± 0.01 |
| qwen3next ?B Q4_K - Medium     |  42.01 GiB |    79.67 B | Vulkan     |  99 |      512 |  0 |    0 |           pp512 |         35.20 ± 0.23 |
| qwen3next ?B Q4_K - Medium     |  42.01 GiB |    79.67 B | Vulkan     |  99 |      512 |  0 |    0 |           tg128 |          8.80 ± 0.01 |
| qwen3next ?B Q4_K - Medium     |  42.01 GiB |    79.67 B | Vulkan     |  99 |      512 |  1 |    0 |           pp512 |         35.16 ± 0.23 |
| qwen3next ?B Q4_K - Medium     |  42.01 GiB |    79.67 B | Vulkan     |  99 |      512 |  1 |    0 |           tg128 |          8.79 ± 0.01 |

build: 13628d8bd (7243)

Qwen3-30B-A3B-Thinking-2507 llama-bench More than double inference speed for pps and tg

Details

bash  llama-bench -m /home/tipu/AI/models/unsloth/Qwen3-Think-A3B-GGUF/Qwen3-30B-A3B-Thinking-2507-UD-Q4_K_XL.gguf -ngl 99 --ubatch-size 128,512 --batch-size 2048 --mmap 0 -fa 0,1 --prio 3
load_backend: loaded RPC backend from /home/tipu/Applications/llamacpp/libggml-rpc.so
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV RENOIR) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
load_backend: loaded Vulkan backend from /home/tipu/Applications/llamacpp/libggml-vulkan.so
load_backend: loaded CPU backend from /home/tipu/Applications/llamacpp/libggml-cpu-haswell.so
| model                          |       size |     params | backend    | ngl | n_ubatch | fa | mmap |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | -: | ---: | --------------: | -------------------: |
| qwen3moe 30B.A3B Q4_K - Medium |  16.49 GiB |    30.53 B | Vulkan     |  99 |      128 |  0 |    0 |           pp512 |         55.86 ± 0.91 |
| qwen3moe 30B.A3B Q4_K - Medium |  16.49 GiB |    30.53 B | Vulkan     |  99 |      128 |  0 |    0 |           tg128 |         20.83 ± 0.03 |
| qwen3moe 30B.A3B Q4_K - Medium |  16.49 GiB |    30.53 B | Vulkan     |  99 |      128 |  1 |    0 |           pp512 |         53.52 ± 0.67 |
| qwen3moe 30B.A3B Q4_K - Medium |  16.49 GiB |    30.53 B | Vulkan     |  99 |      128 |  1 |    0 |           tg128 |         20.72 ± 0.03 |
| qwen3moe 30B.A3B Q4_K - Medium |  16.49 GiB |    30.53 B | Vulkan     |  99 |      512 |  0 |    0 |           pp512 |         89.25 ± 0.20 |
| qwen3moe 30B.A3B Q4_K - Medium |  16.49 GiB |    30.53 B | Vulkan     |  99 |      512 |  0 |    0 |           tg128 |         20.87 ± 0.01 |
| qwen3moe 30B.A3B Q4_K - Medium |  16.49 GiB |    30.53 B | Vulkan     |  99 |      512 |  1 |    0 |           pp512 |         84.70 ± 0.53 |
| qwen3moe 30B.A3B Q4_K - Medium |  16.49 GiB |    30.53 B | Vulkan     |  99 |      512 |  1 |    0 |           tg128 |         20.75 ± 0.13 |

build: 13628d8bd (7243)

[jeffbolznv]

Hi, I've only barely looked at this model. It seemed like it had a ton of CONT ops in the model which made me wonder if the model implementation was complete. I'll look more closely tomorrow.

[engrtipusultan]

Thank you. I know you guys have been working hard on so many different fronts. I really appreciate you and all other maintainers and contributors for providing us with this community driven free software which has Vulkan based inference. This makes local AI accessible for everyone, even those with limited budgets.

[pwilkin]

I'm waiting for the CUMSUM/TRI ops merge to start actually optimizing the graph. There is likely quite a lot there to be shaved just by optimizing the CONT/RESHAPEs, but the focus was on correctness first since this was a very complex architecture to do.

[engrtipusultan]

I believe in your PR is marked for CUDA not vulkan

[jeffbolznv]

I thought the Vulkan backend was already accelerating all the ops, but I just noticed that this model is using ggml_diag() which is a very old op that none(?) of the backends (except CPU) currently support. I can add it. I thought maybe it would make sense to fold it into TRI but I see it takes as input a vector with ne[1]==1 which is different.

[pwilkin]

I believe in your PR is marked for CUDA not vulkan

I'm the author of the Qwen3 Next Llama.cpp implementation :)

@jeffbolznv it shouldn't really affect the efficiency too much though since it's only built once at the beginning of the entire graph (unless it poisons the graph the way a_ssm did previously, i.e. recurrently forcing splits, but that would be weird). I'll install Vulkan and try to dump the graph, see if I can spot anything.

[engrtipusultan]

I believe in your PR is marked for CUDA not vulkan

I'm the author of the Qwen3 Next Llama.cpp implementation

I know that and we are really thankful to you for that. What I meant was that #17584 had CUDA and NVIDIA mentioned not vulkan. I thought new ops might require separate implementation for vulkan acceleration.

[jeffbolznv]

It may be poisoning it a bit. There are splits for it at the beginning:

## SPLIT #0: Vulkan0 # 0 inputs
node #  0 (      FILL):               (view) (  64K) [Vulka         ] use=1:               leaf_0 (  64K) [Vulka         ]
node #  1 (       TRI):               node_1 (  64K) [Vulka         ] use=72:               (view) (  64K) [Vulka         ]
node #  2 (      FILL):               (view) (   0K) [Vulka         ] use=1:               leaf_1 (   0K) [Vulka         ]

## SPLIT #1: CPU # 1 inputs: [ (view) (   0K)] 
node #  3 (      DIAG):               node_3 (  64K) [  CPU         ] use=72:        CPU# (view)#0 (   0K) [ NULL         ]
node #  4 (   RESHAPE): cache_r_l0 (reshaped (  96K) [Vulka         ] use=3: conv_states_updated- (  96K) [Vulka         ]
node #  5 (      VIEW): cache_r_l0 (reshaped (  96K) [Vulka         ] use=1: cache_r_l0 (reshaped (  96K) [Vulka         ]

## SPLIT #2: Vulkan0 # 2 inputs: [ (view) (   0K)] [ (view) (   0K)] 
node #  6 (     SCALE): cache_r_l0 (reshaped (  96K) [Vulka         ] use=0: cache_r_l0 (reshaped (  96K) [Vulka         ]
node #  7 (      VIEW):               (view) (   0K) [  CPU         ] use=72:               leaf_3 (   0K) [  CPU         ]
node #  8 (  GET_ROWS):        conv_states-0 (  96K) [Vulka         ] use=1: cache_r_l0 (reshaped (  96K) [Vulka         ]    Vulkan0# (view)#0 (   0K) [ NULL         ]
node #  9 (      VIEW):               (view) (   0K) [  CPU         ] use=72:               leaf_3 (   0K) [  CPU         ]
node # 10 (  GET_ROWS):              node_10 (   0K) [Vulka         ] use=1: cache_r_l0 (reshaped (  96K) [Vulka         ]    Vulkan0# (view)#0 (   0K) [ NULL         ]
node # 11 (      VIEW):    cache_r_l0 (view) (   0K) [Vulka         ] use=1: conv_states_updated- (  96K) [Vulka         ]
node # 12 (       CPY): cache_r_l0 (view) (c (   0K) [Vulka         ] use=0:              node_10 (   0K) [Vulka         ]    cache_r_l0 (view) (   0K) [Vulka         ]
node # 13 (   RESHAPE): conv_states_reshaped (  96K) [Vulka         ] use=1:        conv_states-0 (  96K) [Vulka         ]

## SPLIT #3: CPU # 0 inputs
node # 14 (  GET_ROWS):   model.embed_tokens (   1M) [  CPU         ] use=2:    token_embd.weight ( 166M) [  CPU         ]               leaf_6 (   0K) [  CPU         ]

## SPLIT #4: Vulkan0 # 30 inputs: [model.embed_tokens (   1M)] [node_3 (view) (  31K)] [node_3 (  64K)] [node_3 (view) (  31K)] [node_3 (view) (  31K)] [leaf_65 (   0K)] [leaf_69 (   1K)] [leaf_71 (   1K)] [leaf_73 ( 128K)] [node_3 (view) (  31K)] [node_3 (view) (  31K)] [node_3 (view) (  31K)] [node_3 (view) (  31K)] [node_3 (view) (  31K)] [node_3 (view) (  31K)] [node_3 (view) (  31K)] [node_3 (view) (  31K)] [node_3 (view) (  31K)] [node_3 (view) (  31K)] [node_3 (view) (  31K)] [node_3 (view) (  31K)] [node_3 (view) (  31K)] [node_3 (view) (  31K)] [node_3 (view) (  31K)] [node_3 (view) (  31K)] [node_3 (view) (  31K)] [node_3 (view) (  31K)] [node_3 (view) (  31K)] [node_3 (view) (  31K)] [node_3 (view) (  31K)] 
node # 15 (  RMS_NORM):               norm-0 (   1M) [Vulka         ] use=1: Vulkan0#model.embed_ (   1M) [ NULL         ]
node # 16 (       MUL):          attn_norm-0 (   1M) [Vulka         ] use=2:               norm-0 (   1M) [Vulka         ] blk.0.attn_norm.weig (   8K) [Vulka         ]
node # 17 (   MUL_MAT): linear_attn_mixed_qk (   6M) [Vulka         ] use=1:  blk.0.ssm_in.weight (   7M) [Vulka         ]          attn_norm-0 (   1M) [Vulka         ]

then also one much later:

node #7689 (      CONT): v_beta-32 (cont) (tr (   2M) [Vulka         ] use=1: v_beta-32 (cont) (tr (   2M) [Vulka         ]
node #7690 (      VIEW):        node_3 (view) (  31K) [  CPU         ] use=4:               node_3 (  64K) [  CPU         ]

## SPLIT #5: Vulkan0 # 13 inputs: [node_3 (view) (  31K)] [node_3 (view) (  31K)] [node_3 (view) (  31K)] [node_3 (view) (  31K)] [node_3 (view) (  31K)] [node_3 (view) (  31K)] [node_3 (view) (  31K)] [node_3 (view) (  31K)] [node_3 (view) (  31K)] [node_3 (view) (  31K)] [node_3 (view) (  31K)] [node_3 (view) (  31K)] [leaf_901 (   0K)] 
node #7691 (    REPEAT):            node_7691 (   1M) [Vulka         ] use=1: Vulkan0#node_3 (view (  31K) [ NULL         ]
node #7692 (      VIEW):            k_conv-32 (   3M) [Vulka         ] use=1:      conv_qkv_mix-32 (   4M) [Vulka         ]
node #7693 (      CONT):     k_conv-32 (cont) (   1M) [Vulka         ] use=1:            k_conv-32 (   3M) [Vulka         ]
node #7694 (   RESHAPE): k_conv-32 (cont) (re (   1M) [Vulka         ] use=1:     k_conv-32 (cont) (   1M) [Vulka         ]

[jeffbolznv]

As an experiment, I replaced it with:

ggml_tensor * identity = ggml_tri(ctx0, ggml_tri(ctx0, ggml_fill_inplace(ctx0, ggml_new_tensor_2d(ctx0, GGML_TYPE_F32, ubatch.n_seq_tokens, ubatch.n_seq_tokens), 1.0f), GGML_TRI_TYPE_UPPER_DIAG), GGML_TRI_TYPE_LOWER_DIAG);

then there are only 3 splits:

## SPLIT #0: Vulkan0 # 2 inputs: [ (view) (   0K)] [ (view) (   0K)] 
node #  0 (      FILL):               (view) (  64K) [Vulka         ] use=1:               leaf_0 (  64K) [Vulka         ]
node #  1 (       TRI):               node_1 (  64K) [Vulka         ] use=72:               (view) (  64K) [Vulka         ]
node #  2 (      FILL):               (view) (  64K) [Vulka         ] use=1:               leaf_1 (  64K) [Vulka         ]
node #  3 (       TRI):               node_3 (  64K) [Vulka         ] use=1:               (view) (  64K) [Vulka         ]
node #  4 (       TRI):               node_4 (  64K) [Vulka         ] use=72:               node_3 (  64K) [Vulka         ]
node #  5 (   RESHAPE): cache_r_l0 (reshaped (  96K) [Vulka         ] use=3: conv_states_updated- (  96K) [Vulka         ]
node #  6 (      VIEW): cache_r_l0 (reshaped (  96K) [Vulka         ] use=1: cache_r_l0 (reshaped (  96K) [Vulka         ]
node #  7 (     SCALE): cache_r_l0 (reshaped (  96K) [Vulka         ] use=0: cache_r_l0 (reshaped (  96K) [Vulka         ]
node #  8 (      VIEW):               (view) (   0K) [  CPU         ] use=72:               leaf_3 (   0K) [  CPU         ]
node #  9 (  GET_ROWS):        conv_states-0 (  96K) [Vulka         ] use=1: cache_r_l0 (reshaped (  96K) [Vulka         ]    Vulkan0# (view)#0 (   0K) [ NULL         ]
node # 10 (      VIEW):               (view) (   0K) [  CPU         ] use=72:               leaf_3 (   0K) [  CPU         ]
node # 11 (  GET_ROWS):              node_11 (   0K) [Vulka         ] use=1: cache_r_l0 (reshaped (  96K) [Vulka         ]    Vulkan0# (view)#0 (   0K) [ NULL         ]
node # 12 (      VIEW):    cache_r_l0 (view) (   0K) [Vulka         ] use=1: conv_states_updated- (  96K) [Vulka         ]
node # 13 (       CPY): cache_r_l0 (view) (c (   0K) [Vulka         ] use=0:              node_11 (   0K) [Vulka         ]    cache_r_l0 (view) (   0K) [Vulka         ]
node # 14 (   RESHAPE): conv_states_reshaped (  96K) [Vulka         ] use=1:        conv_states-0 (  96K) [Vulka         ]

## SPLIT #1: CPU # 0 inputs
node # 15 (  GET_ROWS):   model.embed_tokens (   1M) [  CPU         ] use=2:    token_embd.weight ( 166M) [  CPU         ]               leaf_6 (   0K) [  CPU         ]

## SPLIT #2: Vulkan0 # 6 inputs: [model.embed_tokens (   1M)] [leaf_65 (   0K)] [leaf_69 (   1K)] [leaf_71 (   1K)] [leaf_73 ( 128K)] [leaf_901 (   0K)] 
node # 16 (  RMS_NORM):               norm-0 (   1M) [Vulka         ] use=1: Vulkan0#model.embed_ (   1M) [ NULL         ]
node # 17 (       MUL):          attn_norm-0 (   1M) [Vulka         ] use=2:               norm-0 (   1M) [Vulka         ] blk.0.attn_norm.weig (   8K) [Vulka         ]
node # 18 (   MUL_MAT): linear_attn_mixed_qk (   6M) [Vulka         ] use=1:  blk.0.ssm_in.weight (   7M) [Vulka         ]          attn_norm-0 (   1M) [Vulka         ]

It is maybe marginally faster.

[pwilkin]

I know we and we are really thankful to you for that. What I meant was that #17584 had CUDA and NVIDIA mentioned not vulkan. I thought it might require separate implementation for vulkan acceleration.

Don't think so - as @jeffbolznv mentioned, the main cause of the inefficiency now is probably the number of RESHAPE/CONT/PERMUTE operations.

[jeffbolznv]

As a baseline, this is where performance is on 5090:

Z:\github\jeffbolznv\llama.cpp\build\bin\RelWithDebInfo>llama-bench.exe -fa 1 -n 128 -p 512 -r 10 --prio 1 -m c:\models\Qwen3-Next-80B-A3B-Instruct-Q2_K_L.gguf
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 5090 (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
load_backend: loaded Vulkan backend from Z:\github\jeffbolznv\llama.cpp\build\bin\RelWithDebInfo\ggml-vulkan.dll
load_backend: loaded CPU backend from Z:\github\jeffbolznv\llama.cpp\build\bin\RelWithDebInfo\ggml-cpu.dll
| model                          |       size |     params | backend    | ngl | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -: | --------------: | -------------------: |
| qwen3next ?B Q2_K - Medium     |  27.23 GiB |    79.67 B | Vulkan     |  99 |  1 |           pp512 |      3048.72 ± 15.50 |
| qwen3next ?B Q2_K - Medium     |  27.23 GiB |    79.67 B | Vulkan     |  99 |  1 |           tg128 |         74.15 ± 0.32 |

And here's output with GGML_VK_PERF_LOGGER=1:

| qwen3next ?B Q2_K - Medium     |  27.23 GiB |    79.67 B | Vulkan     |  99 |  1 |           tg128 |         46.47 ± 0.00 |
----------------
Vulkan Timings:
ADD: 57405 x 2.577 us
CONCAT: 18576 x 2.827 us
CONT: 156348 x 2.432 us
CPY: 9417 x 2.316 us
CUMSUM: 4644 x 2.193 us
EXP: 18576 x 2.198 us
FILL: 258 x 3.618 us
FLASH_ATTN_EXT dst(256,16,1,1),  q(256,1,16,1),  k(256,256,2,1),  v(256,256,2,1),  m(256,1,1,1): 1548 x 11.236 us
GET_ROWS: 9546 x 3.629 us
GLU: 12384 x 2.266 us
L2_NORM: 9288 x 2.205 us
MUL: 82044 x 2.481 us
MULTI_ADD ADD: 6192 x 4.236 us
MUL_MAT f32 m=1 n=128 k=1 batch=32: 4644 x 7.298 us (0.561241 GFLOPS/s)
MUL_MAT f32 m=128 n=128 k=1 batch=32: 4644 x 9.425 us (55.6273 GFLOPS/s)
MUL_MAT_ADD MUL_MAT_VEC q3_K m=2048 n=1 k=4096: 1419 x 10.044 us (1670.11 GFLOPS/s)
MUL_MAT_ID_MUL MUL_MAT_ID_VEC q3_K m=2048 n=10 k=512 n_expert=512: 6192 x 12.622 us (1659.79 GFLOPS/s)
MUL_MAT_ID_VEC q2_K m=512 n=10 k=2048 n_expert=512: 12384 x 10.145 us (2066.5 GFLOPS/s)
MUL_MAT_VEC bf16 m=1 n=1 k=2048: 6192 x 6.922 us (0.591582 GFLOPS/s)
MUL_MAT_VEC f32 m=1 n=1 k=128 batch=32: 9288 x 2.19 us (3.72563 GFLOPS/s)
MUL_MAT_VEC f32 m=128 n=1 k=1 batch=32: 9288 x 4.312 us (0.949753 GFLOPS/s)
MUL_MAT_VEC f32 m=128 n=1 k=128 batch=32: 9288 x 4.326 us (241.437 GFLOPS/s)
MUL_MAT_VEC f32 m=512 n=1 k=2048: 6192 x 8.056 us (260.252 GFLOPS/s)
MUL_MAT_VEC q2_K m=12288 n=1 k=2048: 4644 x 16.621 us (3027.39 GFLOPS/s)
MUL_MAT_VEC q2_K m=2048 n=1 k=4096: 4644 x 8.576 us (1955.89 GFLOPS/s)
MUL_MAT_VEC q2_K m=512 n=1 k=2048: 1548 x 4.597 us (456.054 GFLOPS/s)
MUL_MAT_VEC q2_K m=64 n=1 k=2048: 4644 x 4.551 us (57.5829 GFLOPS/s)
MUL_MAT_VEC q2_K m=8192 n=1 k=2048: 1548 x 12.885 us (2603.48 GFLOPS/s)
MUL_MAT_VEC q3_K m=2048 n=1 k=4096: 129 x 9.468 us (1771.72 GFLOPS/s)
MUL_MAT_VEC q4_K m=512 n=1 k=2048: 13932 x 4.627 us (453.105 GFLOPS/s)
MUL_MAT_VEC q6_K m=151936 n=1 k=2048: 129 x 257.681 us (2414.52 GFLOPS/s)
MUL_MAT_VEC q6_K m=2048 n=1 k=512: 6192 x 4.738 us (442.11 GFLOPS/s)
NEG: 9288 x 2.165 us
REPEAT: 29412 x 2.264 us
RMS_NORM_MUL RMS_NORM(128,32,1,1): 4644 x 4.361 us
RMS_NORM_MUL RMS_NORM(2048,1,1,1): 12513 x 4.591 us
RMS_NORM_MUL_ROPE RMS_NORM(256,16,1,1): 1548 x 7.178 us
RMS_NORM_MUL_ROPE_VIEW_SET_ROWS RMS_NORM(256,2,1,1): 1548 x 7.417 us
SCALE: 4788 x 3.707 us
SET_ROWS: 1548 x 2.211 us
SIGMOID: 12384 x 2.207 us
SILU: 9288 x 2.176 us
SOFTPLUS: 4644 x 2.17 us
SOLVE_TRI: 4644 x 2.19 us
SSM_CONV: 4644 x 2.762 us
SUB: 18576 x 2.262 us
TOPK_MOE_EARLY_SOFTMAX_NORM SOFT_MAX: 6192 x 5.289 us
TRI: 129 x 19.844 us
Total time: 2.13372e+06 us.

This isn't completely representative because the timing code breaks concurrency, but it gives a good idea of how many ops there are and how much they cost. So the CONTs are around 15% of the time, MUL+ADD are another 15% or so.

[pwilkin]

@jeffbolznv Just wondering - since your SOLVE_TRI kernel doesn't have the limitation to n=64, what would happen if you bumped chunk_size to 128? For large chunk sizes SOLVE_TRI will quickly dominate the calculations since it's O(n^3), but in this case it's so cheap that maybe we could afford bumping the chunk size?

Though to be honest those stats on 5090 are pretty good, were expecting lower.

[jeffbolznv]

Do you mean for prompt processing or for token gen? I think I saw n_tokens == 1 for token gen (not sure where this came from).

IIRC when I looked a few days ago, I saw n=k=1 for solve_tri for tg and n=k=64 for pp.

[pwilkin]

Yeah, prompt processing, token gen is recurrent case so n=k=1.