Skip to content

ggml-cpu : add LoongArch64 LASX Q4_0 repack path#25991

Open
ztsubaki wants to merge 1 commit into
ggml-org:masterfrom
ztsubaki:la64
Open

ggml-cpu : add LoongArch64 LASX Q4_0 repack path#25991
ztsubaki wants to merge 1 commit into
ggml-org:masterfrom
ztsubaki:la64

Conversation

@ztsubaki

Copy link
Copy Markdown

Overview

LongArch has a LASX SIMD ISA similar to AVX2, which can be accelerated by repack. Previously, ggml-cpu only used LASX in the quantized dot product kernel on LoongArch. This PR adds a LASX kernel for the Q4_0_8x8_Q8_0 repack path for Q4_0 quantized models, with performance far exceeding the original dot product implementation and slightly higher than the OpenBLAS backend.

Additional information

quantize_mat_q8_0_4x8 is edited from x86 AVX2 implementation

Benchmark using Qwen 3.5 0.8B Q4_0 from unsloth on Loongson 3A6000 (4 core LA664)

Before(bec4772):

| model                          |       size |     params | backend    | threads |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | --------------: | -------------------: |
| qwen35 0.8B Q4_0               | 473.21 MiB |   752.39 M | CPU        |       4 |           pp512 |         79.59 ± 0.03 |
| qwen35 0.8B Q4_0               | 473.21 MiB |   752.39 M | CPU        |       4 |           tg128 |         37.44 ± 0.34 |

Current submit:

| model                          |       size |     params | backend    | threads |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | --------------: | -------------------: |
| qwen35 0.8B Q4_0               | 473.21 MiB |   752.39 M | CPU        |       4 |           pp512 |        132.13 ± 0.05 |
| qwen35 0.8B Q4_0               | 473.21 MiB |   752.39 M | CPU        |       4 |           tg128 |         40.16 ± 0.37 |

OpenBLAS backend:

| model                          |       size |     params | backend    | threads |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | --------------: | -------------------: |
| qwen35 0.8B Q4_0               | 473.21 MiB |   752.39 M | BLAS       |       4 |           pp512 |        131.72 ± 3.65 |
| qwen35 0.8B Q4_0               | 473.21 MiB |   752.39 M | BLAS       |       4 |           tg128 |         36.27 ± 0.30 |

Compared to before, pp512 has a 1.66x speedup, tg128 has a 1.07x speedup, and compared to OpenBLAS, pp512 is almost the same, while tg128 has a 1.11x speedup.

The perplexity was verified using the top 10% of the wiki-test2 test set on the same model and the same CPU.

CPU backend(using new LASX repack path):

Final estimate: PPL = 20.6118 +/- 0.46939

OpenBLAS backend:

Final estimate: PPL = 20.5799 +/- 0.46865

Requirements

Use AI to read and review the code, but did not use AI to generate the code.

choose q4_0_8x8_q8_0 repack when building with LASX support, and add
optimized LASX repack gemm, gemv for it.
@ztsubaki
ztsubaki requested a review from ggerganov as a code owner July 22, 2026 10:04
@github-actions github-actions Bot added the ggml changes relating to the ggml tensor library for machine learning label Jul 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ggml changes relating to the ggml tensor library for machine learning

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant