ggml-cpu: add repack GEMM and GEMV for floating-point

[taimur-10x]

Summary

This PR adds repacking and GEMM/GEMV kernels for floating-point (FP16 and FP32) for RVV (with the zvfh extension).

Key Changes

• Added RVV kernels for GEMM with tiling:
  • 7 x {16, 32, 64, 128} (selected based on VLEN)
• Added RVV kernels for GEMV with tiling:
  • 1 x {16, 32, 64, 128} (selected based on VLEN)
• Added scalar functions for repacking. They support arbitrary tile sizes.
• Generic scalar fallbacks for GEMM/GEMV operations.
• ggml_quantize_mat_t is refactored to ggml_repack_mat_t to allow for a common interface for both quantization and floating-point repacking.
• Additional template parameter NB_ROWS added to select the number of rows to interleave for repacking. Previously, this was fixed at 4.

Tile Sizes

The repack operation interleaves N rows of activations with an interleave size of K, and M columns of weights with an interleave size of K.

NxK is fixed at 7x1. This introduces 7 accumulators with LMUL=4 (7 x 4 = 28 registers), each accumulating M results.

M is varied based on the available VLEN:

VLEN	Tile Size (N x M x K)
128	7 x 16 x 1
256	7 x 32 x 1
512	7 x 64 x 1
1024	7 x 128 x 1

M is the maximum number of values that can be loaded in (LMUL=2 for F16, LMUL=4 for F32).

Testing

Kernels were functionally tested on QEMU for VLENs (128-bit, 256-bit, 512-bit and 1024-bit) for a range of input sizes.

Benchmarking Results

End-to-end benchmarking on BananaPI-BPI F3 (VLEN=256) with llama-bench (Threads=8)).

Prefill / Prompt Processing (GEMM)

Tokens / Second

Model	Prompt Size	Repack GEMM (7x32)	Vec Dot
Tinyllama F16 1.1B	28	24.72	8.31
Tinyllama F16 1.1B	32	16.72	8.42
Tinyllama F16 1.1B	64	22.55	8.57
Tinyllama F16 1.1B	128	22.78	8.78
Tinyllama F16 1.1B	256	21.82	8.57
Tinyllama F16 1.1B	512	21.81	8.68

Model	Prompt Size	Repack GEMM (7x32)	Vec Dot
Tinyllama F32 1.1B	28	11.45	3.72
Tinyllama F32 1.1B	32	7.13	3.75
Tinyllama F32 1.1B	64	10.76	3.74
Tinyllama F32 1.1B	128	10.86	3.73
Tinyllama F32 1.1B	256	10.94	3.68
Tinyllama F32 1.1B	512	11.12	3.79

Result: ~2x-3x speedup over vec_dot

Decode (GEMV)

Tokens / Second

Model	Decode Size (Prompt=32)	Repack GEMV (1x32)	Vec Dot
Tinyllama F16 1.1B	10	3.37	3.11
Tinyllama F16 1.1B	16	3.29	3.45
Tinyllama F16 1.1B	32	3.12	3.25
Tinyllama F16 1.1B	64	3.23	3.27
Tinyllama F16 1.1B	100	3.04	3.15
Tinyllama F16 1.1B	128	3.09	3.2
Tinyllama F16 1.1B	256	3.15	3.19

Model	Decode Size (Prompt=32)	Repack GEMV (1x32)	Vec Dot
Tinyllama F32 1.1B	10	1.66	1.74
Tinyllama F32 1.1B	16	1.73	1.63
Tinyllama F32 1.1B	32	1.81	1.68
Tinyllama F32 1.1B	64	1.61	1.69
Tinyllama F32 1.1B	100	1.72	1.75
Tinyllama F32 1.1B	128	1.76	1.72
Tinyllama F32 1.1B	256	1.75	1.69

Result: No noticeable improvement, as decode remains memory-bound.

Perplexity

Calculated on BananaPI-BPI-F3 with VLEN=256

/build/bin/llama-perplexity -f /path/to/wiki.test.raw -m models/tinyllama-1.1B-f16.gguf

Model	Master	Repack
Tinyllama 1.1B F16	8.6971 +/- 0.05412	8.6969 +/- 0.05412

Additional Notes

• Current fallback model requires every architecture to have a scalar fallback for each implementation. This creates a clutter in arch-fallback.h as 7xMx1 is very RVV-specific tiling, and should not be used by other architectures.
• GEMM reaches peak performance when the prompt is a multiple of 7 (for example, prompt=28). To handle leftovers, it defaults to GEMV, which impacts performance. Ideally, there should be leftover Nx32 kernels which handle each leftover case from 2-6 leftover tokens.

Future Work

Subsequent PRs plan to add RVV kernels for quantization types, as well as extend existing quantization support to other VLENs.

References

The selection of the tiling 7x32x1 is based off of the mmt4d kernel in IREE for RVV: iree-org/iree#20263

[taimur-10x]

@ggerganov, could this be reviewed please? Thank you.

[ixgbe]

@taimur-10x , Could you please provide the download links for Tinyllama F16 1.1B and Tinyllama F32 1.1B models? Also, it would be really helpful if you could share the exact commands or script used to obtain these performance numbers (especially for the Prefill / Prompt Processing benchmarks with GEMM). I'd like to reproduce the results.
Thanks in advance! 🙌

[taimur-10x]

@taimur-10x , Could you please provide the download links for Tinyllama F16 1.1B and Tinyllama F32 1.1B models? Also, it would be really helpful if you could share the exact commands or script used to obtain these performance numbers (especially for the Prefill / Prompt Processing benchmarks with GEMM). I'd like to reproduce the results. Thanks in advance! 🙌

Sure, here's the link for the model: https://huggingface.co/TinyLlama/TinyLlama_v1.1
I believe this is in fp32. This can be converted to fp16 through llama-quantize.

All benchmarking was done through llama-bench on BananaPI-BPI-F3.

llama-bench -m models/tinyllama-1.1B-f16.gguf -p 32,64,128,256,512 -n 10,16,32,64,100,128,256 -t 8 -r 10

[ixgbe]

Could you please share the PPL comparison against the master branch? I personally use the following command for verification:
./build/bin/llama-perplexity -f /path/to/wiki.test.raw -m models/tinyllama-1.1B-f16.gguf
Download link: https://cosmo.zip/pub/datasets/wikitext-2-raw/

[taimur-10x]

Could you please share the PPL comparison against the master branch? I personally use the following command for verification: ./build/bin/llama-perplexity -f /path/to/wiki.test.raw -m models/tinyllama-1.1B-f16.gguf Download link: https://cosmo.zip/pub/datasets/wikitext-2-raw/

@ixgbe, here are the perplexity numbers for fp16 (on BananaPI-BPI-F3 with VLEN=256):

/build/bin/llama-perplexity -f /path/to/wiki.test.raw -m models/tinyllama-1.1B-f16.gguf

Model	Master	Repack
Tinyllama 1.1B F16	8.6971 +/- 0.05412	8.6969 +/- 0.05412

[taimur-10x]

@ggml-org/ggml-riscv