CPU-Optimized Kernels for Interleaved GGUF Weights (Following llama.cpp)

[DrJesseGlass]

llama.cpp achieves superior CPU performance through thread-optimized kernels that compute directly on GGUF's native weight layouts. Candle should follow this approach to match llama.cpp's CPU efficiency and support diverse GGUF architectures seamlessly.

GGUF has become the standard format for quantized models, with various internal layouts optimized for different architectures. llama.cpp's success comes from having kernels that compute directly on these native layouts, leveraging thread-optimized parallelization across attention heads and cache-friendly access patterns for interleaved data.

SmolLM3 is the first model we've encountered requiring this infrastructure - its interleaved Q/K layout revealed the opportunity.
As can be seen in: Add SmolLM3: Full and Quantized Implementation #3180. SmolLM3 is the first model in Candle requiring layout-aware kernels. Its GGUF files use an interleaved Q/K weight layout -- heads are stored in an even/odd pattern across 128-row blocks, split between first and second halves of the weight matrix.

GGUF Layout (optimized for llama.cpp's threading):

Block 0 (rows 0-127):    [H0_even, H1_odd, H2_even, H3_odd, ...]
Block 1 (rows 128-255):  [H4_even, H5_odd, H6_even, H7_odd, ...]
...
Second Half (rows 768+): Repeat pattern for remaining heads

Candle Expected Layout (sequential):

[H0_complete, H1_complete, H2_complete, H3_complete, ...]

I would like to implement CPU kernels with thread-optimized matmul for interleaved weight layouts (F16, Q8_0, and Q4_K formats).

Question for Maintainers

Is CPU-optimized threading for GGUF layouts a priority?

[ivarflakstad]

This could be a great improvement 👌
My only concern is how to implement it end-to-end with minimal pain.
Currently quantized tensors are opinionated from the moment they're loaded. Were you thinking of just doing a proof of concept first, or add additional ways to load (and represent) quantized tensors as well?

[DrJesseGlass]

I would start with a Proof of Concept. At this stage, proposing a new attention variant in candle-nn rather than modifying existing structures.

pub fn fused_interleaved_qk_attention(
    q_proj: &QTensor,
    k_proj: &QTensor,
    v_proj: &Tensor,
    xs: &Tensor,
    num_heads: usize,
    head_dim: usize,
    block_size: usize,
) -> Result<Tensor>

Model architects can explicitly choose between fused_interleaved_qk_attention, standard attention, or flash attention using feature flags or runtime dispatch. This new attention variant will call specialized CPU kernels in candle-core.

It should be easy to benchmark. And would start with support for Q8_0.

[ivarflakstad]

Btw we're experimenting with making quantized types a first class DType, which would remove the need for QTensor and QMatmul entirely.

Won't affect the POC ofc, just thought you'd like to know :)