Fix i2_s ARM CPU kernel: correct 128/32 weight-group layout (fixes #585)

[Alexander Hoffman (EUPHEMEME)]

Fixes #585.

What

The i2_s ternary CPU dot-product kernel unpacked weights with a 64-weight / 16-byte group layout, but the gguf packs them in the 128-weight / 32-byte layout (the one dequantize_row_i2_s uses). It read every weight matrix in scrambled order → wrong-but-finite output → garbage generation on all ARM CPU inference. On Apple Silicon the default Metal backend hides it (it dequantizes weights, so uses the correct layout); force CPU with -ngl 0 and it reproduces, exactly like Jetson / ARM-Linux.

Fix

Replace the ARM branch of the ggml_vec_dot_i2_i8_s dispatcher with a correct 128/32-layout NEON decode (dotprod path + scalar fallback). The x86/AVX path is unchanged.

Validation

• First divergent op vs. the Metal reference was [3] MUL_MAT "Qcur-0" (first ternary matmul, layer 0). With the fix, the full CPU forward pass is bit-identical to the Metal/dequant path, modulo the expected int8-vs-f16 activation-quantization difference (~0.3–1% per element).
• BitNet-b1.58-2B-4T on a Jetson Orin Nano (aarch64, clang 14 / gcc 11) goes from !!!! to coherent English.

See #585 for the full op-by-op trace and the in-kernel proof (the NEON arithmetic is correct; only the unpack layout was wrong).

Follow-up (not in this PR): the unused _1xN / _Nx1 / _1x4_32W ARM sub-kernels and the ARM branch of quantize_i2_s still assume the 64/16 layout — dead on the inference read path for this model, but worth aligning.

[Alexander Hoffman (EUPHEMEME)]

@microsoft-github-policy-service agree

[Alexander Hoffman (EUPHEMEME)]

Flagging that this is now an incomplete fix for main, and offering to extend it.

When this PR was opened, the ARM QK_I2_S assumption lived only in src/ggml-bitnet-mad.cpp. As of the currently pinned submodule (Eddie-Wang1120/llama.cpp @ 390c3077), the same

#elif defined(__ARM_NEON)
#include <arm_neon.h>
#define QK_I2_S 64

block now also appears in:

• ggml/src/ggml-cpu/quants.c (lines 1294–1298)
• ggml/src/ggml-cpu/ggml-cpu-i2s.c (lines 15–20)

Both carry the identical 128 / 64 / 128 pattern, so merging this PR as-is would fix one of three copies and leave ARM decoding against the wrong block size in the other two.

Happy to extend this PR to cover both submodule sites, or to split them into a separate PR against the submodule repo — whichever you'd prefer. Just let me know.

Separately, while testing this I found a live SIGSEGV on any i2_s model at n_ubatch >= 32 on builds that link BLAS (the default on macOS via Accelerate). It's independent of this PR — I'll open it as its own issue rather than widen the scope here.