Eval bug: GLM-5.2 (glm_moe_dsa) dense-MLA CUDA path produces subtly corrupted output for ANY real transformer layer offloaded to GPU (partial coherent text mixed with garbage)

[maninara]

Name and Version

version: b10087 (1f66c3c)
Built with: cmake -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120a-real -DGGML_CUDA_FORCE_CUBLAS=ON

Operating systems

Linux

GGML backends

CUDA

Hardware

• 2x NVIDIA RTX PRO 6000 Blackwell Workstation Edition (96GB VRAM each, SM120/compute capability 12.0)
• Intel Xeon 6787P (86 cores / 172 threads)
• Supermicro X14SBM-TP4F, 256GB DDR5 RAM
• CUDA Toolkit 12.9, nvidia-open driver

Models

unsloth/GLM-5.2-GGUF, UD-IQ2_M (222.18 GiB, 2.53 BPW)
arch: glm-dsa, n_layer=78 (real transformer layers, blk.0-77), n_layer_all=79 (blk.78 is the MTP/NextN draft layer, tensors marked unused per #24770)

Problem description & steps to reproduce

Testing with llama-server, single short prompt ("hello"), n_predict=30:

​```
-ngl 0 → correct, coherent English reasoning output
-ngl 1 → offloads output layer only → correct
-ngl 2 → offloads output layer + blk.78 (MTP layer, unused tensors) → correct
-ngl 3 → offloads output layer + blk.78 + blk.77 (first REAL transformer layer to hit GPU) → CORRUPTED

Exact garbled output at -ngl 3:
​```
"edReader is? Hmm, letAsc晴天Hello! I'm an AI assistant basedION6TK6T7QY7N5C0U"
Note this is NOT pure random noise — it contains a fully coherent English fragment ("Hello! I'm an AI assistant based") embedded within garbage tokens (random alphanumeric strings, an unrelated Chinese phrase). This pattern suggests a numerical/masking correctness bug (e.g. attention mask, RoPE application, or KV-cache indexing) in the dense-MLA CUDA path rather than random corruption, memory corruption, or a tokenizer/detokenizer issue.

Key observations ruling out other causes:

• No CUDA errors, NaN/Inf warnings, asserts, or crashes in logs — process completes normally with wrong output
• Reproducible with -DGGML_CUDA_FORCE_CUBLAS=ON (rules out the Blackwell MMQ sharedMemPerBlockOptin driver bug, Eval bug: Fatal MMQ crashes on Blackwell (RTX 5090/5080) due to unhandled sharedMemPerBlockOptin driver bug #23385)
• Reproducible with -DCMAKE_CUDA_ARCHITECTURES=120a-real (real SASS, no PTX JIT — rules out JIT compilation issues)
• Reproducible with very short prompts (13 prompt tokens) and small n_predict — this does NOT require the "long prefill + high ubatch" conditions described in Eval bug: GLM 5.x crashes/gibberish with high ubatch and long prompts #23574, suggesting this may be a related-but-distinct instance of the same underlying dense-MLA-on-CUDA correctness issue, triggered simply by any real transformer layer running on GPU
• CPU-only inference (-ngl 0) is always correct
• The specific boundary is NOT about a "cursed" layer index — it's the transition from "no real attention layers on GPU" (-ngl ≤2, only output+MTP-draft-layer offloaded) to "at least one real attention layer on GPU" (-ngl ≥3)

This points to the dense-MLA CUDA kernel path used by GLM-5.2 (since the DSA lightning-indexer runtime isn't implemented yet, model falls back to llama_model_deepseek2::graph per #24770) having a correctness bug specific to Blackwell (SM120), or possibly affecting CUDA generally but only surfacing here due to GLM-5.2's specific MLA parameter shapes (q_lora_rank=2048, kv_lora_rank=512, key_length=576/mla=256, head_count_kv=1).

First Bad Commit

No response

Relevant log output

Logs

[glm52_ngl2_working.txt](https://github.com/user-attachments/files/30294026/glm52_ngl2_working.txt)

glm52_ngl3_broken.txt