Tensor Parallelism Errors when using 2x Vega II Duo Cards and using q8_0 keys, turbo4 values

[chafey]

ToshLLM version

0.85.7

macOS version

26.5.2

Mac model, CPU, and RAM

MacPro7,1 / Xeon W 28 Core / 512 GB RAM

GPU and VRAM

2x Vega II Duo - Each card connected by its own IFB

Model and quantization

Qwen3 14B

What happened?

Pressing start server fails and shows error "engine exited with code 6 - see the log in Settings"

What did you expect to happen?

Server should start and I should be able to use it

Steps to reproduce

1. Start ToshLLM
2. Press "start server"
3. Observe error

Relevant settings

Split model across all GPUs (experimental): True
How to split it: By tensors
Measured suggestion: q8_0 keys, turbo4 values: applied

NOTE: It seems to be related to the measured suggestion only - if I don't apply that it works (but is not any faster that a single GPU)

Diagnostics or server log

========================================================
 ToshLLM 0.85.7 — server start (x86_64)
 engine : bundled (official)
 model  : Qwen3-14B-Q4_K_M.gguf
 GPUs detected:
    [0] AMD Radeon Pro Vega II Duo · 32 GB · peer group 16020746928242862272 (2 GPUs)
    [1] AMD Radeon Pro Vega II Duo · 32 GB · peer group 16020746928242862272 (2 GPUs)
    [2] AMD Radeon Pro Vega II Duo · 32 GB · peer group 16244674959255310838 (2 GPUs)
    [3] AMD Radeon Pro Vega II Duo · 32 GB · peer group 16244674959255310838 (2 GPUs)
 GPU select: split-all · split-mode tensor | force-VRAM-buffers: no
 settings: ngl=99 ncmoe=0 ctx=16384 fa=auto ctk=q8_0 ctv=turbo4 cacheRAM=2048
 dflash : not installed
 env: GGML_METAL_VRAM_RESERVE_MB=1024 GGML_METAL_DEVICES=4 TOSH_FA_AMD=1 TOSH_MGPU_EVENTS=1
 args: -m /Users/chafey/models/Qwen3-14B-Q4_K_M.gguf -ngl 99 -c 16384 -t 6 -fa 1 --host 127.0.0.1 --port 8080 --load-mode none --jinja -ctk q8_0 -ctv turbo4 --metrics --cache-ram 2048 --cache-reuse 256 --parallel 1 --split-mode tensor --path /Applications/ToshLLM.app/Contents/Resources/web-ui
========================================================
ggml_metal: device 0: AMD Radeon Pro Vega II Duo (peer group 16020746928242862272, bridged) probed SIMD-group width = 64 (32 = Apple/AMD RDNA, 64 = AMD GCN/Vega)
ggml_metal: wave64 mode (SIMD width 64): GPU prefill matmul, CPU decode/reductions for correct output
ggml_metal: wave64 decode ON: quantized/f16/bf16 mat-vec on GPU (see the allowlist in ggml_metal_library_get_pipeline_mul_mv)
ggml_metal: device 1: AMD Radeon Pro Vega II Duo (peer group 16020746928242862272, bridged) probed SIMD-group width = 64 (32 = Apple/AMD RDNA, 64 = AMD GCN/Vega)
ggml_metal: wave64 mode (SIMD width 64): GPU prefill matmul, CPU decode/reductions for correct output
ggml_metal: wave64 decode ON: quantized/f16/bf16 mat-vec on GPU (see the allowlist in ggml_metal_library_get_pipeline_mul_mv)
ggml_metal: device 2: AMD Radeon Pro Vega II Duo (peer group 16244674959255310838, bridged) probed SIMD-group width = 64 (32 = Apple/AMD RDNA, 64 = AMD GCN/Vega)
ggml_metal: wave64 mode (SIMD width 64): GPU prefill matmul, CPU decode/reductions for correct output
ggml_metal: wave64 decode ON: quantized/f16/bf16 mat-vec on GPU (see the allowlist in ggml_metal_library_get_pipeline_mul_mv)
ggml_metal: device 3: AMD Radeon Pro Vega II Duo (peer group 16244674959255310838, bridged) probed SIMD-group width = 64 (32 = Apple/AMD RDNA, 64 = AMD GCN/Vega)
ggml_metal: wave64 mode (SIMD width 64): GPU prefill matmul, CPU decode/reductions for correct output
ggml_metal: wave64 decode ON: quantized/f16/bf16 mat-vec on GPU (see the allowlist in ggml_metal_library_get_pipeline_mul_mv)
0.00.053.252 I cmn  common_param: common_params_print_info: ToshLLM engine 0.85.7 (app 0.85.7)
0.00.053.257 I cmn  common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.053.452 W srv  llama_server: -----------------
0.00.053.455 W srv  llama_server: CORS is set to allow all origins ('*') and no API key is set
0.00.053.455 W srv  llama_server: this can be a security risk (cross-origin attacks)
0.00.053.455 W srv  llama_server: more info: https://github.com/ggml-org/llama.cpp/pull/25655
0.00.053.455 W srv  llama_server: -----------------
0.00.054.768 I srv    load_model: loading model '/Users/chafey/models/Qwen3-14B-Q4_K_M.gguf'
0.00.054.833 W common_fit_params: not fitting params to free device memory: automatic fitting is not implemented for --split-mode tensor; continuing with the parameters as given
0.00.224.843 W load: control-looking token: 128247 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
0.04.428.439 W ggml_backend_metal_comm_init: reducing across 4 devices in 2 steps
0.04.446.981 I cmn          init: llama threadpool init, n_threads = 6
/Users/runner/work/toshllm/toshllm/vendor/llama.cpp/ggml/src/ggml-backend-meta.cpp:1013: ggml op not implemented: TURBO_WHT
WARNING: Using native backtrace. Set GGML_BACKTRACE_LLDB for more info.
WARNING: GGML_BACKTRACE_LLDB may cause native MacOS Terminal.app to crash.
See: https://github.com/ggml-org/llama.cpp/pull/17869
0   llama-server                        0x000000010578204b ggml_print_backtrace + 267
1   llama-server                        0x000000010587269d ggml_abort + 253
2   llama-server                        0x00000001057ae04c _ZL33ggml_backend_meta_get_split_stateR41ggml_backend_meta_simple_tensor_containerPK11ggml_tensorb + 11612
3   llama-server                        0x00000001057a5c09 _ZL41ggml_backend_meta_buffer_init_tensor_implR41ggml_backend_meta_simple_tensor_containerP11ggml_tensor + 137
4   llama-server                        0x00000001057a9547 _ZL36ggml_backend_meta_buffer_init_tensorP19ggml_backend_bufferP11ggml_tensor + 119
5   llama-server                        0x0000000105798b4b ggml_gallocr_alloc_graph + 1947
6   llama-server                        0x00000001057a0db4 ggml_backend_sched_alloc_graph + 292
7   llama-server                        0x000000010549397c _ZN13llama_context14process_ubatchERK12llama_ubatch14llm_graph_typeP22llama_memory_context_iR11ggml_status + 300
8   llama-server                        0x000000010549549b _ZN13llama_context6decodeERK11llama_batch + 2427
9   llama-server                        0x000000010549aeff llama_decode + 15
10  llama-server                        0x000000010527832d _Z23common_init_from_paramsR13common_paramsb + 1821
11  llama-server                        0x00000001050f363c _ZN19server_context_impl10load_modelER13common_params + 3884
12  llama-server                        0x0000000104ffe1a5 _Z12llama_serverR13common_paramsiPPc + 21509
13  llama-server                        0x0000000104ff8d48 _Z12llama_serveriPPc + 168
14  dyld                                0x00007ff809ae6b28 start + 3240

Before submitting

• I searched the existing issues and did not find the same problem.
• I removed usernames, local paths, API keys, prompts, and other private data from logs and screenshots.

[engeldlgado]

I'll look into it, thanks for reporting this. The logs are very helpful.

[engeldlgado]

Fixed in 0.85.8, building now.

Your note that it only happened with the measured suggestion applied is what pointed straight at it. TurboQuant rotates the attention output with an operation of its own, and the tensor split path did not know how to divide that one, so it aborted while allocating the graph. That is the TURBO_WHT line at the top of your backtrace. It had nothing to do with the Vega II Duo or with the split itself: by layers it never goes through that path, which is why layers worked and tensors did not.

One thing I want to be straight about: this machine has a single GPU, so I could not run it. The abort is gone by construction, but whether the split of that operation is numerically right is something only your machine can tell. When you get 0.85.8, the same test you already did is the one I need: q8_0 keys with turbo4 values, split by tensors, and check the answers read sane rather than just that the server starts.

If it comes out garbled, say so and I will take that operation off the split instead of dividing it.