Test: Infinity Fabric Link peer transfer (TOSH_MGPU_PEER) on W6800X Duo / Vega II Duo

[engeldlgado]

Hey @Basten7, following up on something your rig is uniquely suited to test.

ToshLLM 0.83.1 ships an opt-in feature: for multi-GPU layer splits, when two AMD GPUs sit in the same Metal peer group (the two dies of an Infinity Fabric-bridged module), layer activations copy directly GPU→GPU over the Infinity Fabric Link instead of round-tripping through system RAM. It's off by default because it has never run on real bridged hardware yet, and a peer copy that silently returns wrong data has no fallback. Your Mac Pro is the one machine I know of with exactly the right hardware:

• 2× W6800X Duo (dies [5]+[6] and [7]+[8]), wave32 / RDNA2
• 1× Vega II Duo (dies [1]+[2]), wave64 / GCN

Each Duo has an on-module Infinity Fabric bridge, so its two dies should share a peerGroupID and light up this path.

How to enable

Set the env var before launching, same way you ran TOSH_MGPU_EVENTS=1 on #31:

TOSH_MGPU_PEER=1

Test 1: W6800X Duo (do this one first, it's the low-risk case)

The W6800X Duos are wave32, the same architecture I develop on, so the compute kernels are already well-tested here... the only new thing is the transport. That makes this the clean isolation test.

1. In Settings → Multi-GPU, restrict the split to exactly the 2 dies of one W6800X Duo (e.g. devices [5] and [6]), nothing else. That way every layer boundary crosses the on-module bridge.
2. Model: a dense model that needs both dies, e.g. Llama-3.3-70B-Instruct Q4_K_M (~40 GB, splits nicely across the 2×31 GB) or any dense 30 to 70B Q4 you already have. Please avoid MoE and avoid the 397B for this test... the 397B has that separate wave64 batch bug (Benchmark - AMD Radeon Pro Vega II - v0.81.61 vs v0.81.62 over differents quant #31), and MoE muddies the hand-off pattern.
3. Run a normal chat, a few hundred tokens.
4. Repeat the exact same run without TOSH_MGPU_PEER=1, and once more with TOSH_MGPU_EVENTS=1 instead, for comparison.

What to look for

• Log line: Infinity Fabric peer transfer enabled: <src> -> <dst> (peer group N). If it appears, the two dies matched a peer group and the path engaged. If it does not appear, they didn't share a peerGroupID and it fell back to staging (that's data too, not a failure).
• Output coherent? It must be identical quality to default. If it garbles or 0000, that's the bug: drop the token and report.
• tg (t/s) with the flag vs without vs TOSH_MGPU_EVENTS=1.

Test 2: Vega II Duo (only after Test 1)

Same procedure on dies [1]+[2], but heads-up: these are wave64, which has a separate, still-open batch/perplexity issue (#48), so if output garbles here it may be the wave64 path rather than the peer transport. That's why Test 1 (wave32) is the isolation baseline. For the model, use something you've already confirmed coherent on a single Vega die first (a normal dense model, not the 397B), then repeat split across the two Vega dies with and without the flag.

What I'd love back

A small table is plenty:

config	peer log line?	coherent?	tg (t/s)
W6800X Duo, default	n/a
W6800X Duo, TOSH_MGPU_PEER=1	yes/no
W6800X Duo, TOSH_MGPU_EVENTS=1
Vega II Duo, default	n/a
Vega II Duo, TOSH_MGPU_PEER=1	yes/no

No need for a big sweep, one dense model and a couple of configs per Duo. If the peer path is coherent and tg holds or improves on the W6800X Duo, that's enough to graduate it out of opt-in. Thanks as always for the hardware and the patience.

[Basten7]

I'd be happy to test it.

I can't test this from the GUI because TOSH_MGPU_EVENTS=1 is not being passed to the server process.
The GUI launches with:
env:
GGML_METAL_VRAM_RESERVE_MB=1024
TOSH_FA_AMD=1

args:
-m /Volumes/NM790-4To/Models/Qwen3.5-27B-F16.gguf
-ngl 99
-c 16384
-t 6
-fa auto
--host 127.0.0.1
--port 8080
--no-mmap
--jinja
--cache-ram 2048
--cache-reuse 256
--parallel 1
--split-mode layer
--path /Applications/ToshLLM.app/Contents/Resources/test-ui
TOSH_MGPU_EVENTS=1 is missing from the environment, so I can't enable or test the new multi-GPU peer-copy path from the GUI.

So I will continue with the command

[Basten7]

A few comment after...

After reboot my Mac, finally I was able run some test.

A small table is all that's needed:

Configuration	Peer log line?	Output coherent?	Throughput (t/s)
W6800X Duo (default)	n/a	yes	9
W6800X Duo (TOSH_MGPU_PEER=1)	Yes	Yes	9
W6800X Duo (TOSH_MGPU_EVENTS=1)	no	yes	8
Vega II Duo (default)	n/a	yes	10
Vega II Duo (TOSH_MGPU_PEER=1)	No	yes	10

I have some good news with 2 Duo.

I tested with:
/Applications/ToshLLM.app/Contents/Resources/bin/llama-server \
-c 65568 -t 12 -fa on \
--host 127.0.0.1 --port 8082 \
--no-mmap --split-mode layer \
--temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.00 \
-m /Volumes/NM790-II/Qwen3.5-27B-F16.gguf \
-lv 4

Results

Configuration	Peer log line?	Output coherent?	Throughput (t/s)
2× W6800X Duo (TOSH_MGPU_PEER=1)	Yes	Yes	7
2× W6800X Duo (TOSH_MGPU_EVENTS=1)	No	Yes	8

Logs

With 2× W6800X Duo (4 GPUs):
ggml_metal_cpy_xdev_peer: Infinity Fabric peer transfer enabled: AMD Radeon PRO W6800X Duo -> AMD Radeon PRO W6800X Duo (peer group 12073373929257557946)
ggml_metal_cpy_xdev_peer: Infinity Fabric peer transfer enabled: AMD Radeon PRO W6800X Duo -> AMD Radeon PRO W6800X Duo (peer group 12073373929257557946)
ggml_metal_cpy_xdev_peer: Infinity Fabric peer transfer enabled: AMD Radeon PRO W6800X Duo -> AMD Radeon PRO W6800X Duo (peer group 12073373929257557946)

With 1× W6800X Duo (2 GPUs):
ggml_metal_cpy_xdev_peer: Infinity Fabric peer transfer enabled: AMD Radeon PRO W6800X Duo -> AMD Radeon PRO W6800X Duo (peer group 12073373929257557946)

Conclusion

So the Infinity Fabric peer-transfer path is definitely being activated.
Interestingly, performance with 4 GPUs (2× W6800X Duo) remains very close to the 2-GPU (1× Duo) configuration. That suggests the Infinity Fabric peer transfers are effectively avoiding the host-memory bottleneck, keeping scaling overhead low.

[engeldlgado]

@Basten7 even better after your edit, thank you. This is a great result, so let me update my read.

After the reboot the peer path is coherent, not erroring on the W6800X Duo (1 module / 2 dies: peer line Yes, coherent, 9 t/s, matching default). Scratch my earlier "it errored" reading, that was the pre-reboot state. This is the first confirmation the Infinity Fabric peer transport actually works on real bridged hardware.

And the 2× W6800X Duo (4 GPUs) run is the headline: the peer path engages across all four dies in a single peer group (12073373929257557946), stays coherent, and 4-GPU throughput holds close to the 2-GPU config. That is exactly the signature of the transfers avoiding the host round-trip, which is the whole point of the path. Really nice to see it hold across two bridged modules, not just the two dies of one.

On the peer-vs-events numbers (7 vs 8 t/s at 4 GPUs, 9 vs 8 at 2): that's within noise here, and it will stay that way, because F16 makes decode memory-bound. Qwen3.5-27B in F16 is ~2 bytes/param, so ~54 GB of weights read per token, and at the W6800X's 512 GB/s that caps you near 9 t/s regardless of the transport. To actually separate peer from events you want a faster-decoding model where the per-token cross-GPU sync stall is a larger fraction of each token. I'd suggest Llama-3.3-70B-Instruct Q4_K_M: dense (clean per-layer hand-offs), 42.5 GB so it still spans both dies, and notably smaller than your 27B F16 while being a much bigger model.

One thing I'd love you to run: the same peer test on the Vega II Duo. There the peer line was still No, which is odd since it also has an Infinity Fabric Link (dual Vega 20, 84 GB/s peer-to-peer), so I'd expect a peer group too. Could you repeat it on the Vega Duo (ideally with the Q4 model above) and paste the Vega II Duo + TOSH_MGPU_PEER=1 log? The line only prints on a successful peer copy, so that log tells me whether the driver reports no shared peerGroupID there, the remote view came back nil, or the split just didn't route a hand-off between the two Vega dies. If the Vega Duo forms a peer group like the W6800X Duos did, that closes the other half of the validation.

Bottom line: on your hardware the path is now validated as correct, across 2 and 4 GPUs, which is exactly the bar it needed to start graduating out of opt-in. Genuinely useful, thank you.

(FYI 0.83.2 now prints TOSH_MGPU_PEER / TOSH_MGPU_EVENTS and the device list in the startup env line and accepts them from Settings → Extra arguments, so this is all doable from the GUI now, no CLI needed.)

[Basten7]

Peer test on the Vega II Duo is Ok

0.18.217.812 I cmn common_init_: warming up the model with an empty run - please wait ... (--no-warmup to disable)
0.18.390.323 I ggml_metal_cpy_xdev_peer: Infinity Fabric peer transfer enabled: AMD Radeon Pro Vega II Duo -> AMD Radeon Pro Vega II Duo (peer group 9143958961641079496)

[engeldlgado]

@Basten7 that's the full set, thank you. The Vega II Duo forms its own peer group (9143958961641079496) and engages coherently, so both your bridged families are validated now: W6800X Duo on wave32 (2 and 4 GPUs, one peer group) and Vega II Duo on wave64. The earlier No on the Vega and the error on the W6800X were both just pre-reboot state, so a clean reboot is the one thing to note for anyone reproducing.

One more favor, and this is the one that settles the speed story: could you run a couple of your other dense models with the peer path off vs on, same prompt back to back?

• default (off): TOSH_MGPU_PEER unset
• peer on: TOSH_MGPU_PEER=1

Any dense model you already have works; the more it has to split across the dies, the better. A faster-decoding one (Q4 or Q5 rather than F16) will show the difference most clearly, since on a heavy F16 the per-token weight read swamps the transfer and the two land within noise. What I'm after is the t/s delta between off and on on the same model, that's what tells me how much the Infinity Fabric path actually buys on your hardware.

On my side this is already enough to move it out of opt-in: I'll ship it default-on when a peer group is present, with TOSH_MGPU_PEER_DISABLE=1 to turn it off. Your speed numbers will just tell me whether to celebrate it loudly or quietly. Thanks again, the full matrix from you is exactly what this needed.

[Basten7]

I was unable to reproduce the crash after rebooting the system.
Therefore, the problem seems to be isolated to my global session.

Log of the error:
0.00.037.234 W srv llama_server: -----------------
0.00.038.674 I srv load_model: loading model '/Volumes/NM790-II/Qwen3.5-27B-F16.gguf'
0.19.659.921 I srv load_model: initializing, n_slots = 4, n_ctx_slot = 65792, kv_unified = 'true'
0.19.686.607 I srv llama_server: model loaded
0.19.686.615 I srv llama_server: listening on http://127.0.0.1:8082
0.47.903.360 I slot get_availabl: id 3 | task -1 | selected slot by LRU, t_last = -1
0.47.903.433 I slot launch_slot_: id 3 | task 0 | processing task, is_child = 0
0.52.743.422 I slot print_timing: id 3 | task 0 | prompt processing, n_tokens = 123, progress = 0.19, t = 4.84 s / 25.41 tokens per second
1.02.324.344 E ggml_metal_synchronize: error: command buffer 0 failed with status 5
1.02.324.358 E error: Caused GPU Timeout Error (00000002:kIOAccelCommandBufferCallbackErrorTimeout)
1.02.325.026 E ggml_metal_graph_compute: backend is in error state from a previous command buffer failure - recreate the backend to recover
1.02.325.032 E graph_compute: ggml_backend_sched_graph_compute_async failed with error -1
1.02.325.033 E process_ubatch: failed to compute graph, compute status: -1
1.02.325.037 W decode: removing memory module entries for seq_id = 3, pos = [123, +inf)
1.02.325.044 E llama_decode: failed to decode, ret = -3
1.02.325.046 E srv decode: Compute error. off = 0, n_batch = 2048, ret = -3
1.02.325.049 E srv send_error: task id = 0, error: Compute error.
1.02.325.054 I slot release: id 3 | task 0 | stop processing: n_tokens = 625, truncated = 0
1.02.325.066 W srv stop: cancel task, id_task = 0
1.02.325.272 E srv update_slots: decode() failed: Compute error.
^C2.01.946.941 I srv operator(): operator(): cleaning up before exit...

[Basten7]

Tosh log is ok with the env config:

In order to show the infinite fabric Tag the log level need to set at 4.

ToshLLM 0.83.2 — server start (x86_64)
engine : bundled (official)
model : llama3.1-70b-q4_0.gguf
GPUs detected:
[0] AMD Radeon Pro 580 · 8 GB · EXTERNAL/eGPU
[1] AMD Radeon Pro Vega II Duo · 31 GB
[2] AMD Radeon Pro Vega II Duo · 31 GB
[3] AMD Radeon PRO W6800 · 29 GB
[4] AMD Radeon PRO W6800X · 31 GB
[5] AMD Radeon PRO W6800X Duo · 31 GB
[6] AMD Radeon PRO W6800X Duo · 31 GB
[7] AMD Radeon PRO W6800X Duo · 31 GB
[8] AMD Radeon PRO W6800X Duo · 31 GB
GPU select: default (macOS picks) | force-VRAM-buffers: no
settings: ngl=99 ncmoe=0 ctx=16384 fa=auto ctk=f16 ctv=f16 cacheRAM=2048
dflash : not installed
env: GGML_METAL_VRAM_RESERVE_MB=1024 GGML_METAL_DEVICE_LIST=7,8 TOSH_FA_AMD=1 TOSH_MGPU_PEER=1
args: -m /Volumes/NM790-4To/Models/llama3.1-70b-q4_0.gguf -ngl 99 -c 16384 -t 6 -fa auto --host 127.0.0.1 --port 8080 --no-mmap --jinja --cache-ram 2048 --cache-reuse 256 --parallel 1 --split-mode layer --path /Applications/ToshLLM.app/Contents/Resources/test-ui
ggml_metal: probed SIMD-group width = 32 (32 = Apple/AMD RDNA, 64 = AMD GCN/Vega)
ggml_metal: probed SIMD-group width = 32 (32 = Apple/AMD RDNA, 64 = AMD GCN/Vega)
0.00.033.469 I cmn common_param: common_params_print_info: ToshLLM engine 0.83.2 (app 0.83.2)
0.00.033.474 I cmn common_param: common_params_print_info: verbosity = 3 (adjust with the -lv N CLI arg)
0.00.033.752 W srv llama_server: -----------------
0.00.033.755 W srv llama_server: CORS is set to allow all origins ('*') and no API key is set
0.00.033.756 W srv llama_server: this can be a security risk (cross-origin attacks)
0.00.033.756 W srv llama_server: more info: ggml-org/llama.cpp#25655
0.00.033.756 W srv llama_server: -----------------
0.00.035.057 I srv load_model: loading model '/Volumes/NM790-4To/Models/llama3.1-70b-q4_0.gguf'
0.12.979.987 I srv load_model: initializing, n_slots = 1, n_ctx_slot = 16384, kv_unified = 'false'
0.12.983.783 I srv llama_server: model loaded
0.12.983.796 I srv llama_server: listening on http://127.0.0.1:8080
0.23.041.894 I slot get_availabl: id 0 | task -1 | selected slot by LRU, t_last = -1
0.23.041.929 I slot launch_slot_: id 0 | task 0 | processing task, is_child = 0
0.32.776.740 I slot print_timing: id 0 | task 0 | n_decoded = 100, tg = 11.23 t/s, tg_3s = 11.23 t/s
0.35.832.031 I slot print_timing: id 0 | task 0 | n_decoded = 134, tg = 11.20 t/s, tg_3s = 11.13 t/s
0.38.887.409 I slot print_timing: id 0 | task 0 | n_decoded = 168, tg = 11.19 t/s, tg_3s = 11.13 t/s
0.41.941.436 I slot print_timing: id 0 | task 0 | n_decoded = 202, tg = 11.18 t/s, tg_3s = 11.13 t/s
0.45.000.485 I slot print_timing: id 0 | task 0 | n_decoded = 236, tg = 11.17 t/s, tg_3s = 11.11 t/s
0.48.073.648 I slot print_timing: id 0 | task 0 | n_decoded = 270, tg = 11.16 t/s, tg_3s = 11.06 t/s
0.51.146.793 I slot print_timing: id 0 | task 0 | n_decoded = 304, tg = 11.15 t/s, tg_3s = 11.06 t/s
0.54.219.707 I slot print_timing: id 0 | task 0 | n_decoded = 338, tg = 11.14 t/s, tg_3s = 11.06 t/s
0.57.293.346 I slot print_timing: id 0 | task 0 | n_decoded = 372, tg = 11.13 t/s, tg_3s = 11.06 t/s
1.00.365.786 I slot print_timing: id 0 | task 0 | n_decoded = 406, tg = 11.13 t/s, tg_3s = 11.07 t/s
1.03.438.043 I slot print_timing: id 0 | task 0 | n_decoded = 440, tg = 11.12 t/s, tg_3s = 11.07 t/s
1.06.511.692 I slot print_timing: id 0 | task 0 | n_decoded = 474, tg = 11.12 t/s, tg_3s = 11.06 t/s
1.06.963.836 I srv operator(): operator(): cleaning up before exit...

[Basten7]

Performance Results for Llama-3.3-70B-Instruct-Q4_K_M.gguf

Test 1: Without TOSH_MGPU_PEER

unset TOSH_MGPU_PEER
 "/Applications/ToshLLM.app/Contents/Resources/bin/llama-server" -c 65568 -t 12 -fa on --host 127.0.0.1 --port 8082 --no-mmap --split-mode layer --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.00 -m /Downloads/Llama-3.3-70B-Instruct-Q4_K_M.gguf   -lv 4

2.20.907.754 I slot print_timing: id  3 | task 0 | prompt eval time =    2397.51 ms /    61 tokens (   39.30 ms per token,    25.44 tokens per second)
2.20.907.758 I slot print_timing: id  3 | task 0 |        eval time =   48180.58 ms /   482 tokens (   99.96 ms per token,    10.00 tokens per second)
2.20.907.759 I slot print_timing: id  3 | task 0 |       total time =   50578.09 ms /   543 tokens
2.20.907.759 I slot print_timing: id  3 | task 0 |    graphs reused =        479
2.20.907.783 I slot      release: id  3 | task 0 | stop processing: n_tokens = 542, truncated = 0
2.20.907.796 I srv  update_slots: all slots are idle

Test 2: With export TOSH_MGPU_PEER=1

export TOSH_MGPU_PEER=1
"/Applications/ToshLLM.app/Contents/Resources/bin/llama-server" -c 65568 -t 12 -fa on --host 127.0.0.1 --port 8082 --no-mmap --split-mode layer --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.00 -m /Downloads/Llama-3.3-70B-Instruct-Q4_K_M.gguf   -lv 4

1.14.704.323 I slot print_timing: id  3 | task 0 | prompt eval time =     957.80 ms /    61 tokens (   15.70 ms per token,    63.69 tokens per second)
1.14.704.328 I slot print_timing: id  3 | task 0 |        eval time =   49127.88 ms /   492 tokens (   99.85 ms per token,    10.01 tokens per second)
1.14.704.329 I slot print_timing: id  3 | task 0 |       total time =   50085.68 ms /   553 tokens
1.14.704.330 I slot print_timing: id  3 | task 0 |    graphs reused =        489
1.14.704.354 I slot      release: id  3 | task 0 | stop processing: n_tokens = 552, truncated = 0
1.14.704.366 I srv  update_slots: all slots are idle

Summary of Results

Metric	Without TOSH_MGPU_PEER	With TOSH_MGPU_PEER=1
Prompt Eval Time	2397.51 ms	957.80 ms
Prompt Tokens/sec	25.44 t/s	63.69 t/s
Eval Time	48180.58 ms	49127.88 ms
Eval Tokens/sec	10.00 t/s	10.01 t/s
Total Time	50578.09 ms	50085.68 ms
Total Tokens	543	553
Graphs Reused	479	489

Observations:

Enabling TOSH_MGPU_PEER=1 significantly improved prompt evaluation time, reducing it from 2397.51 ms to 957.80 ms, and increased the token generation rate during prompt processing from 25.44 t/s to 63.69 t/s.

The eval time and tokens per second during generation remained nearly identical (~10 t/s).

Total processing time saw a slight improvement (from 50578.09 ms to 50085.68 ms).

More graphs were reused in the second test (489 vs 479), suggesting better resource utilization or caching when TOSH_MGPU_PEER=1 is enabled.

[engeldlgado]

@Basten7 that's a decisive result, and a clean one, thank you. On Llama-3.3-70B Q4_K_M, peer off vs on, same prompt:

• Prompt processing: 25.4 -> 63.7 t/s (prompt eval 2398 ms -> 958 ms)
• Generation (decode): 10.0 -> 10.0 t/s, unchanged

The win is in prefill / time-to-first-token, and that part I can tie to the code: prompt processing sends large activations between the dies, and the default path drains both GPUs and bounces them through system RAM at every layer boundary. The peer copy removes both, so prompt eval more than halves. Generation speed didn't move here (10.0 both ways), so on this model the peer path speeds up prompt processing and leaves tokens/sec untouched. I'll leave the "why" of the flat decode at "the numbers show no change" rather than guess at it.

I don't want to oversell it either way: the size of the prefill gain depends on the split (more dies, longer prompts) and it did nothing for generation here, so I'll describe it as faster prompt processing on bridged multi-GPU, not a blanket "faster."

On next steps: this is enough to make the peer path a first-class option instead of a hidden flag, so I'm adding a Settings toggle for it (off by default) in the next build, so anyone with a fabric link can turn it on and back off if it doesn't suit their rig. Before I'd make it the default, I want to harden something I found reviewing the code: today if a peer copy's command buffer fails (like the status-5 timeout you hit before the reboot), the backend drops into an unrecoverable state with no fallback for that copy. The fix is a one-time synchronous probe on the first peer copy of a session: if it fails, peer quietly falls back to the safe host-staging path instead of ending the session; if it passes, the fast async path runs the rest. That's going in alongside the toggle.

Thanks again, the off-vs-on numbers on the same model are what settle it.

[engeldlgado]

@Basten7 quick heads-up: 0.83.3 is out with both pieces. There is now a Settings toggle (Multi-GPU section) to turn the peer path on from the GUI, no env var needed, and the first peer copy of each session is probed: if it cannot complete, it falls back to host staging instead of ending the session. Whenever you have a moment, a run from the toggle would be great. Thanks again for all the testing.

[Basten7]

Test on larger Prompt: 7911 tokens

Summary of Results

Metric	With TOSH_MGPU_PEER=1	With unset TOSH_MGPU_PEER
Prompt Eval Time	52304.22 ms	51875.16 ms
Prompt Tokens/sec	139.40 t/s	140.55 t/s
Eval Time	83013.78 ms	88552.51 ms
Eval Tokens/sec	8.64 t/s	7.00 t/s
Total Time	135318.00 ms	140427.67 ms
Total Tokens	8008	7911
Graphs Reused	1372	614

Observations:

• Prompt evaluation time and speed were very similar between the two tests, with a slight edge to unset TOSH_MGPU_PEER (51875.16 ms / 140.55 t/s) compared to TOSH_MGPU_PEER=1 (52304.22 ms / 139.40 t/s).

• Eval time was significantly faster with TOSH_MGPU_PEER=1 (83013.78 ms) compared to unset TOSH_MGPU_PEER (88552.51 ms).

• Eval tokens per second was notably higher with TOSH_MGPU_PEER=1 (8.64 t/s) versus unset TOSH_MGPU_PEER (7.00 t/s), indicating better generation performance when the environment variable is set.

• Total processing time was shorter with TOSH_MGPU_PEER=1 (135318.00 ms) compared to unset TOSH_MGPU_PEER (140427.67 ms), primarily due to the faster evaluation phase.

• The number of graphs reused was substantially higher with TOSH_MGPU_PEER=1 (1372) compared to unset TOSH_MGPU_PEER (614).

[Basten7]

Performance Results: Llama-3.1-70B-Q4_0.gguf with and without TOSH_MGPU_PEER=1

Test 1: Without TOSH_MGPU_PEER (server-2026-07-23_22-29-33.log)

 ToshLLM 0.83.3 — server start (x86_64)
 engine : bundled (official)
 model  : llama3.1-70b-q4_0.gguf
 GPUs detected:
    [0] AMD Radeon Pro 580 · 8 GB · EXTERNAL/eGPU
    [1] AMD Radeon Pro Vega II Duo · 31 GB
    [2] AMD Radeon Pro Vega II Duo · 31 GB
    [3] AMD Radeon PRO W6800 · 29 GB
    [4] AMD Radeon PRO W6800X · 31 GB
    [5] AMD Radeon PRO W6800X Duo · 31 GB
    [6] AMD Radeon PRO W6800X Duo · 31 GB
    [7] AMD Radeon PRO W6800X Duo · 31 GB
    [8] AMD Radeon PRO W6800X Duo · 31 GB
 GPU select: split-all | force-VRAM-buffers: no
 settings: ngl=99 ncmoe=0 ctx=16384 fa=auto ctk=f16 ctv=f16 cacheRAM=2048
 dflash : not installed
 env: GGML_METAL_VRAM_RESERVE_MB=1024 GGML_METAL_DEVICE_LIST=7,8 TOSH_FA_AMD=1
 args: -m /Models/llama3.1-70b-q4_0.gguf -ngl 99 -c 16384 -t 6 -fa auto --host 127.0.0.1 --port 8080 --no-mmap --jinja --cache-ram 2048 --cache-reuse 256 --parallel 1 --split-mode layer --path /Applications/ToshLLM.app/Contents/Resources/test-ui


4.57.701.783 I slot print_timing: id  0 | task 0 | prompt eval time =   48839.27 ms /  7292 tokens (    6.70 ms per token,   149.31 tokens per second)
4.57.701.788 I slot print_timing: id  0 | task 0 |        eval time =  220607.96 ms /  2048 tokens (  107.72 ms per token,     9.28 tokens per second)
4.57.701.788 I slot print_timing: id  0 | task 0 |       total time =  269447.23 ms /  9340 tokens
4.57.701.789 I slot print_timing: id  0 | task 0 |    graphs reused =       2039
4.57.702.038 I slot      release: id  0 | task 0 | stop processing: n_tokens = 9339, truncated = 0

Test 2: With export TOSH_MGPU_PEER=1 (server-2026-07-23_22-26-11.log)

 ToshLLM 0.83.3 — server start (x86_64)
 engine : bundled (official)
 model  : llama3.1-70b-q4_0.gguf
 GPUs detected:
    [0] AMD Radeon Pro 580 · 8 GB · EXTERNAL/eGPU
    [1] AMD Radeon Pro Vega II Duo · 31 GB
    [2] AMD Radeon Pro Vega II Duo · 31 GB
    [3] AMD Radeon PRO W6800 · 29 GB
    [4] AMD Radeon PRO W6800X · 31 GB
    [5] AMD Radeon PRO W6800X Duo · 31 GB
    [6] AMD Radeon PRO W6800X Duo · 31 GB
    [7] AMD Radeon PRO W6800X Duo · 31 GB
    [8] AMD Radeon PRO W6800X Duo · 31 GB
 GPU select: split-all | force-VRAM-buffers: no
 settings: ngl=99 ncmoe=0 ctx=16384 fa=auto ctk=f16 ctv=f16 cacheRAM=2048
 dflash : not installed
 env: GGML_METAL_VRAM_RESERVE_MB=1024 GGML_METAL_DEVICE_LIST=7,8 TOSH_FA_AMD=1 TOSH_MGPU_PEER=1
 args: -m /Models/llama3.1-70b-q4_0.gguf -ngl 99 -c 16384 -t 6 -fa auto --host 127.0.0.1 --port 8080 --no-mmap --jinja --cache-ram 2048 --cache-reuse 256 --parallel 1 --split-mode layer --path /Applications/ToshLLM.app/Contents/Resources/test-ui

2.48.753.282 I slot print_timing: id  0 | task 0 | prompt eval time =   48352.27 ms /  7292 tokens (    6.63 ms per token,   150.81 tokens per second)
2.48.753.289 I slot print_timing: id  0 | task 0 |        eval time =   75099.34 ms /   708 tokens (  106.07 ms per token,     9.43 tokens per second)
2.48.753.290 I slot print_timing: id  0 | task 0 |       total time =  123451.61 ms /  8000 tokens
2.48.753.290 I slot print_timing: id  0 | task 0 |    graphs reused =        704
2.48.753.524 I slot      release: id  0 | task 0 | stop processing: n_tokens = 7999, truncated = 0

Summary of Results

Metric	Without TOSH_MGPU_PEER	With TOSH_MGPU_PEER=1
Prompt Eval Time	48839.27 ms	48352.27 ms
Prompt Tokens/sec	149.31 t/s	150.81 t/s
Eval Time	220607.96 ms	75099.34 ms
Eval Tokens/sec	9.28 t/s	9.43 t/s
Total Time	269447.23 ms	123451.61 ms
Total Tokens	9340	8000
Graphs Reused	2039	704

[engeldlgado]

@Basten7 these are useful, but before I read anything into them I want the comparison to be clean, because the two runs in the Q4_0 test aren't equivalent yet. Two things:

1. Did the peer path actually engage? The Infinity Fabric peer transfer enabled ... line (needs -lv 4) in the TOSH_MGPU_PEER=1 run confirms it. If it isn't in the log, the copy fell back to host staging and both runs used the same path, which alone would explain "no difference." I couldn't tell from these logs whether peer was live or fell back.

2. The two runs generated a different number of tokens, so the totals aren't comparable. Reading straight from your logs: the eval time line is the generation (decode) phase, and its token count is how many tokens the model generated:

• without peer: eval time = 220607 ms / 2048 tokens -> generated 2048 tokens
• with peer: eval time = 75099 ms / 708 tokens -> generated 708 tokens

So the total time gap (269 s vs 123 s) and the total tokens (9340 vs 8000, which is prompt 7292 + generated) come from that length difference, not from the peer path. The numbers that are comparable are the per-token rates, and there the two runs are basically identical:

• prefill: 149.31 vs 150.81 t/s
• decode: 107.72 vs 106.07 ms/token (9.28 vs 9.43 t/s)

If you fix the output to the same number of tokens both ways (e.g. --n-predict 512), or just compare the per-token t/s, that isolates whether the peer copy is doing anything.

With the peer log line plus a same-length run, I can actually tell if the peer path helps on a long-prompt 2-die split. Right now the totals differ only because the runs generated different amounts, so I honestly can't conclude either way. Thanks, really.

[engeldlgado]

@Basten7 heads-up before more tests: 0.83.4 changes two things relevant to what we have been chasing.

1. I removed the first-copy probe I added in 0.83.3. I could not test it on peer-group hardware (I do not have any), so it went into the critical path unvalidated, and it was a real suspect for the "no difference" on your 2-die long-prompt run: if the very first copy hit a transient fault (like the pre-reboot status-5), the probe would quietly fall back to host staging for the rest of the session. The peer copy is now back to the exact direct/async version that gave the fast prefill on the short-prompt run.

2. The confirmation line now prints at the normal log level, so you no longer need -lv 4. You will see ggml_metal: Infinity Fabric peer transfer enabled: ... (peer group N) in the normal log whenever the path actually engages.

So on 0.83.4 the two things I need are both easy now: (a) does that line appear (peer engaged), and (b) a same-generation-length off-vs-on comparison (fix --n-predict so both runs generate the same number of tokens).

One thing I want to be straight about: I stated a couple of things earlier more confidently than I should have. The "2.5x" I treated as a headline was really one specific configuration (a short prompt split across all your GPUs, including the eGPU over Thunderbolt), not a general number, and I do not actually know it holds elsewhere. And when I read your long-prompt run as "no difference from the peer," I could not confirm the peer even engaged in that run, so that conclusion was not solid either. I would rather flag that than have you chase a number I oversold. The clean 0.83.4 runs (with the log line visible and a fixed generation length) are what will actually settle it, in whichever direction. Thanks for the patience on this.