NvLink Bridge across GPUs from two physical workstations? Possible?

[magikRUKKOLA]

Have anyone tried that? I was thinking to connect two-slot NvLink to the RTX 3090 from two different machines to eliminate any need of using infiniband and RDMA.
Not sure if that will work, but if so that would allow to easily connect two workstations with 8 GPUs each. So there will be total of 16 GPUs and 384 GB VRAM total.
If the p2p driver from @aikitoria would work as well, that would be a perfect solution for the local inference.
16 GPUs and 8 NvLink bridges spaced across two workstations.

[aikitoria]

These are only for use within the same device

[magikRUKKOLA]

Uh oh. So the only option is to use PCIe bifurcation then?

[EDIT]: edit, that is, to NvLink the full-speed (x16) GPU with the bifurcated (x4 ?) via the NvLink.

[ikawrakow]

@magikRUKKOLA

In the early days of using ik_llama.cpp for giant MoE models, there was @davidsyoung who was trying to use ik_llama.cpp on a 16x3090 box they had built. Perhaps you try talking to @davidsyoung?

[magikRUKKOLA]

@davidsyoung

Hi there! Got 16 x 3090 RTX and two kinds of CPUs -- the AMD Threadripper PRO 3995wx and Xeon QYFS.
How to connect them together properly?

[magikRUKKOLA]

I think the easiest solution is to use bifurcation for the rest of the PCIe slots (three in my case). So I can add another four GPUs with PCIe4 x8 speed. The question is where to put the NvLinks? With four NvLinks (2-slot bridges) there are three possibilities.

x16_gpu <-> x16_gpu
x16_gpu <-> x16_gpu
x8_gpu <-> x8_gpu
x8_gpu <-> x8_gpu

---

x16_gpu <-> x16_gpu
x16_gpu <-> x8_gpu
x16_gpu <-> x8_gpu
x8_gpu <-> x8_gpu

---

x16_gpu <-> x8_gpu
x16_gpu <-> x8_gpu
x16_gpu <-> x8_gpu
x16_gpu <-> x8_gpu

---

I was obviosly thinking of the latter one.

[aikitoria]

It is worth noting NVLINK only has meaningful bandwidth advantage between the paired GPUs, not really meaningfully lower latency, and the P2P driver cannot be used at the same time as NVLINK. With a properly designed inference engine and single user use case, NVLINK is ~useless

[magikRUKKOLA]

@aikitoria

and the P2P driver cannot be used at the same time as NVLINK.

Yeah, that have to be fixed somehow. :)