nvme pcie4 x4 to pcie4 x8 adapter?

[magikRUKKOLA]

It seems clear nowadays that one have to pursue the fastest RAM/VRAM to get the better inference etc. So it somewhat looks like the best price/performance setup is a AMD Threadripper PRO 3xxx/5xxx series and a bunch of RTX 3090.

So ...

So the seven GPUs would comfortably use pcie4 x16. Suppose we could connect another GPU via pcie4 x8 and compensate for it via NvLink bridge.

Now we have (*potentially) 192 GB VRAM GDDR6X. That is cool, but clearly not enough for Kimi-K2-Thinking. Hence the question: what to do? I was thinking about some kind of adapter to install into the m.2 nvme slots to get pcie4 x8 for the RDMA communication to the [similar] machine with 8 GPUs. Does anyone have any ideas where to get such an adapter which would unite two x4 lanes into x8?

[ikawrakow]

@magikRUKKOLA With current RAM prices and many GPUs, the system price increase of using a more recent CPU wouldn't be very noticeable. You may want to think about an EPYC of the 9005 series instead of a 3xxx or 5xxx Threadripper. For instance, the EPYC 9355 was just $2k last I checked, and will wipe the floor with the 3xxx or 5xxx in terms of both, number crunching performance and memory bandwidth you get.

[ikawrakow]

Soon people will realize that it is cheaper to buy a GPU to act as RAM module, and then used GPU prices will go crazy too...

[magikRUKKOLA]

@ikawrakow

As related to the Zen5 box. Given the prices of DDR5 RAM what if we will get 16GB sticks. So the total of 384GB DDR5-4800 MT/s.
Then if we will use say Kimi-K2-Thinking, we would be forced to offload to the GPU(s) because we have no RAM lol. (!) So that would require about 8 additional GPU(s) of 24GB VRAM each. :)

384GB DDR5 (2 CPU, 12 channels) + 192GB VRAM GDDR6X (8x3090 RTX)

2 sticks of DDR5 (16GB each) (non-ECC lol) cost around 320 EUR right now. So for the whole machine the RAM price is under 4k EUR.

Will that work?

[magikRUKKOLA]

@ikawrakow

I found some additional deals for the RAM so the DDR5 4800 M/T used goes for 430 EUR per 64GB ECC. Realistically, one would have to get about [30] sticks of those (some for the backups). That alone is 13k EUR. Yeah, the EPYC 9575F QS is 1.8k EUR each. So the whole box would be about 20k EUR for sure. And its 24 channels of DDR5 4800 M/T ECC.

Alternatively, 1.2k EUR for the 16 channels of DDR4 3200 M/T (non-ECC).

echo "scale=2;(4800*24)/(3200*16)"|bc
2.25

The Zen2 DDR4 system will be 2.25 times slower that Zen5 DDR5 memory-wise.

Is it worth paying ten times the price?

[ikawrakow]

Is it worth paying ten times the price?

Me for sure wouldn't spend 20k. But your comparison is a bit of apples to oranges.

But at the end, as far as I can tell, you like building rigs, so it is up to you what you would like to build.

[magikRUKKOLA]

@ikawrakow

Okay cool. I found some more deals with 32GB DDR5 4800 MT/s ECC for 200 EUR each. So 6k EUR and 768GB total.
That seems much better. We don't need an additional capacity (1.5TB) and it two times cheaper.

I will likely get Zen2 box as well since its still 16 channels and 512GB RAM.

I will research the CPUs and motherboards for Zen5 later on. Does Zen5 offer any cool instructions for us to use?

I also found out that its better to use the server PSUs for the GPUs (to kinda solve the ground loops problem). Example: https://www.newegg.com/hp-578322-b21-1200w/p/N82E16817292058
The only problem that they are noisy so I have to replace the fans as well.

[Panchovix]

A NVMe to PCIe adapter will do at most X4 lanes.

So a M2 X4 to PCIe X8/X16 adapter will be electrical X4 but mechanical X8/X16.

At PCIe 4.0 that is about 6 to 7 gigabytes/s.

[magikRUKKOLA]

A NVMe to PCIe adapter will do at most X4 lanes.

So a M2 X4 to PCIe X8/X16 adapter will be electrical X4 but mechanical X8/X16.

At PCIe 4.0 that is about 6 to 7 gigabytes/s.

So no adapters exists to unite two nvme (pcie4 x4) to one pcie4 x8?

[Panchovix]

@magikRUKKOLA not that I know sadly, except if you use a "reverse" switch? Not sure if they exist as well.

I searched a bit to do that on gen 5 (so 2X4 to 1X8) but no luck sadly.