Tensor parallelism fails on 2x Tesla V100: AllReduce kernel not compiled for compute_70 (works in runtime 2.25.2)

[crawforddmorrison-crypto]

Which version of LM Studio?
LM Studio 0.4.23

Which operating system?
Windows11 pro

What is the bug?

Summary

After the CUDA runtime auto-updated from 2.25.2 to 2.31.2, tensor parallelism no longer works on 2× Tesla V100. The AllReduce kernel is not compiled for compute capability 7.0 (Volta).

The error reports the compiled architecture list as 500, 610, 750, 800 — Maxwell and Pascal are present, but Volta (700) is missing. This does not look like an intentional drop of pre-Turing support, since 500 and 610 would have to go with it.

Environment

• Runtime: llama.cpp-win-x86_64-nvidia-cuda-avx2
  • 2.25.2 — works
  • 2.26.0 — fails
  • 2.27.1 — fails
  • 2.28.2 — fails
  • 2.29.1 — fails
  • 2.31.2 — fails
• GPUs: 2× Tesla V100-PCIE-32GB (compute capability 7.0), PCIe 3.0 x8/x8
• Model: gemma-4-31B-it-uncensored-heretic Q8_0 (GGUF)

lms runtime survey output:

GPU/ACCELERATORS                        VRAM
Tesla V100-PCIE-32GB (CUDA, Discrete)   31.86 GiB
Tesla V100-PCIE-32GB (CUDA, Discrete)   31.86 GiB
CPU: x86_64 (AVX, AVX2)
RAM: 63.91 GiB

Scope

This is specific to the tensor parallelism path, not to Volta support in general:

• Same runtime 2.31.2 with "Split evenly": works, both GPUs load and generate normally
• Same runtime 2.31.2 with tensor parallelism: fails as above

So only the AllReduce kernel appears to lack device code for sm_70.

Note on upstream

Upstream llama.cpp does build this path for Volta — see ggml-org/llama.cpp#27366, where the reporter builds with -DCMAKE_CUDA_ARCHITECTURES=70 on V100s and reaches runtime (they hit a different bug). That suggests the missing 700 comes from the runtime build configuration rather than from upstream source.

Request

Please add compute_70 to the architecture list for the CUDA runtime build, or document that Volta is no longer supported for tensor parallelism.

Logs

llm-engine\llama.cpp\ggml\src\ggml-cuda\allreduce.cu:167: ERROR: CUDA kernel ggml_cuda_ar_kernel has no device code compatible with CUDA arch 610. ggml-cuda.cu was compiled for: 500,610,750,800

llm-engine\llama.cpp\ggml\src\ggml-cuda\ggml-cuda.cu:106: CUDA error
E CUDA error: unspecified launch failure
E   current device: 0, in function ggml_cuda_ar_acquire_slot at llm-engine\llama.cpp\ggml\src\ggml-cuda\allreduce.cu:368
E   cudaEventSynchronize(p->ev_pool[i][slot].ker)

The process then exits with code 3221226505 (0xC0000409), and LM Studio reports:

Engine protocol runtime llama-server exited before becoming healthy. exitCode=3221226505

Note: the message says "arch 610", but there is no Pascal GPU in this machine — only the two V100s shown above. The reported architecture number appears to be wrong; the missing entry is 700.

To Reproduce
Steps to reproduce the behavior:

1. Install runtime llama.cpp-win-x86_64-nvidia-cuda-avx2 2.31.2 (or 2.29.1)
2. Open the GPU controls and make sure both V100s are enabled
3. Enable tensor parallelism
4. Load any GGUF model
5. Loading fails and the engine process exits; see the log above

[MilkyWay008]

The missing compute_70 in the AllReduce kernel is a build-config thing, not your setup, the CUDA runtime bundles dropped Volta from the arch list somewhere between 2.25.2 and 2.26.0. Since "Split evenly" works on the same runtime, pinning back to 2.25.2 is the reliable fix for now, or keep split-evenly until they re-add sm_70 to the build. There's a llama.cpp issue on this, ggml-org/llama.cpp#27366, that builds with -DCMAKE_CUDA_ARCHITECTURES=70 if you want to build your own runtime. Oh also btw, I keep a portable Hermes OTG on USB as my rescue agent, super handy for diagnosing and fixing broken installs like this, https://github.com/MilkyWay008/Hermes-OTG, imo it'll help.