[Linux] CUDA12 backend silently falls back to CPU due to hardcoded RUNPATH in shipped .so files

[Cognameton]

Summary

On Linux, LLamaSharp.Backend.Cuda12 0.25.0 silently falls back to CPU
regardless of GpuLayerCount. No error is thrown. The model loads and
runs normally, just entirely on CPU.

Root Cause

The shipped Linux .so files (libggml-cuda.so, libllama.so, etc.)
have a hardcoded absolute RUNPATH:

RUNPATH: /home/runner/work/LLamaSharp/LLamaSharp/build/bin

This is the CI build machine's path. On any end-user system that path
does not exist. When libggml-cuda.so loads and looks for its dependency
libggml-base.so, the OS dynamic linker cannot find it, the CUDA backend
fails to initialise, and LLamaSharp silently falls back to the CPU backend.

Impact

Every Linux user installing LLamaSharp.Backend.Cuda12 via NuGet is
running on CPU without knowing it. The only clue is dev = CPU in the
llama.cpp layer log, which most users won't see or know to look for.

Workaround

private static void PreloadCudaBackend()
{
if (!OperatingSystem.IsLinux()) return;

  var nativeDir = Path.Combine(AppContext.BaseDirectory, "runtimes", "linux-x64", "native");
  var cudaDir   = Path.Combine(nativeDir, "cuda12");
  if (!Directory.Exists(cudaDir)) return;                                                                                                                                                 

  // Detect best CPU AVX level — libggml.so (cuda12) depends on libggml-cpu.so                                                                                                            
  var cpuDir = BestCpuDir(nativeDir);
                                                                                                                                                                                          
  LoadLib(Path.Combine(cudaDir, "libggml-base.so"));                                                                                                                                      
  if (cpuDir != null) LoadLib(Path.Combine(cpuDir, "libggml-cpu.so"));
  LoadLib(Path.Combine(cudaDir, "libggml-cuda.so"));                                                                                                                                      
  LoadLib(Path.Combine(cudaDir, "libggml.so"));                                                                                                                                           
  LoadLib(Path.Combine(cudaDir, "libllama.so"));
  LoadLib(Path.Combine(cudaDir, "libmtmd.so"));                                                                                                                                           

}

private static string? BestCpuDir(string nativeDir)
{
try
{
var flags = File.ReadAllText("/proc/cpuinfo");
string[] candidates = flags.Contains("avx512") ? new[] { "avx512", "avx2", "avx", "noavx" }
: flags.Contains("avx2") ? new[] { "avx2", "avx", "noavx" }
: flags.Contains(" avx ") ? new[] { "avx", "noavx" }
: new[] { "noavx" };
foreach (var c in candidates)
{
var dir = Path.Combine(nativeDir, c);
if (Directory.Exists(dir)) return dir;
}
}
catch { }
return null;
}

private static void LoadLib(string path)
{
if (!File.Exists(path)) return;
try { NativeLibrary.Load(path); }
catch (Exception ex)
{
Console.WriteLine($"[WARN] Pre-load failed: {Path.GetFileName(path)} — {ex.Message}");
}
}

Worth noting in the update that libggml.so in the cuda12 folder is a combined backend dispatcher that depends on both libggml-cuda.so and libggml-cpu.so, so both need to be pre-loaded.

Fix

Build the Linux native libraries with -Wl,-rpath,$ORIGIN so the RUNPATH
resolves relative to the library's own directory rather than the CI
machine's absolute path.

Environment

• LLamaSharp: 0.25.0
• OS: Linux (Ubuntu 25.04)
• GPU: NVIDIA RTX 3060, Driver 580.126.09, CUDA 13.0
• dotnet: 8.0