Ship 1.4.0-beta: real Multi-GPU pipeline/tensor and API prompt fail-fast.

Enable Settings-driven GPU split, combined VRAM fit in Admin, CUDA remap, and surface prompt_too_long instead of client 408 hangs.

Co-authored-by: Cursor <cursoragent@cursor.com>

Author: VitorCastroSF

README.md

@@ -1,4 +1,4 @@
-# ExLlamaSharp
+# ExLlamaSharp
 
 **Local LLM server for Windows with NVIDIA GPUs.**
 
@@ -11,7 +11,7 @@ Inspired by:
 - **ExLlamaV3** — fast EXL3 inference on NVIDIA
 - **Open WebUI** — browser-based administration
 
-**Current release: 1.2.1** — product-final (4 bars at 100%): Core EXL3 OpenAI chat, Models/Jobs/Keys, Admin avançado (LoRA, speculative, multi-GPU honesty, webhooks, tenants), Agentic tools + vision multimodal chat. OpenAI **images/audio generation** remain **501** (separate media track). Setup.exe bundles the ExLlamaV3 CUDA `.pyd`, worker deps, Python installer and VC++. PyTorch CUDA is downloaded during install. Admin → Models shows a VRAM fit badge (Fits / Tight / Too large; estimate only).
+**Current release: 1.4.0-beta** — real Multi-GPU (pipeline / tensor), combined VRAM fit in Admin, CUDA device remap (strongest GPU first), and faster fail for oversized prompts (`prompt_too_long` instead of hanging to 408). Core EXL3 OpenAI chat, Models/Jobs/Keys, LoRA, speculative, webhooks, tenants, agentic tools. Vision models skip the vision tower under multi-GPU (text-only). OpenAI **images/audio generation** remain **501**. Setup.exe bundles the ExLlamaV3 CUDA `.pyd`, worker deps, Python installer and VC++.
 
 Default after install: **http://127.0.0.1:14563**
 
@@ -24,12 +24,12 @@ Default after install: **http://127.0.0.1:14563**
 
 ## Download (Windows x64)
 
-[**ExLlamaSharp-Setup-win-x64.exe**](https://github.com/vitorcastro78/ExLlamaSharp/releases/latest/download/ExLlamaSharp-Setup-win-x64.exe) — latest GitHub Release.
+[**ExLlamaSharp-Setup-win-x64.exe**](https://github.com/Kortexio/ExLlamaSharp/releases/latest/download/ExLlamaSharp-Setup-win-x64.exe) — latest GitHub Release.
 
 One-liner (downloads Setup and launches UAC):
 
 ```powershell
-irm https://raw.githubusercontent.com/vitorcastro78/ExLlamaSharp/main/packaging/install-web.ps1 | iex
+irm https://raw.githubusercontent.com/Kortexio/ExLlamaSharp/main/packaging/install-web.ps1 | iex
 ```
 
 Double-click the Setup.exe → allow UAC → PyTorch CUDA downloads into the venv, ExLlamaV3 extension installs from the package → open **http://127.0.0.1:14563**.
@@ -220,7 +220,7 @@ Persisted server settings:
 |-----|----------|
 | Network | Bind address, port, CORS, TLS cert path |
 | Performance | Max sequences, chunk size, batched tokens, GPU memory util, request timeout |
-| Multi-GPU | `CUDA_VISIBLE_DEVICES`, parallelism mode (validated; worker gets device list) |
+| Multi-GPU | PCI devices, `none` / `tensor` / `pipeline`, GPU memory util, optional `GpuSplitGb`. Save recycles the worker. |
 | Speculative | Enable + draft model + draft K (forwarded to worker) |
 | Startup | Load last model on startup, models path |
 | Hugging Face | Optional `hf_…` token (also reads `HF_TOKEN`) |
@@ -261,7 +261,7 @@ By design (not product gaps for the EXL3 text + vision chat product):
 - **OpenAI images / audio generation** — **501**; separate Media track
 - **Native DLL generate** — worker-only for production text; DLL remains CI / scheduler ABI
 - **MCP / hosted ReAct agent** — not embedded; use OpenAI `tools` / `tool_calls` with your own agent loop
-- **Multi-GPU TP/PP/MP** — not supported by the EXL3 worker; Settings reject those modes. Multi-GPU visibility via `CUDA_VISIBLE_DEVICES` only
+- **Multi-GPU model-parallel (MP)** — not supported. **Tensor** and **layer autosplit** (`pipeline`) are supported via the EXL3 worker on N NVIDIA GPUs (VRAM-proportional split, optional `GpuSplitGb`).
 
 A/B tests: create via `/api/v1/ab`, then send `X-Ab-Test-Id` (or `model: "ab:<guid>"`) on chat/completions. The server assigns A/B via consistent hash, may load the selected model when it differs from the one currently on the GPU, tags audit, and returns `X-Ab-Variant`.
 

docs/admin-guide.md

@@ -1,4 +1,4 @@
-# ExLlamaSharp Admin Guide
+# ExLlamaSharp Admin Guide
 
 Operational guide for administrators of the Windows service and Blazor UI.
 
@@ -44,11 +44,11 @@ Important fields:
 | Backup | `AutoBackupSchedule` (`disabled` / `daily` / `weekly`) |
 | Webhooks | `WebhookUrl`, `WebhookSecret` |
 | Features | content moderation, multi-tenancy, advanced metrics |
-| GPU | `CudaVisibleDevices` (e.g. `0` or `0,1`), `ParallelismMode` (`none` / `tensor` / `pipeline` / `model`) |
+| GPU | `CudaVisibleDevices` (PCI indices, e.g. `0,1`), `ParallelismMode` (`none` / `tensor` / `pipeline`), `GpuMemoryUtilization`, optional `GpuSplitGb` |
 | Speculative | `SpeculativeEnabled`, `DraftModelId`, `DraftK` |
 | Paths | `ModelsPath` |
 
-After changing GPU / parallelism / bind address, restart the Windows service so the process picks up the new host environment.
+Saving GPU / parallelism settings recycles the Python worker and reloads the model. Bind address / port still need a Server restart.
 
 ## Backup & restore
 
@@ -60,16 +60,26 @@ Backups do **not** include multi‑GB weight files — back up the `models\` fol
 
 ## Multi-GPU
 
-1. Confirm GPUs with `nvidia-smi` and **About** / Diagnostics.
-2. Set `CudaVisibleDevices` to the device indices to expose (e.g. `0,1`).
-3. Set `ParallelismMode`:
-   - `none` — single GPU
-   - `tensor` (TP) — split layers across GPUs (latency / large models)
-   - `pipeline` (PP) — pipeline stages across GPUs
-   - `model` (MP) — whole models on different devices (routing / multi-model)
-4. Restart service and load the model again.
+Works with any mix of NVIDIA GPUs (equal or different VRAM). No SKU is hardcoded.
 
-Server helpers: `MultiGpuPlanner` validates device lists and maps modes for the native engine config. NCCL / full TP-PP production paths depend on the native CUDA build (not stub).
+1. Confirm cards on **About** (PCI index, name, VRAM, UUID) or `nvidia-smi`.
+2. Set `CudaVisibleDevices` to the **PCI / nvidia-smi** indices to use (`0,1` or `0,2`, …). Empty = all.
+3. The worker process is started with `CUDA_DEVICE_ORDER=PCI_BUS_ID` and `CUDA_VISIBLE_DEVICES` **reordered** so `cuda:0` is the highest-VRAM GPU in that set.
+4. `ParallelismMode`:
+   - `none` — load on `cuda:0` only
+   - `tensor` — ExLlamaV3 tensor parallelism (`tensor_p=True`, backend `native`)
+   - `pipeline` — layer autosplit (`tensor_p=False` + `use_per_device`)
+   - `model` is **rejected** (not implemented)
+5. Split: auto `VRAM[i] × GpuMemoryUtilization`, or override `GpuSplitGb` in **remapped** order (e.g. `10,4.5` for a 12 GB + 6 GB pair). Count must match visible GPUs.
+6. Save Settings (or PATCH `/api/v1/settings`) — the worker is recycled and the loaded model is queued again.
+
+`tensor` / `pipeline` require at least two indices. OOM usually hits the smallest card first — lower util or set `GpuSplitGb` with more reserve on the display GPU.
+
+Examples (docs only — no SKU is hardcoded):
+
+- Two equal 12 GB cards: `CudaVisibleDevices=0,1`, `tensor`, empty `GpuSplitGb`, util `0.85` → about `10.2,10.2`.
+- 12 GB + 6 GB: same devices, or `GpuSplitGb=10,4.5` in **remapped** order (`cuda:0` is the 12 GB card even if `nvidia-smi` lists it as index 1).
+- Three cards: `0,1,2` and a 3-value split. Same code path.
 
 ## Webhooks
 

docs/architecture.md

@@ -16,18 +16,18 @@
 │  ├─ EF Core + SQLite (ProgramData)      │
 │  ├─ Auth (API keys), rate limit, audit  │
 │  └─ ExLlamaSharp C# library             │
-│       LibraryImport → exllamasharp.dll  │
+│       EXL3 Python worker (production)   │
 └─────────────────────────────────────────┘
-              │  C ABI / P/Invoke
+              │  JSONL stdin/stdout
 ┌─────────────────────────────────────────┐
-│  native/exllamasharp (C++ / optional CUDA)│
-│  ├─ Scheduler (continuous batching)     │
-│  ├─ PageTable (KV pages / prefix cache) │
-│  └─ Kernels / EXL3 path (non-stub)      │
+│  tools/exl3_worker + ExLlamaV3          │
+│  ├─ Model.load (tensor_p / autosplit)   │
+│  ├─ Cache / Generator / Tokenizer       │
+│  └─ CUDA kernels from the venv          │
 └─────────────────────────────────────────┘
 ```
 
-Optional later: `third_party/exllamav3` git submodule for upstream EXL3 kernels (see `third_party/README.md`).
+Production inference is the **EXL3 Python worker**, not `exllamasharp_native.dll` (that stub is disabled). Multi-GPU TP / layer autosplit goes through `model.load`.
 
 ## .NET 10 host
 
@@ -46,26 +46,28 @@ Performance knobs: Server GC, sustained low-latency mode at startup, in-memory k
 | Mode | CMake | Behavior |
 |------|-------|----------|
 | Stub | `-DEXL_STUB=ON` | No CUDA/LibTorch; deterministic fake generate; real scheduler ABI |
-| CUDA | `-DEXL_STUB=OFF` + LibTorch + toolkit | Full path toward EXL3 GEMM / multi-GPU |
+| CUDA | `-DEXL_STUB=OFF` + LibTorch + toolkit | Optional native experiment — **not** the production multi-GPU path |
 
 Build helper: `packaging/build-native-stub.ps1`. Details: `native/exllamasharp/README.md`.
 
-C ABI (`exllamasharp.h`) is the stability boundary — .NET uses source-generated `LibraryImport` (`NativeMethods`).
+The native DLL is **disabled** at runtime (`EngineHostService`). Multi-GPU TP / autosplit is ExLlamaV3 `model.load` in the Python worker.
+
+C ABI (`exllamasharp.h`) is the leftover native boundary — .NET still has `LibraryImport` (`NativeMethods`) but the Server does not load that DLL for inference.
 
 ## Request path (chat)
 
 1. Client → `POST /v1/chat/completions` with Bearer key.
 2. Middleware: auth + rate limit; optional moderation.
 3. Chat template formats messages → token ids.
-4. `EngineHostService` submits a job to `ExLlamaEngine` (native or mock).
-5. Native scheduler batches; tokens stream back as SSE chunks if requested.
+4. `EngineHostService` submits a job to `ExLlamaV3WorkerEngine` (or mock in Development).
+5. The Python worker iterates the ExLlamaV3 Generator; tokens stream back as SSE chunks if requested.
 6. Audit / webhooks / metrics updated asynchronously.
 
 ## Multi-GPU & advanced
 
 Server-side helpers prepare config for the engine:
 
-- `MultiGpuPlanner` — TP / PP / MP from `CudaVisibleDevices` + `ParallelismMode` (validated; worker receives `CUDA_VISIBLE_DEVICES`)
+- `MultiGpuPlanner` — validates `none` / `tensor` / `pipeline` + device list + optional `GpuSplitGb`; worker spawn remaps `CUDA_VISIBLE_DEVICES` strongest-first and `model.load` gets `tensor_p` / `use_per_device`
 - `SpeculativeDecodingOptions` — draft model + `DraftK` (forwarded to EXL3 worker)
 - `ArchitectureDetector` — llama / qwen / mixtral / llava from `config.json`
 - `QuantizationModes` — EXL3 convert via `exllamav3.conversion.convert_model`

docs/comparison.md

@@ -9,7 +9,7 @@ Positioning: **Ollama’s ease + strong local NVIDIA EXL3 serving + Windows admi
 | Web UI admin | Yes (Blazor) | No | No (API only) | Yes (desktop) |
 | OpenAI-compatible API | Yes (`/v1` + Ollama-style `options`) | Yes | Yes | Yes |
 | No Docker required | Yes | Yes* | Typically containers/Linux | Yes |
-| Multi-GPU TP / PP / MP | **No** (rejected in Settings); `CUDA_VISIBLE_DEVICES` only | Limited | Yes | Limited |
+| Multi-GPU TP / layer autosplit | **Yes** (ExLlamaV3 `tensor` / `pipeline`; N NVIDIA, split by VRAM). MP not supported | Limited | Yes | Limited |
 | Non-technical friendly | Yes (wizard + UI) | Yes | No | Yes |
 | Team / tenant management | Yes (optional MultiTenancy) | No | DIY | No |
 | API keys, quotas, audit | Yes | Basic | DIY / gateway | Basic |

docs/troubleshooting.md

@@ -1,4 +1,4 @@
-# ExLlamaSharp Troubleshooting
+# ExLlamaSharp Troubleshooting
 
 Aligns with the UI page **Diagnostics** (`/diagnostics`) and `GET /health`.
 
@@ -69,6 +69,20 @@ Components reported by `HealthService`:
 - Ensure only intended devices in `CudaVisibleDevices`.
 - Close other GPU apps (browsers with HW accel, games).
 
+### Tensor parallel: Timed out waiting for worker
+
+**Symptom:** Load with `parallelism_mode=tensor` fails with `TimeoutError: Timed out waiting for worker` (ExLlamaV3 `model_tp.py`). nvidia-smi stays almost idle; leftover `python ... spawn_main` processes sit at ~8 MB.
+
+**Cause:** Tensor parallel starts extra Python processes. On Windows those children re-enter the worker script and never become TP workers when the host is the long-lived JSONL process. A console `python -c` load can succeed while the Admin/Server load fails. Pipeline mode does not spawn those children.
+
+**Fix:** Use **pipeline** (and a KV cache that fits) to load across both GPUs. For Qwen3-32B 4.0bpw on 12 GB + 8 GB use `GpuSplitGb=10.8,6.2` and keep **Max batched tokens** around 2048–4096 — 16384 plus the 32B weights does not fit. Recycle the worker after a failed load (Save on Settings, or restart the Server) so zombie `spawn_main` processes are gone.
+
+### VLM + multi-GPU: vision skipped
+
+**Symptom:** Models such as `Qwen3.8-27B-exl3` (`Qwen3_5ForConditionalGeneration` + `vision_config`) used to fail load under tensor/pipeline with `vision models are not supported…`.
+
+**Behaviour now:** The language model still loads across GPUs; the vision tower is skipped (`vision_capable=false`). Text chat works. Image/video inputs need **ParallelismMode=none** (and enough VRAM on one card), then reload.
+
 ### Slow tokens / queue buildup
 
 - Check `GET /metrics` (`jobs_waiting`, `tokens_per_second`).
@@ -83,9 +97,16 @@ Components reported by `HealthService`:
 
 ### Multi-GPU not used
 
-- Mode still `none`, or only one device in `CudaVisibleDevices`.
-- Stub/mock builds ignore real TP/PP — need CUDA native `exllamasharp.dll`.
-- Restart after Settings changes.
+- `ParallelismMode` still `none`, or only one index in `CudaVisibleDevices`.
+- `nvidia-smi` index is **not** `cuda:N` after remap — check worker log (`cuda:0` = highest VRAM).
+- Production path is the **EXL3 Python worker**, not `exllamasharp_native.dll` / mock.
+- After Settings save the worker should recycle automatically; if VRAM is still on one UUID only, reload the model and read `use_per_device` in the worker log.
+
+### OOM on the smaller GPU
+
+- Auto-split is `VRAM[i] × GpuMemoryUtilization`. A 6 GB card with util 0.9 only has ~5.4 GB for weights.
+- Display GPU keeps 1.5 GB (others 0.5 GB) folded into `use_per_device` — ExLlamaV3 does not accept use and reserve together. Lower util or set `GpuSplitGb` (remapped order, e.g. `10,4.5`).
+- Tensor / pipeline need ≥2 devices. Speculative, vision, and LoRA are rejected under those modes.
 
 ## Quick CLI checks
 

packaging/Build-Installer.ps1

@@ -256,7 +256,7 @@ Run Uninstall.bat as Administrator.
 Setup-Exl3Python.bat - reinstall PyTorch into %ProgramData%\ExLlamaSharp\venv
 "@ | Set-Content -Path (Join-Path $Stage "README.txt") -Encoding UTF8
 
-$version = "1.3.2.1"
+$version = "1.4.0-beta"
 $info = @{
     product = "ExLlamaSharp"
     version = $version

packaging/ExLlamaSharp.iss

@@ -3,7 +3,7 @@
 ;   & "${env:LocalAppData}\Programs\Inno Setup 6\ISCC.exe" packaging\ExLlamaSharp.iss
 
 #define MyAppName "ExLlamaSharp"
-#define MyAppVersion "1.3.2.1"
+#define MyAppVersion "1.4.0-beta"
 #define MyAppPublisher "ExLlamaSharp"
 #define MyAppURL "http://127.0.0.1:14563"
 ; Stage folder produced by Build-Installer.ps1 (relative to this .iss)
@@ -13,8 +13,8 @@
 AppId={{8F3E2A91-6C4B-4D7E-9A12-E5B8C0D4F617}
 AppName={#MyAppName}
 AppVersion={#MyAppVersion}
-VersionInfoVersion=1.3.2.1
-VersionInfoProductVersion=1.3.2.1
+VersionInfoVersion=1.4.0.0
+VersionInfoProductVersion=1.4.0.0
 AppMutex=Global\ExLlamaSharp.Server
 AppPublisher={#MyAppPublisher}
 AppPublisherURL={#MyAppURL}

packaging/Redeploy-Local.ps1

@@ -78,12 +78,14 @@ $elevLines = @(
     'if ($LASTEXITCODE -ge 8) { throw "robocopy failed $LASTEXITCODE" }'
     '$tray = Join-Path "' + $PubTray + '" "ExLlamaSharp.Tray.exe"'
     'if (Test-Path $tray) { Copy-Item $tray (Join-Path $dest "ExLlamaSharp.Tray.exe") -Force; Log "copied Tray.exe" }'
-    '$workerSrc = Join-Path "' + $Root + '" "tools\exl3_worker\worker.py"'
+    '$workerSrcDir = Join-Path "' + $Root + '" "tools\exl3_worker"'
     '$workerDstDir = Join-Path $dest "tools\exl3_worker"'
-    'if (Test-Path $workerSrc) {'
+    'if (Test-Path $workerSrcDir) {'
     '    New-Item -ItemType Directory -Force -Path $workerDstDir | Out-Null'
-    '    Copy-Item $workerSrc (Join-Path $workerDstDir "worker.py") -Force'
-    '    Log "copied worker.py"'
+    '    Copy-Item (Join-Path $workerSrcDir "worker.py") (Join-Path $workerDstDir "worker.py") -Force'
+    '    $launch = Join-Path $workerSrcDir "launch.py"'
+    '    if (Test-Path $launch) { Copy-Item $launch (Join-Path $workerDstDir "launch.py") -Force }'
+    '    Log "copied exl3_worker scripts"'
     '}'
     '$data = "' + $DataRoot + '"'
     'New-Item -ItemType Directory -Force -Path $data | Out-Null'

src/ExLlamaSharp.Server/Components/Pages/About.razor

@@ -40,9 +40,25 @@ else
             @if (_info.Gpu.Available)
             {
                 <ul class="mb-0">
-                    <li>@_info.Gpu.Name</li>
-                    <li>VRAM: @(_info.Gpu.VramTotalMb?.ToString("0") ?? "?") MB</li>
-                    <li>Compute: @(_info.Gpu.ComputeCapability ?? "—")</li>
+                    @if (_info.Gpu.Devices.Count > 0)
+                    {
+                        @foreach (var g in _info.Gpu.Devices)
+                        {
+                            <li>
+                                PCI @g.Index
+                                @(g.CudaIndex is int cuda ? $" — cuda:{cuda}" : " — not in CUDA_VISIBLE_DEVICES")
+                                — @g.Name — @g.VramUsedMb.ToString("0") / @g.VramTotalMb.ToString("0") MB
+                                @(string.IsNullOrEmpty(g.Uuid) ? "" : $" — {g.Uuid}")
+                                @(g.DisplayActive == true ? " — display" : "")
+                            </li>
+                        }
+                    }
+                    else
+                    {
+                        <li>@_info.Gpu.Name</li>
+                        <li>VRAM: @(_info.Gpu.VramTotalMb?.ToString("0") ?? "?") MB</li>
+                    }
+                    <li>Primary (highest VRAM): @_info.Gpu.Name</li>
                 </ul>
             }
             else