Running models larger in size than the GPU memory using CPU

[K4TEL]

Figure out how to run models larger in parameter size than the GPU memory available - use CPU memory for this matter

The unified memory mechanism is to be explored and applied in practice

[K4TEL]

Best fit: vLLM's UVA (Unified Virtual Addressing) CPU offload

Given the profile you describe — one GPU card, single node, 20+ CPU cores, Linux cluster — the closest thing to a real "unified memory mechanism" that's actually production-supported (not experimental) is vLLM's cpu_offload_gb with the UVA backend, not Ollama's layer-split and not raw CUDA UVM. Here's why, and how it differs from what you're already doing.

The real architectural choice, not just a flag

There are two genuinely different ways to use CPU memory when a model is bigger than VRAM, and they have opposite implications for a node that's also running your 32-worker async queue:

1. Compute offload (what Ollama/llama.cpp does today). The layers that don't fit on GPU are executed on CPU — actual matmuls, using CPU threads. num_gpu: 20 means only the first 20 layers go to GPU, the rest use CPU. This genuinely uses your 20+ cores as compute, not just RAM — but it competes for those same cores with FastText, regex, and the queue-polling workers your pipeline already runs concurrently.

2. Memory-only offload via unified/zero-copy addressing (vLLM's UVA, and raw CUDA UVM). The GPU stays the only compute engine. Weights that don't fit in VRAM sit in CPU-pinned memory, and the GPU reads them directly through a unified address space — no CPU compute involved at all. vLLM documents this explicitly: "uva": UVA (Unified Virtual Addressing) zero-copy offloading, configured via cpu_offload_gb — the space in GiB to offload to CPU, per GPU. This can be seen as a virtual way to increase the GPU memory size: a 24 GB GPU set to 10 effectively becomes a 34 GB GPU.

For your setup specifically — where the CPU pool is a scarce, contended resource doing FastText/regex/queue work — option 2 is the better fit. It lets you grow effective GPU capacity using system RAM without stealing cycles from the CPU workers, which matches the split you already designed (GPU = memory-bound transformer work, CPU = compute-bound everything else).

How to apply it in practice

vllm serve Qwen/Qwen2.5-14B-Instruct-AWQ \
  --cpu-offload-gb 12 \
  --gpu-memory-utilization 0.90

This treats your physical card as VRAM + 12GB of usable model space. Note that this requires fast CPU-GPU interconnect, as part of the model is loaded from CPU memory to GPU memory on the fly in each forward pass — so the practical ceiling depends on whether your node has PCIe Gen4/5 x16 to that card or something narrower; that's worth checking before committing to this path.

One important nuance to flag for the issue

vLLM's maintainers explicitly declined to support full, automatic, system-wide CUDA Unified Memory oversubscription as a general feature ([Feature]: Support for NVIDIA Unified memory #10267 (unified memory support, closed), marked not-planned). What you get instead is the narrower, scoped cpu_offload_gb/UVA mechanism above — controlled, per-GPU, with a hard GiB budget you set yourself. That's a meaningful distinction for the issue writeup: "unified memory" in vLLM means a bounded, opt-in offload pool, not transparent automatic paging.

Where this leaves the other options

Option	Uses your 20+ CPUs for...	Fit for your node
vLLM cpu_offload_gb (UVA)	Memory only — GPU does all compute	Best match for "unified memory + CPU RAM" with cores reserved for the existing pipeline
Ollama/llama.cpp num_gpu split	Memory and compute	Fine if the LLM call isn't running concurrently with the heavy CPU queue, but competes for cores if it is
GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 (llama.cpp)	Memory only, via raw CUDA UVM	Same idea as vLLM's UVA but less mature outside Grace Hopper-class NVLink hardware — testing on dense models showed it performed very poorly without manual tensor-placement patches
HF Accelerate device_map="auto"	Memory only, layer-by-layer	Only relevant for your smaller Transformers-native models (perplexity scorer, embeddings), not the Ollama-served LLMs

Honest caveat before you commit engineering time

This means adding vLLM as a second serving runtime alongside Ollama, which is a real infra decision, not just a config flag — worth deciding deliberately rather than as a side effect of this issue. And regardless of mechanism, the PCIe bottleneck doesn't disappear: even vLLM's own framing is "virtual" extra VRAM, not free extra VRAM — every offloaded weight still pays a transfer cost per forward pass. So I'd treat this as: vLLM UVA for headroom on Qwen-14B/GLM-4v-9b when they occasionally exceed VRAM, while keeping AWQ/quantization as the primary strategy so offload is the exception, not the steady state.