Qwen3.8-Flash-Next-NVFP4 cannot start on a 12 GB RTX 3060: ~9.9 GiB of unquantized (BF16) attention/embedding weights leave no room for MoE cache + KV

[erikash]

What happened

Trying to serve RadixArk/Qwen3.8-Flash-Next-NVFP4 on a 12 GB RTX 3060 with plenty of host RAM (196 GB). The dense (non-expert) weights alone occupy ~9.9 GiB of the 11.50 GiB free, so no backend can allocate its minimum GPU-side cache and startup fails every time.

Expected: with --moe-backend cpu (experts computed on CPU) the GPU-side MoE cache should be small enough, or --moe-cache-size should let me shrink it. Instead the CPU backend still allocates a fixed two-layer prefill buffer (moe_cache_size=1024, ~3 GiB) and ignores --moe-cache-size.

Related: #150 (same assertion, comment there shows the negative budget on a 10 GiB 4090) and #214 (Qwen3.8-Flash-Next support). This report adds a real 12 GB card, and shows that --moe-backend cpu does not help either.

Attempt 1 – default (auto → offload)

Free memory before loading model: 11.50 GiB
Resolved config: moe_backend='offload', attention_backend='qsa_sparse', cache_type='hybrid_radix', page_size=64
PLE disk backend: io_uring, O_DIRECT, wait-sync
AssertionError: cache budget too small: minimum plan (moe=1024 slots, kv=128 pages) needs 3046637568 B > budget -1868787708 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)

Attempt 2 – --moe-backend cpu --memory-ratio 0.98 --kv-reserve-tokens 4096

MoE backend 'cpu': decode computes experts on CPU; GPU keeps a two-layer prefill buffer (moe_cache_size=1024)
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 800.00 MiB. GPU 0 has a total capacity of 11.63 GiB of which 202.88 MiB is free. Including non-PyTorch memory, this process has 11.30 GiB memory in use.
  File ".../freetoken/engine/engine.py", line 628, in _init_offload_moe_cache
  File ".../freetoken/moe/offload_cache.py", line 339, in set_bank_sources

Attempt 3 – --moe-backend cpu --memory-ratio 0.99 --moe-cache-size 256 --kv-reserve-tokens 2048 --max-running-requests 1

Identical OOM. The log still prints moe_cache_size=1024, so --moe-cache-size is not honoured by the cpu backend.

Questions / suggestions:

1. Is a ~10 GiB resident footprint for this checkpoint expected on a 12 GB card, i.e. is 16 GB the practical floor for Qwen3.8-Flash-Next? If so, documenting the minimum VRAM per model in docs/models.md would save people a 135 GB download.
2. Could the cpu backend honour --moe-cache-size (or size its prefill buffer from the remaining budget) so that small cards can at least run in CPU-expert mode?
3. Could the budget assertion report the resident weight size explicitly? "budget -1868787708 B" plus the hint to raise memory_ratio is misleading when the budget is negative before any cache is allocated.

For comparison, the same GPU runs the Unsloth UD-Q4_K_XL GGUF of this model in llama.cpp with -ngl 99 --cpu-moe at ~8 GiB VRAM (21 tok/s decode), so a CPU-expert mode with a smaller GPU footprint seems feasible for this card.

How did you install FreeToken

Nightly wheel from the FreeToken-Web beta release (needed for Qwen4ExpForConditionalGeneration; PyPI 0.1.2 rejects the architecture), installed into a uv venv:

uv venv --python 3.12 .venv
uv pip install "freetoken[accel]"          # 0.1.2 from PyPI first
uv pip install ./freetoken-0.1.2+gaf71ba432-cp312-cp312-linux_x86_64.whl \
               ./freetoken_kernel_cache-0.1.2+cu130.gaf71ba432-py3-none-linux_x86_64.whl

FreeToken version

0.1.2+gaf71ba432 (nightly, commit af71ba4). ft --version prints freetoken version 0.1.2.

OS

Linux

OS details

Ubuntu 26.04.1 LTS, kernel 7.0.0-31-generic, Python 3.12.14 (uv-managed), torch 2.11.0+cu130, CUDA toolkit 13.3 (/usr/local/cuda), running as a Proxmox VM with the GPU passed through via vfio-pci.

GPU and driver

NVIDIA GeForce RTX 3060 12 GB (GA106, sm_86), driver 595.71.05. Single GPU. nvidia-smi shows 1 MiB used before launch; the engine reports 11.50 GiB free.

CPU and system RAM

64 vCPU (AMD EPYC 7702P host), 196 GB RAM in the VM.

Checkpoint

RadixArk/Qwen3.8-Flash-Next-NVFP4 (126 GiB on disk, hf_quant_config producer modelopt 0.46.0).

Command

ft serve --model-path ~/models/hf/Qwen3.8-Flash-Next-NVFP4 --host 0.0.0.0 --port 8080 \
  --served-model-name qwen3.8-flash --reasoning-parser qwen3 --tool-call-parser qwen \
  --moe-backend cpu --memory-ratio 0.99 --moe-cache-size 256 --kv-reserve-tokens 2048 --max-running-requests 1

(plus the two earlier variants described above)

Full log

Attempt 3, progress bars stripped:

ft serve log

[2026-09-07|10:25:34] INFO     Parsed arguments:
ServerArgs(model_path='~/models/hf/Qwen3.8-Flash-Next-NVFP4', tp_info=DistributedInfo(rank=0, size=1), dtype=torch.bfloat16, max_running_req=1, attention_backend='auto', moe_backend='cpu', nvfp4_backend='triton', ple_backend='disk', expert_load='auto', moe_cache_size=256, moe_cache_rate=None, moe_cache_auto=False, kv_reserve_tokens=2048, moe_cache_policy='lru', moe_prefill_overlap=True, moe_prefill_hit_d2d=False, moe_collect_stats=False, moe_cpu_threads=0, moe_cpu_layers=None, moe_hybrid_max_fetch=-1, cuda_graph_bs=None, cuda_graph_max_bs=None, page_size=1, memory_ratio=0.99, linear_state_cache_ratio=2.0, swa_full_tokens_ratio=0.2, swa_num_pages_override=None, distributed_timeout=60.0, use_dummy_weight=False, use_pynccl=True, max_seq_len_override=None, num_page_override=None, num_token_override=None, max_extend_tokens=8192, cache_type='radix', offline_mode=False, decode_log_interval=40, special_token_ckpt=False, _unique_suffix='.pid=8334', server_host='0.0.0.0', server_port=8080, num_tokenizer=0, silent_output=False, shell_mode=False, served_model_name='qwen3.8-flash', tool_call_parser='qwen', reasoning_parser='qwen3', sampling_defaults='model', max_output_tokens=None, enable_cache_report=False, cors_origins='tauri://localhost,http://tauri.localhost,http://localhost:1420', gpu=(), gpu_assigned=None)
[2026-09-07|10:25:34|FrontendAPI] INFO     Default sampling config (source=model): temperature=1.0, top_k=20, top_p=0.95
INFO:     Started server process [8334]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
~/freetoken/.venv/lib/python3.12/site-packages/freetoken/engine/engine.py:1034: FutureWarning: torch.cuda._set_allocator_settings is deprecated. Use torch._C._accelerator_setAllocatorSettings instead.
  torch.cuda.memory._set_allocator_settings("expandable_segments:True")
[2026-09-07|10:25:37|core|rank=0] INFO     Enabled expandable_segments (override via PYTORCH_ALLOC_CONF)
[2026-09-07|10:25:37|core|rank=0] INFO     Auto-selected attention backend: qsa_sparse
[2026-09-07|10:25:37|core|rank=0] WARNING  Page size is overridden to 64 for the qsa_sparse backend
[2026-09-07|10:25:37|core|rank=0] INFO     MoE backend 'cpu': decode computes experts on CPU; GPU keeps a two-layer prefill buffer (moe_cache_size=1024)
[2026-09-07|10:25:37|core|rank=0] INFO     Resolved config: moe_backend='cpu', attention_backend='qsa_sparse', cache_type='hybrid_radix', page_size=64
[2026-09-07|10:25:37|core|rank=0] INFO     Free memory before loading model: 11.50 GiB
[2026-09-07|10:25:41|core|rank=0] INFO     PLE disk backend: io_uring, O_DIRECT, wait-sync
[2026-09-07|10:25:42|core|rank=0] INFO     expert banks: slow path (parallel build)
Loading experts (parallel):   0%|          | 0.00/63.3G [00:00<?, ?B/s]
Loading experts (parallel):   1%|          | 338M/63.3G [00:00<02:32, 443MB/s]
Loading experts (parallel):   1%|          | 675M/63.3G [00:01<01:31, 732MB/s]
Loading experts (parallel):   2%|▏         | 0.99G/63.3G [00:01<01:14, 902MB/s]
Loading experts (parallel):   2%|▏         | 1.32G/63.3G [00:01<01:08, 976MB/s]
Loading experts (parallel):   3%|▎         | 1.65G/63.3G [00:02<01:24, 781MB/s]
Loading experts (parallel):   3%|▎         | 1.98G/63.3G [00:02<01:15, 876MB/s]
Loading experts (parallel):   4%|▎         | 2.31G/63.3G [00:02<01:08, 962MB/s]
Loading experts (parallel):   4%|▍         | 2.64G/63.3G [00:03<01:03, 1.02GB/s]
Loading experts (parallel):   5%|▍         | 2.97G/63.3G [00:03<01:04, 1.00GB/s]
Loading experts (parallel):   5%|▌         | 3.30G/63.3G [00:03<01:00, 1.06GB/s]
Loading experts (parallel):   6%|▌         | 3.63G/63.3G [00:04<00:59, 1.08GB/s]
Loading experts (parallel):   6%|▌         | 3.96G/63.3G [00:04<00:56, 1.12GB/s]
Loading experts (parallel):   7%|▋         | 4.29G/63.3G [00:04<01:00, 1.04GB/s]
Loading experts (parallel):   7%|▋         | 4.62G/63.3G [00:05<00:59, 1.07GB/s]
Loading experts (parallel):   8%|▊         | 4.95G/63.3G [00:05<00:57, 1.10GB/s]
Loading experts (parallel):   8%|▊         | 5.28G/63.3G [00:05<00:55, 1.11GB/s]
Loading experts (parallel):   9%|▉         | 5.61G/63.3G [00:06<00:59, 1.05GB/s]
Loading experts (parallel):   9%|▉         | 5.94G/63.3G [00:06<00:56, 1.09GB/s]
Loading experts (parallel):  10%|▉         | 6.27G/63.3G [00:06<00:55, 1.11GB/s]
Loading experts (parallel):  10%|█         | 6.60G/63.3G [00:07<00:54, 1.12GB/s]
Loading experts (parallel):  11%|█         | 6.93G/63.3G [00:07<00:57, 1.06GB/s]
Loading experts (parallel):  11%|█▏        | 7.26G/63.3G [00:07<00:54, 1.10GB/s]
Loading experts (parallel):  12%|█▏        | 7.58G/63.3G [00:08<00:52, 1.13GB/s]
Loading experts (parallel):  12%|█▏        | 7.91G/63.3G [00:08<00:51, 1.15GB/s]
Loading experts (parallel):  13%|█▎        | 8.24G/63.3G [00:08<00:54, 1.09GB/s]
Loading experts (parallel):  14%|█▎        | 8.57G/63.3G [00:08<00:50, 1.15GB/s]
Loading experts (parallel):  14%|█▍        | 8.90G/63.3G [00:09<00:51, 1.13GB/s]
Loading experts (parallel):  15%|█▍        | 9.23G/63.3G [00:09<00:49, 1.17GB/s]
Loading experts (parallel):  15%|█▌        | 9.56G/63.3G [00:09<00:56, 1.03GB/s]
Loading experts (parallel):  16%|█▌        | 9.89G/63.3G [00:10<00:53, 1.08GB/s]
Loading experts (parallel):  16%|█▌        | 10.2G/63.3G [00:10<00:52, 1.08GB/s]
Loading experts (parallel):  17%|█▋        | 10.6G/63.3G [00:10<00:52, 1.07GB/s]
Loading experts (parallel):  17%|█▋        | 10.9G/63.3G [00:11<00:54, 1.04GB/s]
Loading experts (parallel):  18%|█▊        | 11.2G/63.3G [00:11<00:51, 1.09GB/s]
Loading experts (parallel):  18%|█▊        | 11.5G/63.3G [00:11<00:49, 1.11GB/s]
Loading experts (parallel):  19%|█▊        | 11.9G/63.3G [00:12<00:48, 1.13GB/s]
Loading experts (parallel):  19%|█▉        | 12.2G/63.3G [00:12<00:52, 1.05GB/s]
Loading experts (parallel):  20%|█▉        | 12.5G/63.3G [00:12<00:52, 1.04GB/s]
Loading experts (parallel):  20%|██        | 12.9G/63.3G [00:13<00:51, 1.05GB/s]
Loading experts (parallel):  21%|██        | 13.2G/63.3G [00:13<00:48, 1.10GB/s]
Loading experts (parallel):  21%|██▏       | 13.5G/63.3G [00:13<00:51, 1.04GB/s]
Loading experts (parallel):  22%|██▏       | 13.9G/63.3G [00:14<00:49, 1.08GB/s]
Loading experts (parallel):  22%|██▏       | 14.2G/63.3G [00:14<00:46, 1.14GB/s]
Loading experts (parallel):  23%|██▎       | 14.5G/63.3G [00:14<00:47, 1.11GB/s]
Loading experts (parallel):  23%|██▎       | 14.8G/63.3G [00:15<00:49, 1.04GB/s]
Loading experts (parallel):  24%|██▍       | 15.2G/63.3G [00:15<00:46, 1.10GB/s]
Loading experts (parallel):  24%|██▍       | 15.5G/63.3G [00:15<00:45, 1.13GB/s]
Loading experts (parallel):  25%|██▍       | 15.8G/63.3G [00:16<00:43, 1.16GB/s]
Loading experts (parallel):  26%|██▌       | 16.2G/63.3G [00:16<00:47, 1.06GB/s]
Loading experts (parallel):  26%|██▌       | 16.5G/63.3G [00:16<00:45, 1.11GB/s]
Loading experts (parallel):  27%|██▋       | 16.8G/63.3G [00:17<00:43, 1.14GB/s]
Loading experts (parallel):  27%|██▋       | 17.1G/63.3G [00:17<00:42, 1.16GB/s]
Loading experts (parallel):  28%|██▊       | 17.5G/63.3G [00:17<00:45, 1.07GB/s]
Loading experts (parallel):  28%|██▊       | 17.8G/63.3G [00:18<00:43, 1.11GB/s]
Loading experts (parallel):  29%|██▊       | 18.1G/63.3G [00:18<00:42, 1.15GB/s]
Loading experts (parallel):  29%|██▉       | 18.5G/63.3G [00:18<00:41, 1.16GB/s]
Loading experts (parallel):  30%|██▉       | 18.8G/63.3G [00:19<00:44, 1.08GB/s]
Loading experts (parallel):  30%|███       | 19.1G/63.3G [00:19<00:45, 1.05GB/s]
Loading experts (parallel):  31%|███       | 19.5G/63.3G [00:19<00:42, 1.11GB/s]
Loading experts (parallel):  31%|███       | 19.8G/63.3G [00:19<00:40, 1.14GB/s]
Loading experts (parallel):  32%|███▏      | 20.1G/63.3G [00:20<00:44, 1.05GB/s]
Loading experts (parallel):  32%|███▏      | 20.4G/63.3G [00:20<00:41, 1.12GB/s]
Loading experts (parallel):  33%|███▎      | 20.8G/63.3G [00:20<00:41, 1.11GB/s]
Loading experts (parallel):  33%|███▎      | 21.1G/63.3G [00:21<00:39, 1.16GB/s]
Loading experts (parallel):  34%|███▍      | 21.4G/63.3G [00:21<00:42, 1.05GB/s]
Loading experts (parallel):  34%|███▍      | 21.8G/63.3G [00:21<00:40, 1.09GB/s]
Loading experts (parallel):  35%|███▍      | 22.1G/63.3G [00:22<00:39, 1.13GB/s]
Loading experts (parallel):  35%|███▌      | 22.4G/63.3G [00:22<00:41, 1.05GB/s]
Loading experts (parallel):  36%|███▌      | 22.8G/63.3G [00:22<00:43, 1.00GB/s]
Loading experts (parallel):  36%|███▋      | 23.1G/63.3G [00:23<00:41, 1.03GB/s]
Loading experts (parallel):  37%|███▋      | 23.4G/63.3G [00:23<00:41, 1.04GB/s]
Loading experts (parallel):  37%|███▋      | 23.7G/63.3G [00:23<00:38, 1.11GB/s]
Loading experts (parallel):  38%|███▊      | 24.1G/63.3G [00:24<00:40, 1.04GB/s]
Loading experts (parallel):  39%|███▊      | 24.4G/63.3G [00:24<00:37, 1.10GB/s]
Loading experts (parallel):  39%|███▉      | 24.7G/63.3G [00:24<00:36, 1.12GB/s]
Loading experts (parallel):  40%|███▉      | 25.1G/63.3G [00:25<00:36, 1.14GB/s]
Loading experts (parallel):  40%|████      | 25.4G/63.3G [00:25<00:38, 1.05GB/s]
Loading experts (parallel):  41%|████      | 25.7G/63.3G [00:25<00:37, 1.09GB/s]
Loading experts (parallel):  41%|████      | 26.1G/63.3G [00:26<00:36, 1.10GB/s]
Loading experts (parallel):  42%|████▏     | 26.4G/63.3G [00:26<00:34, 1.16GB/s]
Loading experts (parallel):  42%|████▏     | 26.7G/63.3G [00:26<00:37, 1.06GB/s]
Loading experts (parallel):  43%|████▎     | 27.0G/63.3G [00:27<00:35, 1.10GB/s]
Loading experts (parallel):  43%|████▎     | 27.4G/63.3G [00:27<00:34, 1.11GB/s]
Loading experts (parallel):  44%|████▎     | 27.7G/63.3G [00:27<00:33, 1.13GB/s]
Loading experts (parallel):  44%|████▍     | 28.0G/63.3G [00:28<00:36, 1.03GB/s]
Loading experts (parallel):  45%|████▍     | 28.4G/63.3G [00:28<00:34, 1.10GB/s]
Loading experts (parallel):  45%|████▌     | 28.7G/63.3G [00:28<00:32, 1.13GB/s]
Loading experts (parallel):  46%|████▌     | 29.0G/63.3G [00:29<00:32, 1.15GB/s]
Loading experts (parallel):  46%|████▋     | 29.4G/63.3G [00:29<00:34, 1.06GB/s]
Loading experts (parallel):  47%|████▋     | 29.7G/63.3G [00:29<00:33, 1.09GB/s]
Loading experts (parallel):  47%|████▋     | 30.0G/63.3G [00:30<00:32, 1.10GB/s]
Loading experts (parallel):  48%|████▊     | 30.3G/63.3G [00:30<00:31, 1.11GB/s]
Loading experts (parallel):  48%|████▊     | 30.7G/63.3G [00:30<00:34, 1.03GB/s]
Loading experts (parallel):  49%|████▉     | 31.0G/63.3G [00:31<00:32, 1.08GB/s]
Loading experts (parallel):  49%|████▉     | 31.3G/63.3G [00:31<00:30, 1.13GB/s]
Loading experts (parallel):  50%|████▉     | 31.7G/63.3G [00:31<00:29, 1.16GB/s]
Loading experts (parallel):  51%|█████     | 32.0G/63.3G [00:31<00:31, 1.08GB/s]
Loading experts (parallel):  51%|█████     | 32.3G/63.3G [00:32<00:29, 1.11GB/s]
Loading experts (parallel):  52%|█████▏    | 32.6G/63.3G [00:32<00:28, 1.16GB/s]
Loading experts (parallel):  52%|█████▏    | 33.0G/63.3G [00:32<00:27, 1.17GB/s]
Loading experts (parallel):  53%|█████▎    | 33.3G/63.3G [00:33<00:30, 1.07GB/s]
Loading experts (parallel):  53%|█████▎    | 33.6G/63.3G [00:33<00:28, 1.11GB/s]
Loading experts (parallel):  54%|█████▎    | 34.0G/63.3G [00:33<00:27, 1.14GB/s]
Loading experts (parallel):  54%|█████▍    | 34.3G/63.3G [00:34<00:26, 1.16GB/s]
Loading experts (parallel):  55%|█████▍    | 34.6G/63.3G [00:34<00:28, 1.07GB/s]
Loading experts (parallel):  55%|█████▌    | 35.0G/63.3G [00:34<00:27, 1.10GB/s]
Loading experts (parallel):  56%|█████▌    | 35.3G/63.3G [00:35<00:26, 1.12GB/s]
Loading experts (parallel):  56%|█████▌    | 35.6G/63.3G [00:35<00:26, 1.14GB/s]
Loading experts (parallel):  57%|█████▋    | 35.9G/63.3G [00:35<00:27, 1.07GB/s]
Loading experts (parallel):  57%|█████▋    | 36.3G/63.3G [00:36<00:25, 1.13GB/s]
Loading experts (parallel):  58%|█████▊    | 36.6G/63.3G [00:36<00:24, 1.15GB/s]
Loading experts (parallel):  58%|█████▊    | 36.9G/63.3G [00:36<00:23, 1.18GB/s]
Loading experts (parallel):  59%|█████▉    | 37.3G/63.3G [00:37<00:26, 1.08GB/s]
Loading experts (parallel):  59%|█████▉    | 37.6G/63.3G [00:37<00:24, 1.11GB/s]
Loading experts (parallel):  60%|█████▉    | 37.9G/63.3G [00:37<00:23, 1.16GB/s]
Loading experts (parallel):  60%|██████    | 38.3G/63.3G [00:37<00:22, 1.17GB/s]
Loading experts (parallel):  61%|██████    | 38.6G/63.3G [00:38<00:24, 1.08GB/s]
Loading experts (parallel):  61%|██████▏   | 38.9G/63.3G [00:38<00:22, 1.14GB/s]
Loading experts (parallel):  62%|██████▏   | 39.2G/63.3G [00:38<00:22, 1.17GB/s]
Loading experts (parallel):  62%|██████▏   | 39.6G/63.3G [00:39<00:21, 1.19GB/s]
Loading experts (parallel):  63%|██████▎   | 39.9G/63.3G [00:39<00:22, 1.10GB/s]
Loading experts (parallel):  64%|██████▎   | 40.2G/63.3G [00:39<00:22, 1.12GB/s]
Loading experts (parallel):  64%|██████▍   | 40.6G/63.3G [00:40<00:21, 1.14GB/s]
Loading experts (parallel):  65%|██████▍   | 40.9G/63.3G [00:40<00:20, 1.20GB/s]
Loading experts (parallel):  65%|██████▌   | 41.2G/63.3G [00:40<00:21, 1.10GB/s]
Loading experts (parallel):  66%|██████▌   | 41.6G/63.3G [00:41<00:21, 1.10GB/s]
Loading experts (parallel):  66%|██████▌   | 41.9G/63.3G [00:41<00:20, 1.14GB/s]
Loading experts (parallel):  67%|██████▋   | 42.2G/63.3G [00:41<00:19, 1.17GB/s]
Loading experts (parallel):  67%|██████▋   | 42.5G/63.3G [00:42<00:20, 1.07GB/s]
Loading experts (parallel):  68%|██████▊   | 42.9G/63.3G [00:42<00:20, 1.08GB/s]
Loading experts (parallel):  68%|██████▊   | 43.2G/63.3G [00:42<00:19, 1.13GB/s]
Loading experts (parallel):  69%|██████▊   | 43.5G/63.3G [00:42<00:18, 1.17GB/s]
Loading experts (parallel):  69%|██████▉   | 43.9G/63.3G [00:43<00:19, 1.04GB/s]
Loading experts (parallel):  70%|██████▉   | 44.2G/63.3G [00:43<00:19, 1.06GB/s]
Loading experts (parallel):  70%|███████   | 44.5G/63.3G [00:43<00:18, 1.10GB/s]
Loading experts (parallel):  71%|███████   | 44.9G/63.3G [00:44<00:17, 1.14GB/s]
Loading experts (parallel):  71%|███████▏  | 45.2G/63.3G [00:44<00:18, 1.08GB/s]
Loading experts (parallel):  72%|███████▏  | 45.5G/63.3G [00:44<00:17, 1.11GB/s]
Loading experts (parallel):  72%|███████▏  | 45.8G/63.3G [00:45<00:16, 1.14GB/s]
Loading experts (parallel):  73%|███████▎  | 46.2G/63.3G [00:45<00:15, 1.16GB/s]
Loading experts (parallel):  73%|███████▎  | 46.5G/63.3G [00:45<00:16, 1.09GB/s]
Loading experts (parallel):  74%|███████▍  | 46.8G/63.3G [00:46<00:15, 1.13GB/s]
Loading experts (parallel):  74%|███████▍  | 47.2G/63.3G [00:46<00:15, 1.15GB/s]
Loading experts (parallel):  75%|███████▍  | 47.5G/63.3G [00:46<00:14, 1.15GB/s]
Loading experts (parallel):  76%|███████▌  | 47.8G/63.3G [00:47<00:15, 1.08GB/s]
Loading experts (parallel):  76%|███████▌  | 48.1G/63.3G [00:47<00:15, 1.08GB/s]
Loading experts (parallel):  77%|███████▋  | 48.5G/63.3G [00:47<00:14, 1.08GB/s]
Loading experts (parallel):  77%|███████▋  | 48.8G/63.3G [00:48<00:14, 1.08GB/s]
Loading experts (parallel):  78%|███████▊  | 49.1G/63.3G [00:48<00:14, 1.03GB/s]
Loading experts (parallel):  78%|███████▊  | 49.5G/63.3G [00:48<00:13, 1.07GB/s]
Loading experts (parallel):  79%|███████▊  | 49.8G/63.3G [00:49<00:13, 1.12GB/s]
Loading experts (parallel):  79%|███████▉  | 50.1G/63.3G [00:49<00:12, 1.12GB/s]
Loading experts (parallel):  80%|███████▉  | 50.5G/63.3G [00:49<00:13, 1.06GB/s]
Loading experts (parallel):  80%|████████  | 50.8G/63.3G [00:50<00:12, 1.10GB/s]
Loading experts (parallel):  81%|████████  | 51.1G/63.3G [00:50<00:11, 1.14GB/s]
Loading experts (parallel):  81%|████████  | 51.4G/63.3G [00:50<00:10, 1.18GB/s]
Loading experts (parallel):  82%|████████▏ | 51.8G/63.3G [00:51<00:11, 1.09GB/s]
Loading experts (parallel):  82%|████████▏ | 52.1G/63.3G [00:51<00:10, 1.13GB/s]
Loading experts (parallel):  83%|████████▎ | 52.4G/63.3G [00:51<00:10, 1.14GB/s]
Loading experts (parallel):  83%|████████▎ | 52.8G/63.3G [00:51<00:09, 1.15GB/s]
Loading experts (parallel):  84%|████████▍ | 53.1G/63.3G [00:52<00:12, 869MB/s] 
Loading experts (parallel):  84%|████████▍ | 53.4G/63.3G [00:52<00:11, 944MB/s]
Loading experts (parallel):  85%|████████▍ | 53.8G/63.3G [00:53<00:10, 1.00GB/s]
Loading experts (parallel):  85%|████████▌ | 54.1G/63.3G [00:53<00:09, 1.05GB/s]
Loading experts (parallel):  86%|████████▌ | 54.4G/63.3G [00:53<00:09, 958MB/s] 
Loading experts (parallel):  86%|████████▋ | 54.7G/63.3G [00:54<00:09, 981MB/s]
Loading experts (parallel):  87%|████████▋ | 55.1G/63.3G [00:54<00:08, 1.04GB/s]
Loading experts (parallel):  87%|████████▋ | 55.4G/63.3G [00:54<00:08, 998MB/s] 
Loading experts (parallel):  88%|████████▊ | 55.7G/63.3G [00:55<00:08, 962MB/s]
Loading experts (parallel):  89%|████████▊ | 56.1G/63.3G [00:55<00:07, 1.04GB/s]
Loading experts (parallel):  89%|████████▉ | 56.4G/63.3G [00:55<00:06, 1.06GB/s]
Loading experts (parallel):  90%|████████▉ | 56.7G/63.3G [00:56<00:06, 1.06GB/s]
Loading experts (parallel):  90%|█████████ | 57.1G/63.3G [00:56<00:06, 1.00GB/s]
Loading experts (parallel):  91%|█████████ | 57.4G/63.3G [00:56<00:06, 1.06GB/s]
Loading experts (parallel):  91%|█████████ | 57.7G/63.3G [00:57<00:05, 1.10GB/s]
Loading experts (parallel):  92%|█████████▏| 58.0G/63.3G [00:57<00:05, 1.12GB/s]
Loading experts (parallel):  92%|█████████▏| 58.4G/63.3G [00:57<00:05, 1.04GB/s]
Loading experts (parallel):  93%|█████████▎| 58.7G/63.3G [00:58<00:04, 1.10GB/s]
Loading experts (parallel):  93%|█████████▎| 59.0G/63.3G [00:58<00:04, 1.14GB/s]
Loading experts (parallel):  94%|█████████▎| 59.4G/63.3G [00:58<00:03, 1.17GB/s]
Loading experts (parallel):  94%|█████████▍| 59.7G/63.3G [00:59<00:03, 1.08GB/s]
Loading experts (parallel):  95%|█████████▍| 60.0G/63.3G [00:59<00:03, 1.09GB/s]
Loading experts (parallel):  95%|█████████▌| 60.4G/63.3G [00:59<00:02, 1.10GB/s]
Loading experts (parallel):  96%|█████████▌| 60.7G/63.3G [01:00<00:02, 1.10GB/s]
Loading experts (parallel):  96%|█████████▋| 61.0G/63.3G [01:00<00:02, 1.03GB/s]
Loading experts (parallel):  97%|█████████▋| 61.3G/63.3G [01:00<00:01, 1.09GB/s]
Loading experts (parallel):  97%|█████████▋| 61.7G/63.3G [01:01<00:01, 1.11GB/s]
Loading experts (parallel):  98%|█████████▊| 62.0G/63.3G [01:01<00:01, 1.16GB/s]
Loading experts (parallel):  98%|█████████▊| 62.3G/63.3G [01:01<00:01, 1.04GB/s]
Loading experts (parallel):  99%|█████████▉| 62.7G/63.3G [01:01<00:00, 1.21GB/s]
Loading experts (parallel):  99%|█████████▉| 63.0G/63.3G [01:02<00:00, 1.38GB/s]
Loading experts (parallel): 100%|██████████| 63.3G/63.3G [01:02<00:00, 1.54GB/s]
Process freetoken-TP0-scheduler:
[2026-09-07|10:26:48|FrontendAPI] ERROR    Backend supervisor: OutOfMemoryError: CUDA out of memory. Tried to allocate 800.00 MiB. GPU 0 has a total capacity of 11.63 GiB of which 202.88 MiB is free. Including non-PyTorch memory, this process has 11.30 GiB memory in use. Of the allocated memory 11.08 GiB is allocated by PyTorch, and 80.85 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
Traceback (most recent call last):
  File "~/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "~/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "~/freetoken/.venv/lib/python3.12/site-packages/freetoken/server/launch.py", line 81, in _run_scheduler
    scheduler = Scheduler(args)
                ^^^^^^^^^^^^^^^
  File "~/freetoken/.venv/lib/python3.12/site-packages/freetoken/scheduler/scheduler.py", line 65, in __init__
    self.engine = Engine(config)
                  ^^^^^^^^^^^^^^
  File "~/freetoken/.venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 343, in __init__
    self._init_offload_moe_cache(config)
  File "~/freetoken/.venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 628, in _init_offload_moe_cache
    cache.set_bank_sources(banks.sources, layer_residency=banks.layer_residency)
  File "~/freetoken/.venv/lib/python3.12/site-packages/freetoken/moe/offload_cache.py", line 339, in set_bank_sources
    self.bank_caches[name] = torch.empty(
                             ^^^^^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 800.00 MiB. GPU 0 has a total capacity of 11.63 GiB of which 202.88 MiB is free. Including non-PyTorch memory, this process has 11.30 GiB memory in use. Of the allocated memory 11.08 GiB is allocated by PyTorch, and 80.85 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
[2026-09-07|10:26:58|FrontendAPI] ERROR    Backend worker is gone and cannot be restarted; stopping the API server
INFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [8334]

Anything else

nvidia/Gemma-4-26B-A4B-NVFP4 works fine on the same setup with the offload backend (~37 tok/s decode), so the toolchain itself is healthy.

[erikash]

Follow-up: the ~9.9 GiB resident footprint is not the experts, it is everything the checkpoint leaves in BF16.

hf_quant_config.json of RadixArk/Qwen3.8-Flash-Next-NVFP4 excludes from quantization: embed_tokens, lm_head, *.self_attn.*, *.linear_attn.*, *.mlp.gate*, *.mlp.shared_expert.*, *hyper_connection*, mtp.*, *.ple.*. Only the routed experts are NVFP4, and those are the part that lives in host RAM anyway. Qwen/Qwen3.8-Flash-Next-FP8 has the same exclusion list, so there is no smaller checkpoint to try.

Rough BF16 size of the resident part (48 layers, hidden 2560, vocab 248320):

component	~GiB
embeddings + lm_head	2.4
36 linear-attention layers	3.4
12 full-attention layers	0.8
shared experts, routers, MTP, hyper-connections	1.0
CUDA context / kernels / workspaces	1.5–2

≈ 9–10 GiB before any MoE cache or KV, which matches the log.

So on a 12 GB card the blocker is the unquantized dense layers, not the MoE cache sizing. For comparison, the Unsloth UD-Q4_K_XL GGUF quantizes attention and embeddings too, and llama.cpp runs it with -ngl 99 --cpu-moe in ~8 GiB VRAM on the same GPU.

Two possible directions:

1. Document a minimum VRAM per checkpoint (this one is effectively ≥16 GB).
2. Support quantized (FP8/INT8/NVFP4) attention + embedding weights, either via ft checkpoint or by accepting checkpoints that quantize them, so the resident part fits small cards.