Skip to content

models.md

Latest commit

 

History

History
49 lines (42 loc) · 3.92 KB

models.md

File metadata and controls

49 lines (42 loc) · 3.92 KB
Raw

Supported models

FreeToken loads HF safetensors checkpoints directly (plus native GGUF for Gemma-4). The checkpoints below are known-good — the prebuilt kernels are tuned for them; other checkpoints of the same architectures work too.

Model HF checkpoints
DeepSeek-V4 deepseek-ai/DeepSeek-V4-Flash-0731
GLM-5.3-Flash RedHatAI/GLM-5.3-Flash-NVFP4
GLM-5.2 nvidia/GLM-5.2-NVFP4
GLM-4.7 nvidia/GLM-4.7-NVFP4
Qwen3.8-Flash-Next Qwen/Qwen3.8-Flash-Next-FP8, RadixArk/Qwen3.8-Flash-Next-NVFP4
Qwen3.6 / Qwen3.5 MoE Qwen/Qwen3.6-35B-A3B (-FP8), nvidia/Qwen3.6-35B-A3B-NVFP4, Qwen/Qwen3.5-35B-A3B (-FP8)
Qwen3.8 / Qwen3.6 dense Qwen/Qwen3.8-27B (-FP8), RadixArk/Qwen3.8-27B-NVFP4, Qwen/Qwen3.6-27B (-FP8), nvidia/Qwen3.6-27B-NVFP4
Qwen3-MoE Qwen/Qwen3-30B-A3B
gpt-oss openai/gpt-oss-120b, openai/gpt-oss-20b
Gemma-4 google/gemma-4-26B-A4B-it, nvidia/Gemma-4-26B-A4B-NVFP4, google/gemma-4-12B-it, nvidia/Gemma-4-31B-IT-NVFP4 ..
MiniMax-M2.5 nvidia/MiniMax-M2.5-NVFP4
Muse-Glimmer meta-models/Muse-Glimmer-30B, RedHatAI/Muse-Glimmer-30B-NVFP4

MoE backends

ft serve --moe-backend {auto,fused,offload,cpu,hybrid}:

  • fused — experts resident on GPU (needs the VRAM); never auto-selected.
  • offload — experts live in host RAM, an LRU cache of expert slots on GPU; misses stream over PCIe.
  • cpu — misses are computed on the CPU instead of fetched.
  • hybrid — per step, fetches some misses over PCIe and computes the rest on CPU, overlapped. Run ft bench bw once per machine to calibrate the split.
  • auto — dense models always resolve to fused; MoE models resolve to offload, upgraded to hybrid when a cached ft bench bw profile recommends it.

Notes

  • ft checkpoint conversion is optional — it pre-converts a checkpoint into FreeToken's fast-load format, and ft serve --model auto-detects the result.
  • DeepSeek-V4 checkpoints must keep the inference/config.json subdir — the authoritative model args are read from there.
  • Qwen3.8-Flash-Next keeps a 47.7 GiB PLE n-gram table pinned in host RAM.
  • Multimodal checkpoints are served text-only.
  • --kv-cache-dtype fp8 (see cli.md) covers the plain paged, hybrid-SWA and QSA sparse KV pools — gpt-oss, Qwen3/3.5/3.6, GLM-4.x, Gemma-4, MiniMax-M2.5, Muse-Glimmer, Llama/Qwen2/Mistral, Qwen3.8-Flash-Next (on QSA only the selected K/V rows are read back as codes; block selection keeps 16-bit index keys). MLA/DSA (GLM-5.2), DeepSeek-V4's tiered pool and MiniMax-M3's block-sparse pool stay 16-bit and reject it.