Skip to content

mistral-small-3.2-24b-nvfp4.md

Latest commit

 

History

History
163 lines (136 loc) · 8.17 KB

mistral-small-3.2-24b-nvfp4.md

File metadata and controls

163 lines (136 loc) · 8.17 KB
Raw

Fallback model: RedHatAI/Mistral-Small-3.2-24B-Instruct-2506-NVFP4

The opt-in dense warm fallback for the gateway fleet. The fleet runs one generate backend by default — the primary (sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP) keeps its solo headroom — so this is not loaded by default; wire it in as a second generate backend via the FALLBACK_* keys (below). When you do add a fallback, this dense Mistral is the recommended choice: it supersedes the Qwen3.6-35B-A3B MoE, which never loaded on this GB10 (OOM co-resident, stall solo — see docs/qwen3.6-35b-a3b-nvfp4.md). Mistral is dense, loads reliably, and serves text + tool calls. It is one entry in lobes's supported catalog (lobes overview --list); see docs/gateway-fleet.md for the fleet topology and the catalog-vs-warm distinction (what you can load vs. what's loaded now).

Source: https://huggingface.co/RedHatAI/Mistral-Small-3.2-24B-Instruct-2506-NVFP4.

Status: load-tested 2026-05-30 on the DGX Spark (GB10). Loaded solo on port 8001, reached /health, passed the arithmetic + tool-calling probes, and benchmarked at ~14.9 tok/s decode. The serving recipe below is the one that actually works on the nvcr.io/nvidia/vllm:26.04-py3 build — two non-obvious flags are required (see "Serving gotchas").

What it is

  • 24B dense instruct model, quantized to NVFP4 (vLLM picks the NvFp4LinearBackend.FLASHINFER_CUTLASS GEMM). The checkpoint is compressed-tensors format (RedHatAI / llm-compressor), so vLLM auto-detects the quantization — it is not the ModelOpt FP4 the nvidia/ + mmangkad/ checkpoints use.
  • Vision-capable (Mistral3ForConditionalGeneration, a Pixtral-style encoder), but the fleet serves it text-only (images limited to 0 — see below). Tool calling works; vision is unused by the fallback.
  • Instruct, not a thinker: no <think> reasoning trace (it reasons inline in content). Served without --reasoning-parser.
  • 128K native context (max_model_len capped to 32K for the first load).
  • Repo is public (no HF_TOKEN). ~16 GiB on disk (4 safetensors shards).

How it runs in the fleet

The fleet ships one generate backend; a warm fallback is opt-in and is not scaffolded by lobes init --fleet. Wiring Mistral in is a manual, two-part change (see docs/gateway-fleet.md → Adding a fallback):

  1. Add a vllm-fallback service to the fleet docker-compose.yml (mirror vllm-primary) that serves Mistral with its own knobs, and add it to the gateway service's depends_on:

    command:
  - vllm
  - serve
  - RedHatAI/Mistral-Small-3.2-24B-Instruct-2506-NVFP4
  - --served-model-name=RedHatAI/Mistral-Small-3.2-24B-Instruct-2506-NVFP4
  - --quantization=compressed-tensors
  - --max-model-len=32768
  - --gpu-memory-utilization=0.35   # also drop the primary's util — the two must sum < 1.0
  - --tool-call-parser=mistral
  - --tokenizer-mode=mistral             # required for Mistral tool calls
  - '--limit-mm-per-prompt={"image":0}'  # disable vision (no params.json) — see gotchas below
  # no --reasoning-parser: instruct model, answers inline in `content`

  2. Point the gateway at it via the only two fallback keys the gateway reads (uncomment them in the fleet .env); the gateway adds the second backend only when these are set:

    FALLBACK_URL=http://vllm-fallback:8000
FALLBACK_SERVED_NAME=RedHatAI/Mistral-Small-3.2-24B-Instruct-2506-NVFP4

Once wired, address it through the gateway by its served name:

curl -s http://localhost:8000/v1/chat/completions \
  -d '{"model":"RedHatAI/Mistral-Small-3.2-24B-Instruct-2506-NVFP4","messages":[{"role":"user","content":"hi"}]}'

Serving gotchas (the recipe that works)

The working vllm serve flags on the nv26.04 build are:

--tokenizer-mode=mistral
--limit-mm-per-prompt={"image":0}
--enable-auto-tool-choice  --tool-call-parser=mistral
--kv-cache-dtype=fp8  --max-model-len=32768  --enable-prefix-caching
# NVFP4 is compressed-tensors → auto-detected (no --quantization needed;
# pass --quantization=compressed-tensors to be explicit). No --reasoning-parser.

Two flags are non-obvious and were found the hard way:

  1. Tool calling needs the mistral tokenizer. Served with the HF tokenizer (--tokenizer-mode auto, the default), the model loads and answers text fine, but a tool_choice:"auto" request returns the raw markup [TOOL_CALLS]get_weather[ARGS]{"city":"Paris"} in content with an empty tool_calls — the mistral parser doesn't recognise the HF chat template's format. The mistral tokenizer (the checkpoint ships tekken.json) makes the parser produce a proper tool_calls array (finish_reason: tool_calls, content: null).
  2. The mistral tokenizer alone crashes the multimodal profiler. With --tokenizer-mode mistral and no image limit, engine init dies in the Pixtral dummy-input path: AssertionError: Expected to decode 1 token, got 3 (the mistral_common tokenizer vs. the Pixtral image token). --limit-mm-per-prompt {"image":0} disables image inputs and sidesteps the crash — acceptable because the fallback is text + tools only.

The full mistral stack (--config-format mistral --load-format mistral) is not an option: this NVFP4 repo ships HF-format config.json + sharded safetensors (no params.json/consolidated.safetensors), so only the tokenizer is loaded in mistral mode.

Running it as a standalone single model. Mistral's supported path is the fleet's opt-in fallback (wire it via FALLBACK_URL/FALLBACK_SERVED_NAME + a vllm-fallback service — see the recipe above). lobes switch RedHatAI/Mistral-Small-3.2-24B-Instruct-2506-NVFP4 sets the catalog quantization (compressed-tensors) and the mistral tool parser, but the single-model compose targets the dense/Qwen primaries and does not add --tokenizer-mode mistral / --limit-mm-per-prompt {"image":0} — add those to the compose command by hand if you serve Mistral standalone, or tool calls will leak as text.

Benchmark — 2026-05-30, DGX Spark (GB10)

Loaded solo on port 8001 (the 27B primary stopped first to free memory), gpu-memory-utilization=0.4.

Property Value
Image / engine nvcr.io/nvidia/vllm:26.04-py3 / vLLM 0.19.0+...nv26.04
Weights on disk ~16 GiB (4 safetensors shards)
Model loading 15.05 GiB, weights loaded in ~83 s
KV cache 30.69 GiB available; max concurrency 12.28x at 32K tokens
GPU memory (EngineCore) ~49,557 MiB (gpu-memory-utilization=0.4)
Health / models /health 200; /v1/models lists the model, max_model_len 32768
Correctness 17 × 23 = 391 ✅; 14:45→17:10 = 145 min computed correctly (see note)
Reasoning trace none (instruct model — answers inline in content)
Tool calling (tool_choice:auto) ✅ proper tool_calls array (finish_reason: tool_calls)
Decode throughput 14.9 / 14.9 tok/s (batch=1, greedy, 512 tokens forced; identical across 2 runs)
Prefill 2,009 prompt tokens + 16 gen in 1.49 s (~1,350 tok/s)

Decode at ~15 tok/s is ~50 % faster than the dense 32B (~9.7 tok/s) — a smaller 24B reads fewer bytes per token — and it actually loads, unlike the 35B MoE. Suitable as the opt-in warm fallback the gateway fails over to once a fallback is configured.

Note on the time-duration probe. lobes assess marks 14:45→17:10 = 145 min as FAIL because its strict "145" in content check didn't match that one greedy run's verbose output. Manual re-runs of the exact prompt return the correct answer (the model derives 145 and boxes it). Treat the FAIL as a checker-strictness / fp8-KV-nondeterminism artifact, not a wrong answer.

Known noise

env_file: .env passes the compose-interpolation vars into the container, so vLLM logs harmless Unknown vLLM environment variable detected: VLLM_* warnings at startup. They do not affect serving.