Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,39 @@ vllm serve unsloth/Qwen3.5-4B-MTP-GGUF:Q4_K_M \
For a GGUF without a `nextn` block, omit `--speculative-config`; the backbone
loads normally without MTP.

## Adding an architecture

`gguf_map` diffs a GGUF's tensor names against the parameter names the target
model expects, and reports what is left over on each side and where shapes
disagree. It reads headers over an HTTP range request and builds the target on
meta tensors, so it needs no GPU and downloads no weights.

```bash
python -m vllm_gguf_plugin.tools.gguf_map \
--gguf allenai/OLMoE-1B-7B-0924-Instruct-GGUF:olmoe-1b-7b-0924-instruct-q4_k_m.gguf \
--target-model allenai/OLMoE-1B-7B-0924-Instruct \
--target vllm --emit-adapter
```

`--target` picks what to reconcile against: `vllm` (the names `load_weights`
accepts), `hf` (a `transformers` state dict), or `safetensors` (the
checkpoint's own headers). Only the last works before either library supports
the architecture, which is the usual case when GGUF support lands first.

Pass `--gguf` more than once for a vision-language model, whose projector ships
as a separate mmproj file:

```bash
python -m vllm_gguf_plugin.tools.gguf_map \
--gguf meta-models/Muse-Glimmer-30B-GGUF:Muse-Glimmer-30B-KQuant-Dynamic-Q4_K_XL.gguf \
meta-models/Muse-Glimmer-30B-GGUF:mmproj-Muse-Glimmer-30B-Q4_K_M.gguf \
--target-model meta-models/Muse-Glimmer-30B --target safetensors
```

`--emit-adapter` prints a `WeightsMapper` skeleton to start from. It is a
starting point, not a result: value-level transforms produce neither a name nor
a shape mismatch, so confirm numerics separately.

## Tested model coverage

The plugin uses vLLM's model implementations and a generic GGUF weight
Expand Down
Loading