Merge branch 'main' into awq-mse-observer-alignment

Author: dzhengAP

.github/mergify.yml

@@ -1,43 +1,4 @@
-queue_rules:
-  - name: default
-    merge_method: merge
-    commit_message_template: |
-      {{ title }} (#{{ number }})
-
-      {{ body }}
-
-      Signed-off-by: Mergify <noreply@mergify.com>
-    queue_conditions:
-      - check-success=DCO
-      - check-success=quality-check
-      - check-success=transformers-tests
-      - check-success=base-tests (3.10)
-      - check-success=base-tests (3.13)
-      - check-success=pytorch-tests (3.10)
-      - check-success=pytorch-tests (3.13)
-      - check-success=markdown-link-check
-
 pull_request_rules:
-  - name: Automatically merge when ready
-    conditions:
-      - base=main
-      - label=ready
-      - "#approved-reviews-by>=2"
-      - check-success=DCO
-      - check-success=quality-check
-      - check-success=transformers-tests
-      - check-success=base-tests (3.10)
-      - check-success=base-tests (3.13)
-      - check-success=pytorch-tests (3.10)
-      - check-success=pytorch-tests (3.13)
-      - check-success=markdown-link-check
-      - check-success=ready-label-check
-      - -conflict
-      - -draft
-    actions:
-      queue:
-        name: default
-
   - name: label-documentation
     description: Automatically apply documentation label
     conditions:
@@ -47,6 +8,11 @@ pull_request_rules:
           - files~=^[^/]+\.md$
           - files~=^docs/
           - files~=^examples/
+      - -files~=^src/
+      - -files~=^tests/
+      - -files~=^\.github/
+      - -files~=^Makefile$
+      - -files~=^pyproject\.toml$
     actions:
       label:
         add:

README.md

@@ -37,24 +37,24 @@ Big updates have landed in LLM Compressor! To get a more in-depth look, check ou
 
 Some of the exciting new features include:
 
+* **Qwen3.5 Support**: Qwen 3.5 can now be quantized using LLM Compressor. You will need to update your local transformers version using `uv pip install --upgrade transformers` and install LLM Compressor from source if using `<0.11`. Once updated, you should be able to run examples for the [MoE](examples/quantization_w4a4_fp4/qwen3_5_example.py) and [non-MoE](examples/quantization_w4a4_fp4/qwen3_5_example.py) variants of Qwen 3.5 end-to-end. For models quantized and published by the RedHat team, consider using the [NVFP4](https://huggingface.co/RedHatAI/Qwen3.5-122B-A10B-NVFP4) and FP8 checkpoints for [Qwen3.5-122B](https://huggingface.co/RedHatAI/Qwen3.5-122B-A10B-FP8-dynamic) and [Qwen3.5-397B](https://huggingface.co/RedHatAI/Qwen3.5-397B-A17B-FP8-dynamic).
 * **Updated offloading and model loading support**: Loading transformers models that are offloaded to disk and/or offloaded across distributed process ranks is now supported. Disk offloading allows users to load and compress very large models which normally would not fit in CPU memory. Offloading functionality is no longer supported through accelerate but through model loading utilities added to compressed-tensors. For a full summary of updated loading and offloading functionality, for both single-process and distributed flows, see the [Big Models and Distributed Support guide](docs/guides/big_models_and_distributed/model_loading.md).
 * **Distributed GPTQ Support**: GPTQ now supports Distributed Data Parallel (DDP) functionality to significantly improve calibration runtime. An example using DDP with GPTQ can be found [here](examples/quantization_w4a16/llama3_ddp_example.py).
 * **Updated FP4 Microscale Support**: GPTQ now supports FP4 quantization schemes, including both [MXFP4](examples/quantization_w4a16_fp4/mxfp4/llama3_example.py) and [NVFP4](examples/quantization_w4a4_fp4/llama3_gptq_example.py). MXFP4 support has also been improved with updated weight scale generation. Models with weight-only quantization in the MXFP4 format can now run in vLLM as of vLLM v0.14.0. MXFP4 models with activation quantization are not yet supported in vLLM for compressed-tensors models
 * **New Model-Free PTQ Pathway**: A new model-free PTQ pathway has been added to LLM Compressor, called [`model_free_ptq`](src/llmcompressor/entrypoints/model_free/__init__.py#L36). This pathway allows you to quantize your model without the requirement of Hugging Face model definition and is especially useful in cases where `oneshot` may fail. This pathway is currently supported for data-free pathways only i.e FP8 quantization and was leveraged to quantize the [Mistral Large 3 model](https://huggingface.co/mistralai/Mistral-Large-3-675B-Instruct-2512). Additional [examples](examples/model_free_ptq) have been added illustrating how LLM Compressor can be used for Kimi K2
+* **MXFP8 Microscale Support (Experimental)**: LLM Compressor now supports MXFP8 quantization via PTQ. Both W8A8 ([MXFP8](experimental/mxfp8/qwen3_example_w8a8_mxfp8.py)) and W8A16 weight-only ([MXFP8A16](experimental/mxfp8/qwen3_example_w8a16_mxfp8.py)) modes are available.
 * **Extended KV Cache and Attention Quantization Support**: LLM Compressor now supports attention quantization. KV Cache quantization, which previously only supported per-tensor scales, has been extended to support any quantization scheme including a new `per-head` quantization scheme. Support for these checkpoints is on-going in vLLM and scripts to get started have been added to the [experimental folder](experimental/attention)
 
 
 ### Supported Formats
-* Activation Quantization: W8A8 (int8 and fp8)
-* Mixed Precision: W4A16, W8A16, NVFP4 (W4A4 and W4A16 support)
-* 2:4 Semi-structured and Unstructured Sparsity
+* Activation Quantization: W8A8 (int8 and fp8), MXFP8 (experimental)
+* Mixed Precision: W4A16, W8A16, MXFP8A16 (experimental), NVFP4 (W4A4 and W4A16 support)
 
 ### Supported Algorithms
 * Simple PTQ
 * GPTQ
 * AWQ
 * SmoothQuant
-* SparseGPT
 * AutoRound
 
 ### When to Use Which Optimization
@@ -75,6 +75,8 @@ pip install llmcompressor
 Applying quantization with `llmcompressor`:
 * [Activation quantization to `int8`](examples/quantization_w8a8_int8/README.md)
 * [Activation quantization to `fp8`](examples/quantization_w8a8_fp8/README.md)
+* [Activation quantization to MXFP8 (experimental)](experimental/mxfp8/qwen3_example_w8a8_mxfp8.py)
+* [Weight-only quantization to MXFP8A16 (experimental)](experimental/mxfp8/qwen3_example_w8a16_mxfp8.py)
 * [Activation quantization to `fp4`](examples/quantization_w4a4_fp4/llama3_example.py)
 * [Activation quantization to `fp4` using AutoRound](examples/autoround/quantization_w4a4_fp4/README.md)
 * [Activation quantization to `fp8` and weight quantization to `int4`](examples/quantization_w4a8_fp8/)
@@ -183,3 +185,7 @@ If you find LLM Compressor useful in your research or projects, please consider
     url={https://github.com/vllm-project/llm-compressor},
 }
 ```
+
+
+!!! warning
+    Sparse compression (24 sparsity) is no longer supported by LLM Compressor due lack of hardware support and usage

docs/.nav.yml

@@ -19,20 +19,28 @@ nav:
     - Qwen3:
       - key-models/qwen3/index.md
       - FP8 Example: key-models/qwen3/fp8-example.md
+    - Qwen3.5:
+      - key-models/qwen3.5/index.md
+      - NVFP4A16 VL Example: key-models/qwen3.5/nvfp4-vl-example.md
+      - NVFP4 MoE Example: key-models/qwen3.5/nvfp4-moe-example.md
     - Kimi-K2:
       - key-models/kimi-k2/index.md
       - FP8 Example: key-models/kimi-k2/fp8-example.md
     - Mistral Large 3:
       - key-models/mistral-large-3/index.md
       - FP8 Example: key-models/mistral-large-3/fp8-example.md
-  - Guides:
+  - User Guides:
+    - Entrypoints:
+      - guides/entrypoints/index.md
+      - oneshot: guides/entrypoints/oneshot.md
+      - model-free-ptq: guides/entrypoints/model-free-ptq.md
+    - Compression Schemes: guides/compression_schemes.md
+    - Observers: guides/observers.md
     - Big Models and Distributed Support:
       - Model Loading: guides/big_models_and_distributed/model_loading.md
       - Sequential Onloading: guides/big_models_and_distributed/sequential_onloading.md
       - Distributed Oneshot: guides/big_models_and_distributed/distributed_oneshot.md
-    - Compression Schemes: guides/compression_schemes.md
-    - Saving a Model: guides/saving_a_model.md
-    - Observers: guides/observers.md
+    - Saving a Compressed Model: guides/saving_a_model.md
     - Memory Requirements: guides/memory.md
     - Runtime Performance: guides/runtime.md
   - Examples:

docs/guides/compression_schemes.md

@@ -8,8 +8,6 @@ A full list of supported schemes can be found [here](https://github.com/vllm-pro
 - [W8A8-INT8](#int8_w8a8)
 - [W4A16 and W8A16](#w4a16-and-w8a16)
 - [NVFP4](#nvfp4)
-- [2:4 Semi-structured Sparsity](#semi-structured)
-- [Unstructured Sparsity](#unstructured)
 
 ## PTQ Compression Schemes
 
@@ -63,27 +61,5 @@ A full list of supported schemes can be found [here](https://github.com/vllm-pro
 | Calibration   | Requires a calibration dataset to calibrate activation global scales                                                            |
 | Use case      | Supported on all NVIDIA Blackwell GPUs or later  
 
-## Sparsification Compression Schemes
-
-Sparsification reduces model complexity by pruning selected weight values to zero while retaining essential weights in a subset of parameters. Supported formats include:
-
-
-### Semi-Structured
-| Feature       | Description                                                                                  |
-|---------------|----------------------------------------------------------------------------------------------|
-| 2:4 Semi-structured Sparsity | Uses semi-structured sparsity (SparseGPT), where 2 of every 4 contiguous weights are set to zero. |
-| Weights       | 2:4 sparsity                                                                                |
-| Activations   | N/A                                                                                          |
-| Calibration   | Requires a calibration dataset                                                              |
-| Use case      | Fine-grained sparsity for compression and speedups           |
-
-
-
-### Unstructured
-| Feature       | Description                                                                                  |
-|---------------|----------------------------------------------------------------------------------------------|
-| Unstructured Sparsity | Zeros out individual weights without a regular pattern, removing weights wherever they contribute least. Produces a fine-grained sparse matrix. |
-| Weights       | Sparsified individually (no structure)                                                     |
-| Activations   | N/A                                                                  |
-| Calibration   | Does not require a calibration dataset                                                    |
-| Use case      | Fine-grained sparsity for compression and speedups                                         |
+!!! warning
+    Sparse compression (including 2of4 sparsity) is no longer supported by LLM Compressor due lack of hardware support and user interest. Please see https://github.com/vllm-project/vllm/pull/36799 for more information.

docs/guides/entrypoints/index.md

@@ -0,0 +1,26 @@
+# Entrypoints
+
+LLM Compressor provides two entrypoints for post-training quantization (PTQ), each suited to different scenarios.
+
+## Choosing an Entrypoint
+
+| | [`oneshot`](oneshot.md) | [`model_free_ptq`](model-free-ptq.md) |
+|---|---|---|
+| **Can apply calibration data** | Yes | No — data-free only |
+| **Requires HF model definition** | Yes | No |
+| **Supports GPTQ / AWQ / SmoothQuant** | Yes | No |
+| **Supports FP8 / NVFP4 data-free** | Yes | Yes |
+| **Works when model has no transformers definition** | No | Yes |
+| **Fallback when `oneshot` fails** | — | Yes |
+
+## oneshot
+
+Use `oneshot` when your quantization algorithm or scheme **requires calibration data**, such as GPTQ, AWQ, SmoothQuant, or static activation quantization (FP8 or INT8 with static per tensor activations). It loads the model through Hugging Face `transformers`, runs calibration forward passes, and applies recipe-defined modifiers.
+
+[:octicons-arrow-right-24: oneshot documentation](oneshot.md)
+
+## model_free_ptq
+
+Use `model_free_ptq` when your quantization scheme is **data-free** (e.g. FP8 dynamic, FP8 block, NVFP4A16) and either the model has no Hugging Face model definition, or `oneshot` fails for your model. It works directly on the safetensors checkpoint without loading the model through `transformers`.
+
+[:octicons-arrow-right-24: model_free_ptq documentation](model-free-ptq.md)

docs/guides/entrypoints/model-free-ptq.md

@@ -0,0 +1,139 @@
+# model_free_ptq
+
+`model_free_ptq` is a PTQ entrypoint for **data-free quantization schemes** that operates directly on safetensors checkpoint files without requiring a Hugging Face model definition or loading the model through `transformers`.
+
+## When to Use
+
+Use `model_free_ptq` when:
+
+- Your quantization scheme is **data-free** (e.g. FP8 dynamic, FP8 block, NVFP4A16, MXFP4/MXFP8)
+- The model **does not have a Hugging Face transformers definition** (e.g. a newly released model not yet in transformers)
+- `oneshot` **fails** for your model
+
+For schemes that require calibration data (GPTQ, AWQ, SmoothQuant, static activation quantization), use [`oneshot`](oneshot.md) instead.
+
+## Basic Usage
+
+```python
+from llmcompressor import model_free_ptq
+
+model_free_ptq(
+    model_stub="meta-llama/Meta-Llama-3-8B-Instruct",
+    save_directory="Meta-Llama-3-8B-Instruct-FP8-BLOCK",
+    scheme="FP8_BLOCK",
+    ignore=["lm_head"],
+    device="cuda:0",
+)
+```
+
+## How It Works
+
+`model_free_ptq` processes each `.safetensors` file in the checkpoint independently, without ever loading the full model into memory as a `torch.nn.Module`. For each file:
+
+1. **Validate** — check that all quantizable tensors can be quantized with the given scheme
+2. **Initialize** — create a minimal `torch.nn.Linear` module for each weight tensor
+3. **Calibrate** — compute scale and zero point directly from the weight tensor (data-free)
+4. **Compress** — call `compress_module` from `compressed-tensors` to pack/quantize the weights
+5. **Save** — write the compressed tensors back to disk
+
+After all files are processed, the safetensors index and model config are updated with the quantization metadata.
+
+Multiple files can be processed in parallel using the `max_workers` argument.
+
+## Arguments
+
+| Argument | Type | Default | Description |
+|----------|------|---------|-------------|
+| `model_stub` | `str \| PathLike` | — | HuggingFace model ID or path to a local directory containing safetensors files |
+| `save_directory` | `str \| PathLike` | — | Directory to save the quantized checkpoint |
+| `scheme` | `QuantizationScheme \| str` | — | Quantization scheme to apply. Can be a preset string (e.g. `"FP8_BLOCK"`, `"NVFP4A16"`) or a `QuantizationScheme` object |
+| `ignore` | `Iterable[str]` | `()` | Module names or regex patterns to skip. Modules ending in `"norm"` are always ignored automatically |
+| `max_workers` | `int` | `1` | Number of parallel worker threads for processing safetensors files |
+| `device` | `str \| torch.device \| None` | `None` | Device to use for quantization. Defaults to GPU if available, otherwise CPU |
+| `converter` | `Converter \| None` | `None` | Optional `compressed-tensors` converter to apply before quantization, e.g. to convert modelopt-format checkpoints to compressed-tensors format |
+
+## Standard Flow (Non-Microscale Schemes)
+
+For schemes without a global scale (e.g. `FP8_BLOCK`, `FP8_DYNAMIC`), call `model_free_ptq` directly:
+
+```python
+from llmcompressor import model_free_ptq
+
+model_free_ptq(
+    model_stub="unsloth/Kimi-K2-Thinking-BF16",
+    save_directory="Kimi-K2-Thinking-FP8-BLOCK",
+    scheme="FP8_BLOCK",
+    ignore=[
+        "re:.*gate$",
+        "lm_head",
+        "re:.*kv_a_proj_with_mqa$",
+        "re:.*q_a_proj$",
+        "model.embed_tokens",
+    ],
+    max_workers=15,
+    device="cuda:0",
+)
+```
+
+## Microscale Flow (NVFP4)
+
+NVFP4 requires a **global scale** that is fused across related weight groups (e.g. qkv projections, gate/up projections). For this fusion to work correctly, the weights of each fused group must reside in the **same safetensors shard**.
+
+Standard model checkpoints often split these weights across different shards. To fix this, run the `reindex_fused_weights` CLI tool first to reorganize the checkpoint:
+
+```bash
+llmcompressor.reindex_fused_weights \
+    unsloth/Kimi-K2-Thinking-BF16 \
+    Kimi-K2-Thinking-BF16-reindexed \
+    --num_workers=10
+```
+
+Then run `model_free_ptq` on the reindexed checkpoint:
+
+```python
+from llmcompressor import model_free_ptq
+
+model_free_ptq(
+    model_stub="Kimi-K2-Thinking-BF16-reindexed",
+    save_directory="Kimi-K2-Thinking-NVFP4A16",
+    scheme="NVFP4A16",
+    ignore=[
+        "re:.*gate$",
+        "lm_head",
+        "re:.*kv_a_proj_with_mqa$",
+        "re:.*q_a_proj$",
+        "model.embed_tokens",
+    ],
+    max_workers=15,
+    device="cuda:0",
+)
+```
+
+!!! note
+    Reindexing is only required for **NVFP4**, which uses a global scale. MXFP4 does not use a global scale and does not require reindexing.
+
+## Ignoring Layers
+
+The `ignore` argument accepts module name strings or regex patterns prefixed with `re:`. Modules whose names end in `"norm"` are automatically ignored regardless of the `ignore` list.
+
+```python
+ignore=[
+    "lm_head",            # exact name match
+    "re:.*gate$",         # regex: any module ending in "gate"
+    "model.embed_tokens", # exact name match
+]
+```
+
+## Supported Schemes
+
+`model_free_ptq` supports any data-free weight quantization scheme. Common presets:
+
+| Scheme | Description |
+|--------|-------------|
+| `FP8_DYNAMIC` | FP8 weights with dynamic per-token activation quantization |
+| `FP8_BLOCK` | FP8 weights with block-wise scaling (Blackwell-optimized) |
+| `NVFP4A16` | NVFP4 weight-only quantization with FP8 group scales and a global scale |
+| `MXFP4/MXFP8` | MXFP4 or MXFP8 quantization with MX-format microscales |
+
+Note: Many of these schemes, such as NVFP4 and MXFP4 may potentially lead to improved recovery when applied with a calibration algorithm that requires data, such as GPTQ. Consider comparing performance using oneshot.
+For the full list of supported schemes and formats, see [Compression Schemes](../compression_schemes.md).