Skip to content

Commit 4ddb31a

Browse files
wenhuach21hshen14SunMarc
authored
[new blog post] introducing auto-round (#2826)
* update * update * tiny change * tiny change * Fix typos and update wording * Update the #tasks * Update the device section * update authors * add info in yml * add thumbnail * Update autoround.md Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com> * fix some issues * fix splitting line issue * fix some issues * update a little * add overview * update * switch image source * resize image * tiny change * change to intel gpu * change opt-125m to qwen3 --------- Co-authored-by: Shen, Haihao <haihao.shen@intel.com> Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
1 parent b3fda59 commit 4ddb31a

3 files changed

Lines changed: 215 additions & 0 deletions

File tree

‎_blog.yml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5908,3 +5908,15 @@
59085908
tags:
59095909
- agents
59105910
- inference
5911+
5912+
- local: autoround
5913+
title: "Introducing AutoRound: Intel’s Advanced Quantization for LLMs and VLMs"
5914+
author: wenhuach
5915+
thumbnail: /blog/assets/autoround/thumbnail.png
5916+
date: April 29, 2025
5917+
tags:
5918+
- llms
5919+
- inference
5920+
- quantization
5921+
- intel
5922+

‎assets/autoround/thumbnail.png‎

1.43 MB
Loading

‎autoround.md‎

Lines changed: 203 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,203 @@
1+
---
2+
title: "Introducing AutoRound: Intel’s Advanced Quantization for LLMs and VLMs"
3+
thumbnail: /blog/assets/autoround/thumbnail.png
4+
authors:
5+
- user: wenhuach
6+
- user: Haihao
7+
- user: weiweiz1
8+
- user: n1ck-guo
9+
- user: isaacmac
10+
- user: kding1
11+
- user: IlyasMoutawwakil
12+
- user: marcsun13
13+
- user: medmekk
14+
15+
---
16+
17+
As large language models (LLMs) and vision-language models (VLMs) continue to grow in size and complexity, deploying them efficiently becomes increasingly challenging. Quantization offers a solution by reducing model size and inference latency. Intel's [AutoRound](https://github.com/intel/auto-round) emerges as a cutting-edge quantization tool that balances accuracy, efficiency, and compatibility.
18+
19+
# What is AutoRound?
20+
21+
**AutoRound** is a weight-only post-training quantization (PTQ) method developed by Intel. It uses signed gradient descent to jointly optimize weight rounding and clipping ranges, enabling accurate low-bit quantization (e.g., INT2 - INT8) with minimal accuracy loss in most scenarios. For example, at INT2, it outperforms popular baselines by up to **2.1x higher in relative accuracy**. The image below provides an overview of the core algorithm in AutoRound. For more details, please refer to [our paper](https://arxiv.org/abs/2309.05516).
22+
23+
<p align="center">
24+
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/auto-round/autoround_overview.png" alt="algorithm overview<" width="100%"/>
25+
<br>
26+
<i></i>
27+
</p>
28+
29+
Despite its strong performance, AutoRound is fast and lightweight — quantizing a 72B model takes just **37 minutes on an A100 GPU** under light mode. It also supports mixed-bit tuning, lm-head quantization, GPTQ/AWQ/GGUF format exporting, and flexible tuning recipes.
30+
31+
# Key Advantages
32+
33+
## Superior Accuracy at Low Bit Widths
34+
35+
**AutoRound** delivers highly promising results, particularly in low-bit quantization scenarios. Evaluations across a variety of tasks show that it outperforms popular methods by a wide margin at 2-bit precision [(source)](https://arxiv.org/abs/2309.05516). At 4 bits, AutoRound continues to hold a competitive edge in most cases, as demonstrated on the [Low-Bit Open LLM Leaderboard](https://huggingface.co/spaces/Intel/low_bit_open_llm_leaderboard).
36+
37+
<p align="center">
38+
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/auto-round/int2.png" alt="Average of 10+ tasks at W2g128<" width="60%"/>
39+
<br>
40+
<i>Average of 10+ tasks at W2g128</i>
41+
</p>
42+
43+
44+
<p align="center">
45+
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/auto-round/int4.png" alt="Average of 10+ tasks at W4<" width="60%"/>
46+
<br>
47+
<i>Average of 10+ tasks at W4</i>
48+
</p>
49+
50+
## 2. Broad Compatibility
51+
52+
### Models
53+
54+
**LLMs:** AutoRound supports nearly all popular LLM architectures, including well-known models like Qwen, LLaMA, and DeepSeek. Ready-to-use quantized models are available on Hugging Face through collections such as [OPEA](https://huggingface.co/OPEA), [Kaitchup](https://huggingface.co/kaitchup), and [fbaldassarri](https://huggingface.co/fbaldassarri).
55+
56+
**VLMs:** AutoRound supports over 10 vision-language models (VLMs), including Mistral-Small-3.1, Gemma3, and more. You can find the full list in the [README](https://github.com/intel/auto-round?tab=readme-ov-file#vlm-support-matrix), and ready-to-use quantized models are available in the [OPEA Hugging Face collection](https://huggingface.co/collections/OPEA/vlms-autoround-675bc712fdd6a55ebaf11bfa). For models not yet supported, you can still apply our RTN method with `--iters 0`. No tuning is required, but some accuracy loss is expected.
57+
58+
### Devices
59+
60+
- **CPU**
61+
- **Intel GPU**
62+
- **CUDA**
63+
64+
### Quantization Configurations
65+
66+
- **Int8 Weight Only**
67+
- **Int4 Weight Only**
68+
- **Int3 Weight Only**
69+
- **Int2 Weight Only**
70+
- **Mixed bits Weight only**
71+
72+
### Export Formats
73+
74+
- **AutoRound**
75+
- **GPTQ**
76+
- **AWQ**
77+
- **Some GGUFs**
78+
79+
### 3. Flexible/Efficient Quantization
80+
81+
AutoRound requires only 200 tuning steps and a small calibration dataset (as few as 128 samples) to achieve high accuracy. This efficiency translates to faster quantization times and reduced resource consumption compared to other int2 methods, which are more computationally intensive.
82+
83+
| | AutoAWQ<br/> samples=128<br/>seqlen=512<br/>dataset='pile' | AutoAWQ<br/> samples=512<br/>seqlen=2048<br/>dataset='pile' | GPTQ in Transfomers<br/> samples=?<br/>seqlen=?<br/>dataset='c4' | AutoRoundLight<br/> samples=128<br/>seqlen=2048<br/>dataset='pile-10k' | AutoRound<br/> samples=128<br/>seqlen=2048<br/>dataset='pile-10k' | AutoRound<br/> samples=512<br/>seqlen=2048<br/>dataset='pile-10k |
84+
|-------------|------------------------------------------------------------|-------------------------------------------------------------|------------------------------------------------------------------|------------------------------------------------------------------------|-------------------------------------------------------------------|------------------------------------------------------------------|
85+
| Qwen2.5 3B | 7min | 17min | 13min | 3min | 8min | 9min |
86+
| Llama3.1-8B | 13min | 27min | 22min | 6min | 13min | 17min |
87+
| Qwen2.5 72B | 105min | 230min | OOM | 37min | 120min | 149min |
88+
89+
# Get Started with AutoRound
90+
91+
## Installation
92+
93+
```bash
94+
pip install auto-round
95+
```
96+
97+
## Quantization and Serialization
98+
99+
Currently, only offline mode is supported to generate quantized models.
100+
101+
### Command Line Usage
102+
103+
```bash
104+
auto-round \
105+
--model Qwen/Qwen3-0.6B \
106+
--bits 4 \
107+
--group_size 128 \
108+
--format "auto_round,auto_awq,auto_gptq" \
109+
--output_dir ./tmp_autoround
110+
```
111+
112+
AutoRound also offers another two recipes, `auto-round-best` and `auto-round-light`, designed for optimal accuracy and improved speed, respectively.
113+
114+
```bash
115+
auto-round-best \
116+
--model Qwen/Qwen3-0.6B \
117+
--output_dir ./tmp_autoround
118+
```
119+
120+
For 2 bits, we recommend using `auto-round-best` or `auto-round`. For a comparison of the three recipes, please refer to the table below.
121+
122+
W4G128 Average Accuracy of 13 tasks (mmlu-pro, if_eval, gsm8k, etc) and Time Cost Results (Testing was conducted on the Nvidia A100 80G using the version of PyTorch 2.6.0 with enable_torch_compile):
123+
124+
| Model | Qwen2.5-0.5B-Instruct | Falcon3-3B | Qwen2.5-7B-Instruct | Meta-Llama-3.1-8B-Instruct | Falcon3-10B | Qwen2.5-72B-Instruct |
125+
|---------|-----------------------|-----------------|---------------------|----------------------------|-----------------|----------------------|
126+
| 16bits | 0.4192 | 0.5203 | 0.6470 | 0.6212 | 0.6151 | 0.7229 |
127+
| Best | **0.4137**(7m) | **0.5142**(23m) | 0.6426(58m) | **0.6116**(65m) | **0.6092**(81m) | 0.7242(575m) |
128+
| Default | 0.4129(2m) | 0.5133(6m) | 0.6441(13m) | 0.6106(13m) | 0.6080(18m) | **0.7252**(118m) |
129+
| Light | 0.4052(2m) | 0.5108(3m) | **0.6453**(5m) | 0.6104(6m) | 0.6063(6m) | 0.7243(37m) |
130+
131+
### AutoRound API Usage
132+
133+
This setting offers a better trade-off between accuracy and tuning cost, and is recommended in all scenarios.
134+
135+
```python
136+
from transformers import AutoModelForCausalLM, AutoTokenizer
137+
from auto_round import AutoRound
138+
139+
model_name = "Qwen/Qwen3-0.6B"
140+
model = AutoModelForCausalLM.from_pretrained(model_name)
141+
tokenizer = AutoTokenizer.from_pretrained(model_name)
142+
bits, group_size, sym = 4, 128, True
143+
autoround = AutoRound(
144+
model,
145+
tokenizer,
146+
bits=bits,
147+
group_size=group_size,
148+
sym=sym,
149+
# enable_torch_compile=True,
150+
)
151+
152+
output_dir = "./tmp_autoround"
153+
autoround.quantize_and_save(output_dir, format='auto_round,auto_awq,auto_gptq')
154+
155+
```
156+
157+
For the best/light settings of AutoRound for API usage or mixed-bit configurations, please refer to [AutoRound README](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#api-usage)
158+
159+
160+
## Inference
161+
162+
AutoRound automatically selects the best available backend based on the installed libraries and prompts the user to install additional libraries when a better backend is found. For more details, please refer to [HF README](https://github.com/huggingface/transformers/blob/main/docs/source/en/quantization/auto_round.md#inference) or [AutoRound README](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#4-inference).
163+
### CPU/Intel GPU/CUDA
164+
165+
```python
166+
from transformers import AutoModelForCausalLM, AutoTokenizer
167+
168+
model_name = "OPEA/Qwen2.5-1.5B-Instruct-int4-sym-inc"
169+
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto", torch_dtype="auto")
170+
tokenizer = AutoTokenizer.from_pretrained(model_name)
171+
text = "There is a girl who likes adventure,"
172+
inputs = tokenizer(text, return_tensors="pt").to(model.device)
173+
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50, do_sample=False)[0]))
174+
```
175+
176+
### Convert GPTQ/AWQ to AutoRound
177+
178+
Most GPTQ/AWQ models can be converted to the AutoRound format for better compatibility and support with Intel devices. Please note that the quantization config will be changed if the model is serialized.
179+
180+
```python
181+
from transformers import AutoModelForCausalLM, AutoTokenizer, AutoRoundConfig
182+
183+
model_name = "ybelkada/opt-125m-gptq-4bit"
184+
quantization_config = AutoRoundConfig()
185+
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="cpu", torch_dtype="auto",
186+
quantization_config=quantization_config)
187+
tokenizer = AutoTokenizer.from_pretrained(model_name)
188+
text = "There is a girl who likes adventure,"
189+
inputs = tokenizer(text, return_tensors="pt").to(model.device)
190+
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50, do_sample=False)[0]))
191+
```
192+
193+
# Conclusion
194+
195+
AutoRound offers a meaningful improvement step forward in post-training quantization for large language and vision-language models. By combining high accuracy, exceptional efficiency, and broad compatibility with popular models, devices, and export formats, AutoRound makes low-bit quantization both practical and powerful. Whether you're deploying LLMs at scale or experimenting with edge inference on VLMs, AutoRound provides the tools and flexibility you need to achieve optimal performance with minimal overhead. We invite you to try it out and join the growing community pushing the boundaries of efficient AI deployment.
196+
197+
Contributions to [AutoRound](https://github.com/intel/auto-round/pulls) are welcome and greatly appreciated! Whether it's fixing bugs, improving documentation, adding new features, or suggesting improvements, your help is always valued.
198+
199+
If you encounter any issues with auto-round, please open an issue on the [AutoRound](https://github.com/intel/auto-round/issues) repository.
200+
201+
# Acknowledgement
202+
203+
We would like to thank the open-source low-precision libraries including AutoGPTQ, AutoAWQ, GPTQModel, Triton, Marlin and ExLLaMAV2, whose CUDA kernels are used in AutoRound.

0 commit comments

Comments
 (0)