Skip to content

update requirement of llama3 example for higher accuracy #2377

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
xin3he wants to merge 3 commits into
base: master
Choose a base branch
from
Open

update requirement of llama3 example for higher accuracy #2377

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a719e2b
update requirement of llama3 example for higher accuracy
xin3he Jan 5, 2026
b7eec49
update readme
xin3he Jan 5, 2026
6649115
change llama3.1 8b mxfp8 recipe
xin3he Jan 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 2 additions & 13 deletions ...p/huggingface_models/language-modeling/quantization/auto_round/llama3/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,18 +13,6 @@ pip install auto-round==0.9.3
pip install -r requirements.txt
```

**Before neural-compressor v3.7 and auto-round v0.9.1 release, please install from source for the latest updates:**

```bash
# neural-compressor-pt
INC_PT_ONLY=1 pip install git+https://github.com/intel/neural-compressor.git@master
# auto-round
pip install git+https://github.com/intel/[email protected]
# other requirements
pip install -r requirements.txt
```


## Quantization

### Demo (`MXFP4`, `MXFP8`, `NVFP4`, `uNVFP4`)
Expand Down Expand Up @@ -92,7 +80,8 @@ Here we provide several recipes for Llama3 models. The relative accuracy loss of

#### Llama 3.1 8B MXFP8

AutoRound tuning helps improve the accuracy, `iters` and `nsamples` is higher than default.
RTN (Round-to-Nearest) is enough to keep accuracy.

```bash
# Quantize and export AutoRound format
CUDA_VISIBLE_DEVICES=0 bash run_quant.sh --topology=Llama-3.1-8B --dtype=mxfp8 --input_model=/models/Meta-Llama-3.1-8B-Instruct --output_model=Llama-3.1-8B-MXFP8
Expand Down
15 changes: 7 additions & 8 deletions .../nlp/huggingface_models/language-modeling/quantization/auto_round/llama3/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
transformers==4.56.2
torch==2.7.0
torchvision==0.22.0
lm_eval==0.4.9.1
datasets==3.6.0
deepspeed==0.17.6
auto-round>=0.8.0
neural-compressor-pt>=3.6
transformers==4.57.3
torch==2.9.0
torchvision==0.24.0
lm_eval==0.4.9.2
datasets==4.4.2
auto-round>=0.9.3
neural-compressor-pt>=3.7
5 changes: 2 additions & 3 deletions ...orch/nlp/huggingface_models/language-modeling/quantization/auto_round/llama3/run_quant.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,14 +50,13 @@ case "$TOPOLOGY" in
case "$DTYPE" in
"mxfp8")
echo "Running Llama 3.1 8B MXFP8 quantization..."
CMD="python quantize.py --model_name_or_path \"$INPUT_MODEL\" $COMMON_ARGS --dtype MXFP8 --iters 1000 --nsamples 512 --export_path \"$OUTPUT_MODEL\""
CMD="python quantize.py --model_name_or_path \"$INPUT_MODEL\" $COMMON_ARGS --dtype MXFP8 --iters 0 --export_path \"$OUTPUT_MODEL\""
echo "Executing command: $CMD"
python quantize.py \
--model_name_or_path "$INPUT_MODEL" \
$COMMON_ARGS \
--dtype MXFP8 \
--iters 1000 \
--nsamples 512 \
--iters 0 \
--export_path "$OUTPUT_MODEL"
;;
"mxfp4")
Expand Down