English | 简体中文

📖 Documentation   |   🤗 Hugging Face   |   🤖 ModelScope   |   💬 WeChat |   🫨 Discord

• [25/11/03] We have released v0.2. Quantization support for new models, such as GLM-4.6 and Qwen3-VL, open-sources the Eagle3 speculative decoding training framework, and updates the Diffusion model quantization tools.
• [25/09/30] We have released SpecExit, the reasoning early-exit algorithm: [Paper] | [Docs] | [vLLM Code]🔥🔥🔥
• [25/09/26] We have released TEQUILA, the ternary quantization algorithm [Paper] | [Code]🔥🔥🔥
• [25/09/24] We now support the PTQ quantification of NVFP4 for the Qwen3 series models. We also opensource Qwen3-32B-NVFP4 and Qwen3-235B-A22B-NVFP4 weights.

• Highly Integrated: This toolkit integrates mainstream compression algorithms into a unified framework, offering developers one-click access with exceptional ease of use.
• Continuous Innovation: Beyond integrating widely-used industry algorithms, we are continuously researching better compression algorithms, which will be gradually open-sourced in the future.
• Performance-Driven: We continuously optimize end-to-end performance in model compression workflows and algorithm deployment, such as enabling quantization of models like Qwen3-235B and DeepSeek-R1 on a single GPU.

We recommend using pip to install the latest stable version of AngelSlim:

pip install angelslim

Alternatively, you can clone the repository and install from source in editable mode:

cd AngelSlim && python setup.py install

For more detailed installation instructions, please refer to the Installation Documentation.

• Quantization

  After installing AngelSlim, you can launch static FP8 quantization for the Qwen3-1.7B model with the following one-command script:

  python3 tools/run.py -c configs/qwen3/fp8_static/qwen3-1_7b_fp8_static.yaml

  This example produces quantized model weights by performing PTQ calibration on a model loaded from HuggingFace.

  Code-based Start

  To perform dynamic FP8 quantization on Qwen3-1.7B:

  from angelslim.engine import Engine
  
  slim_engine = Engine()
  # Prepare model
  slim_engine.prepare_model(model_name="Qwen", model_path="Qwen/Qwen3-1.7B",)
  # Initialize compressor
  slim_engine.prepare_compressor("PTQ", default_method="fp8_dynamic")
  # Compress model
  slim_engine.run()
  # Save compressed model
  slim_engine.save("./output")

  For more details, please refer to the Quick Start Documentation.

• Speculative Decoding

  After installing AngelSlim, you can quickly start Eagle3 training with the following scripts:

  # Start the vLLM server
  bash scripts/speculative/run_vllm_server.sh
  # Generate training data
  bash scripts/speculative/generate_data_for_target_model.sh
  # Perform online training for the Eagle3 model
  bash scripts/speculative/train_eagle3_online.sh

  For detailed training configurations and PyTorch performance benchmarks of Eagle3, please refer to the Quick Start Guide for Speculative Sampling.

• Diffusion Model Quantization

  Use the scripts/diffusion/run_diffusion.py for quantization and inference:

  # Online quantization and inference
  python scripts/diffusion/run_diffusion.py \
    --model-name-or-path black-forest-labs/FLUX.1-schnell \
    --quant-type fp8-per-tensor \
    --prompt "A cat holding a sign that says hello world" \
    --height 1024 --width 1024 --steps 4 --guidance 0.0 --seed 0

  For more quantization inference methods, please refer to the Diffusion Model Quantization Documentation.

To test offline inference with a quantized model loaded via transformers, run the following command:

python scripts/deploy/offline.py $MODEL_PATH "Hello, my name is"

Where MODEL_PATH is the path to the quantized model output. please set the deploy_backend: huggingface in the global configuration before quantizing the model, or manually modify the ignored_layers field in the config.json file located in the quantized model output directory to ignore.

After specifying the quantized model path MODEL_PATH, you can deploy an OpenAI-compatible API service using the following LLMs inference frameworks:

• vLLM

  Use the following script to launch a vLLM server, recommended version vllm>=0.8.5.post1. For MOE INT8 quantized models, vllm>=0.9.0 is required.

  bash scripts/deploy/run_vllm.sh --model-path $MODEL_PATH --port 8080 -d 0,1,2,3 -t 4 -p 1 -g 0.8 --max-model-len 4096

  Where -d is the visible devices, -t is tensor parallel size, -p is pipeline parallel size, and -g is the GPU memory utilization.

• SGLang

  Use the following script to launch a SGLang server, recommended version sglang>=0.4.6.post1.

  bash scripts/deploy/run_sglang.sh --model-path $MODEL_PATH --port 8080 -d 0,1,2,3 -t 4 -g 0.8

Invoke requests via OpenAI's API format:

bash scripts/deploy/openai.sh -m $MODEL_PATH -p "Hello, my name is" --port 8080 --max-tokens 4096 --temperature 0.7 --top-p 0.8 --top-k 20 --repetition-penalty 1.05 --system-prompt "You are a helpful assistant."

where -p is the input prompt.

Evaluate the performance of quantized model using lm-evaluation-harness, recommended versionlm-eval>=0.4.8

bash scripts/deploy/lm_eval.sh -d 0,1 -t 2 -g 0.8 -r $RESULT_PATH -b "auto" --tasks ceval-valid,mmlu,gsm8k,humaneval -n 0 $MODEL_PATH

The performance test results for selected models are shown below. For the complete benchmark, refer to the Benchmark documentation

Benchmark results for the Hunyuan-Instruct model with FP8, INT4-AWQ and INT4-GPTQ quantization algorithms on datasets includingOlympiadBench, AIME 2024 and DROP:

Benchmark results for Qwen3 series models with FP8-Static, FP8-Dynamic, INT4-GPTQ, and INT4-AWQ quantization algorithms on datasets including CEVAL, MMLU, GSM8K, and HUMANEVAL:

Benchmark results for DeepSeek-R1-0528 series models with FP8-Block-Wise and W4A8-FP8 quantization algorithms on datasets including GPQA Diamond、AIME 2024、SimpleQA and LiveCodeBench：

Qwen3-VL Benchmark

Benchmark results for Qwen3VL series models with BF16、FP8-Static and FP8-Dynamic quantization algorithms on datasets including MMMU_VAL、DocVQA_VAL and ChartQA_TEST：

Other models such as GLM-4.6, Qwen2.5, and Seed-OSS have been evaluated on benchmarks like CEVAL, MMLU, and GSM8K using quantization strategies including FP8-Static, FP8-Dynamic, INT4-GPTQ, and INT4-AWQ.

Benchmark results for Qwen3 series models with Eagle3 speculative decoding algorithm on datasets including MT-bench, HunmanEval, GSM8K, and Alpaca:

Benchmark results for Hunyuan series models with Eagle3 speculative decoding algorithm on datasets including MT-bench, HunmanEval, GSM8K, and Alpaca:

The code for this project is open-sourced under the License for AngelSlim.

@software{AngelSlim2025,
    title={{AngelSlim}},
    author={Tencent AngelSlim Project Contributors},
    year={2025},
    month={6},
    url={https://github.com/Tencent/AngelSlim},
}

• AngelSlim is continuously iterating and new features will be released soon. If you have any questions or suggestions, please open an issue on GitHub Issues or join our WeChat discussion group.