OA-LAMA: An Outlier-Adaptive LLM Inference Accelerator with Memory-Aligned Mixed-Precision Group Quantization

This codebase utilizes lm_eval to evaluate perplexity and zero-shot accuracy. Code segments from SmoothQuant, GPTQ, and Atom are integrated to reproduce results.

OA-LAMA

The current release features:

• Simulated quantization for accuracy evaluation.
• Perplexity and zero-shot accuracy evaluation

Installation

conda create -n OA-LAMA python=3.10
conda activate OA-LAMA
pip install -r requirements.txt

Usage

Accuracy Evaluation

We provide several scripts to reproduce our results in the paper: To run our W4A4 perplexity evaluation, please execute

bash scripts/eval_ppl.sh 

To get our W4A4 zero shot accuracy on common sense tasks, please execute

bash scripts/eval_zeroshot_acc.sh