Requirements

You can install the required packages with the following command:

Nvidia

conda create -name pangu-dev python=3.12
conda activate pangu-dev
pip install -r requirements.txt

Ascend

Assume the vllm==0.9.2 and vllm-ascend==0.9.2rc1 are already installed, then:

pip install -r requirements-ascend.txt

Evaluation

# eval on openPangu with avg@16 on amc23
ASCEND_RT_VISIBLE_DEVICES=0,1 ./ascend-eval.sh amc23 16

# eval other model
ASCEND_RT_VISIBLE_DEVICES=0,1 ./ascend-eval-other.sh Qwen/Qwen2.5-MATH-7B aime24 16

Acknowledgement

The codebase is adapted from math-evaluation-harness.