LLMEval is a comprehensive evaluation system for assessing Large Language Models (LLMs) on mathematical reasoning benchmarks. It supports both online (API-based) and offline (local inference) modes with built-in answer verification.
- Multiple Inference Backends: Support for vLLM (GPU/NPU) and SGLang with data parallelism
- Flexible Evaluation Modes: Online server mode and offline local inference
- Benchmark Coverage: AIME 2024/2025, MATH-500, GSM8K, and more
- Resume Capability: Automatically resume interrupted evaluations
- Verification Support: Built-in answer extraction and correctness verification
We have successfully reproduced various open-source model results on the AIME 2024 & AIME 2025 benchmarks.
For benchmarks like AIME24, which contains only 30 problems, it is crucial to sample multiple responses as this can introduce high variance across repeated runs. All results below use 64 samples per problem to ensure stability.
| Dataset | LLMEval | Official Report |
|---|---|---|
| AIME24 | 70.625 | 72.6 |
| AIME25 | 55.052 | 59.0 |
| MATH-500 | 93.2 | 94.3 |
| Dataset | LLMEval | Official Report |
|---|---|---|
| AIME24 | 78.65 | 79.5 |
| AIME25 | 67.22 | 69.5 |
| Dataset | LLMEval | Official Report |
|---|---|---|
| AIME24 | 81.25 | 82.2 |
| AIME25 | 72.66 | 73.3 |
| Dataset | LLMEval | Official Report |
|---|---|---|
| AIME24 | 70.41 | 69.0 |
| AIME25 | 59.16 | 53.3 |
| Software | Version |
|---|---|
| Python | >= 3.10 |
| torch | >= 2.0 |
For Huawei Ascend NPU users:
- CANN >= 8.1.RC1
- torch_npu >= 2.5.1
- vllm-ascend >= 0.7.3.post1
git clone https://github.com/jianzhnie/LLMEval.git
# Or use gitee mirror: git clone https://gitee.com/jianzhnie/LLMEval.git
cd LLMEval
pip install -e .For GPU users:
pip install vllm>=0.7.0For Huawei Ascend NPU users:
# Install vllm
git clone -b v0.7.3 --depth 1 https://github.com/vllm-project/vllm.git
cd vllm
pip install -r requirements-build.txt
VLLM_TARGET_DEVICE=empty pip install -e .
# Install vllm-ascend
git clone -b v0.7.3.post1 --depth 1 https://github.com/vllm-project/vllm-ascend.git
cd vllm-ascend
export COMPILE_CUSTOM_KERNELS=1
python setup.py installFirst, start the vLLM server with the following command:
model_path="Qwen/QwQ-32B" # or model to the path where the model is located
model_name="Qwen/QwQ-32B"
num_gpus=8
max_model_len=32768 # ✅ 支持 32k 上下文
gpu_memory_utilization=0.9 # ✅ 提高内存利用率
python -m vllm.entrypoints.openai.api_server \
--model $model_path \
--trust-remote-code \
--served-model-name $model_name \
--tensor-parallel-size $num_gpus \
--gpu-memory-utilization $gpu_memory_utilization \
--max-model-len $max_model_len \
--enforce-eager \
--port 8090Adjust the tensor_parallel_size parameter based on your available devices. Please refer to the script for more details.
Optional, Start SGLang server/router.Since the evaluation could takes days, we also suggest using SGLang with data parallelism to accelerate the evaluation. Refer to SGLang documentation for more details.
# Use router to support better data parallelism
python -m sglang_router.launch_server --model-path Qwen/QwQ-32B --dp-size 4 --host=0.0.0.0 --port=30000Adjust the dp_size parameter based on your available devices. Also adjust the port in following commands.
After starting the vLLM service, run the inference script to generate responses.
output_dir="./output/Qwen/QwQ-32B"
model_name="Qwen/QwQ-32B"
base_url="http://127.0.0.1:8090/v1"
n_samples=64 # Default sample size for aime24 and aime25
# Create output directory if it doesn't exist
mkdir -p "${output_dir}"
# --- Run Inference Tasks ---
# aime24 (repeated sample 64 times)
python ./llmeval/vllm/online_server.py \
--input_file "./data/aime24.jsonl" \
--output_file "${output_dir}/aime24_bz${n_samples}.jsonl" \
--base_url "${base_url}" \
--model_name "${model_name}" \
--n_samples "${n_samples}" \
--system_prompt_type empty \
--max_workers 8
# aime25 (repeated sample 64 times)
python ./llmeval/vllm/online_server.py \
--input_file "./data/aime25.jsonl" \
--output_file "${output_dir}/aime25_bz${n_samples}.jsonl" \
--base_url "${base_url}" \
--model_name "${model_name}" \
--n_samples "${n_samples}" \
--system_prompt_type empty \
--max_workers 8Please refer to the script for more details.
Note: We apply repeated sampling to reduce evaluation variance, but it may take a long time to complete (more than 8 hours depending on your device).
After completing the inference, use the following commands for scoring:
output_dir="./output/Qwen/QwQ-32B"
n_samples=64 # Default sample size for aime24 and aime25
# Evaluation output directory
reval_dir="${output_dir}/eval_score"
# Create evaluation directory if it doesn't exist
mkdir -p "${reval_dir}"
# --- Evaluate Each Task ---
# Evaluate aime24
python ./llmeval/tasks/math_eval/eval.py \
--input_path "${output_dir}/aime24_bz${n_samples}.jsonl" \
--cache_path "${reval_dir}/aime24_bz${n_samples}.jsonl" \
--task_name "math_opensource/aime24" \
--max_workers 16 \
> "${reval_dir}/aime24_bz${n_samples}_res_result.txt"
# Evaluate aime25
python ./llmeval/tasks/math_eval/eval.py \
--input_path "${output_dir}/aime25_bz${n_samples}.jsonl" \
--cache_path "${reval_dir}/aime25_bz${n_samples}.jsonl" \
--task_name "math_opensource/aime25" \
--max_workers 16 \
> "${reval_dir}/aime25_bz${n_samples}_res_result.txt"Please refer to the script for more details.
Common parameters for both online and offline modes:
| Parameter | Default | Description |
|---|---|---|
--n_samples |
1 | Number of samples per problem |
--temperature |
0.6 | Sampling temperature |
--top_p |
0.95 | Nucleus sampling parameter |
--top_k |
40 | Top-k sampling parameter |
--max_tokens |
32768 | Maximum tokens to generate |
Online mode specific:
| Parameter | Default | Description |
|---|---|---|
--base_url |
Required | API server URL |
--model_name |
Required | Model name for API |
--max_workers |
128 | Concurrent request threads |
Offline mode specific:
| Parameter | Default | Description |
|---|---|---|
--model_name_or_path |
Required | Local model path or HuggingFace ID |
--tensor_parallel_size |
1 | GPU count for tensor parallelism |
--gpu_memory_utilization |
0.9 | GPU memory fraction |
--batch_size |
128 | Inference batch size |
math_opensource/aime24math_opensource/aime25math_opensource/math500math_opensource/hmmt25math_opensource/gsm8k
If inference is interrupted, simply re-run the same command. The script automatically:
- Reads existing output file
- Counts completed samples per problem
- Continues from where it left off
For contexts exceeding 32K tokens, use RoPE scaling:
vLLM:
python -m vllm.entrypoints.openai.api_server \
--model Qwen/Qwen3-8B \
--rope-scaling '{"rope_type":"yarn","factor":4.0}' \
--max-model-len 131072SGLang:
python -m sglang.launch_server \
--model-path Qwen/Qwen3-8B \
--json-model-override-args '{"rope_scaling":{"rope_type":"yarn","factor":4.0}}' \
--context-length 131072LLMEval/
├── llmeval/
│ ├── vllm/ # Inference engines
│ │ ├── online_server.py
│ │ ├── offline_infer.py
│ │ └── verifier_offline_infer.py
│ ├── tasks/ # Evaluation tasks
│ │ └── math_eval/
│ │ ├── eval.py
│ │ ├── math_score.py
│ │ └── utils_parser.py
│ └── utils/ # Utilities
│ ├── config.py
│ ├── logger.py
│ ├── template.py
│ └── verifier_template.py
├── scripts/ # Shell script examples
└── data/ # Benchmark datasets
This project is licensed under the MIT License.