[Bug] The chatml_datasets and mcq_rule_evaluator are not functioning properly. #2410
Description
Prerequisite
- I have searched Issues and Discussions but cannot get the expected help.
- The bug has not been fixed in the latest version.
Type
I have modified the code (config is not considered code), or I'm working on my own tasks/models/datasets.
Environment
{'CUDA available': False,
'GCC': 'Apple clang version 17.0.0 (clang-1700.0.13.5)',
'MMEngine': '0.10.7',
'MUSA available': False,
'OpenCV': '4.11.0',
'PyTorch': '2.10.0',
'PyTorch compiling details': 'PyTorch built with:\n'
' - GCC 4.2\n'
' - C++ Version: 201703\n'
' - clang 15.0.0\n'
' - OpenMP 201811\n'
' - LAPACK is enabled (usually provided by '
'MKL)\n'
' - NNPACK is enabled\n'
' - CPU capability usage: DEFAULT\n'
' - Build settings: BLAS_INFO=accelerate, '
'BUILD_TYPE=Release, '
'COMMIT_SHA=449b1768410104d3ed79d3bcfe4ba1d65c7f22c0, '
'CXX_COMPILER=/usr/bin/c++, CXX_FLAGS= '
'-fvisibility-inlines-hidden -DUSE_PTHREADPOOL '
'-DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI '
'-DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON '
'-DUSE_PYTORCH_QNNPACK '
'-DAT_BUILD_ARM_VEC256_WITH_SLEEF -DUSE_XNNPACK '
'-DUSE_PYTORCH_METAL_EXPORT '
'-DSYMBOLICATE_MOBILE_DEBUG_HANDLE '
'-DUSE_COREML_DELEGATE -O2 -fPIC '
'-DC10_NODEPRECATED -Wall -Wextra '
'-Werror=return-type -Werror=non-virtual-dtor '
'-Werror=braced-scalar-init '
'-Werror=range-loop-construct '
'-Werror=bool-operation -Wnarrowing '
'-Wno-missing-field-initializers '
'-Wno-unknown-pragmas -Wno-unused-parameter '
'-Wno-strict-overflow -Wno-strict-aliasing '
'-Wvla-extension -Wsuggest-override '
'-Wnewline-eof -Winconsistent-missing-override '
'-Winconsistent-missing-destructor-override '
'-Wno-pass-failed -Wno-error=old-style-cast '
'-Wconstant-conversion -Qunused-arguments '
'-faligned-new -fno-math-errno '
'-fno-trapping-math -Werror=format -DUSE_MPS '
'-Wno-missing-braces, LAPACK_INFO=accelerate, '
'TORCH_VERSION=2.10.0, USE_CUDA=OFF, '
'USE_CUDNN=OFF, USE_CUSPARSELT=OFF, '
'USE_EIGEN_FOR_BLAS=ON, USE_GFLAGS=OFF, '
'USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=OFF, '
'USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=OFF, '
'USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, '
'USE_ROCM_KERNEL_ASSERT=OFF, USE_XCCL=OFF, '
'USE_XPU=OFF, \n',
'Python': '3.10.19 (main, Oct 9 2025, 15:25:03) [Clang 17.0.0 '
'(clang-1700.6.3.2)]',
'lmdeploy': "not installed:No module named 'lmdeploy'",
'numpy_random_seed': 2147483648,
'opencompass': '0.5.2+57df122',
'sys.platform': 'darwin',
'transformers': '5.2.0'}
Reproduces the problem - code/configuration sample
from opencompass.models import OpenAI
from opencompass.utils.text_postprocessors import extract_non_reasoning_content
API_BASE = ''
API_KEY = ''
MODEL_PATH = 'Qwen'
models = [
dict(
type=OpenAI,
abbr='aip-model',
path=MODEL_PATH,
key=API_KEY,
openai_api_base=API_BASE,
max_seq_len=4096,
temperature=0.7,
extra_body={
'chat_template_kwargs': {
'enable_thinking': False,
},
},
meta_template=dict(
round=[
dict(role='HUMAN', api_role='HUMAN'),
dict(role='BOT', api_role='BOT', generate=True),
],
),
max_out_len=512,
batch_size=1,
run_cfg=dict(num_gpus=0),
pred_postprocessor=dict(type=extract_non_reasoning_content),
)
]
datasets = []
chatml_datasets = [
dict(
abbr='custom_classification',
path='debug_data/opencompass_demo_data.jsonl',
evaluator=dict(
type='mcq_rule_evaluator',
answer_pattern=r'"selected_option"\s*:\s*(\d+)',
),
n=1,
),
]
infer = dict(
partitioner=dict(type='NaivePartitioner'),
runner=dict(
type='LocalRunner',
max_num_workers=1,
task=dict(type='OpenICLInferTask'),
),
)
Reproduces the problem - command or script
opencompass for_local_debug/opencompass_demo.py
Reproduces the problem - error message
/opt/homebrew/lib/python3.10/site-packages/jieba/_compat.py:18: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
import pkg_resources
Traceback (most recent call last):
File "/opt/homebrew/bin/opencompass", line 6, in
sys.exit(main())
File "/opt/homebrew/lib/python3.10/site-packages/opencompass/cli/main.py", line 263, in main
cfg = get_config_from_arg(args)
File "/opt/homebrew/lib/python3.10/site-packages/opencompass/utils/run.py", line 102, in get_config_from_arg
chatml_datasets = consturct_chatml_datasets(config['chatml_datasets'])
File "/opt/homebrew/lib/python3.10/site-packages/opencompass/utils/run.py", line 501, in consturct_chatml_datasets
eval_cfg = func_eval_cfg(dataset['evaluator'])
File "/opt/homebrew/lib/python3.10/site-packages/opencompass/utils/run.py", line 467, in init_mcq_rule_evaluator
eval_cfg = optional_evaluator['rule_evaluator']
KeyError: 'rule_evaluator'
Other information
