Can't run openai api eval

[kratorado]

I deployed a vllm backed openai api server to serve my tuned model. And I want to evaluate the performance using opencompass.
my config file:

from mmengine.config import read_base
from opencompass.models import OpenAI
from opencompass.partitioners import NaivePartitioner
from opencompass.runners.local_api import LocalAPIRunner
from opencompass.tasks import OpenICLInferTask

with read_base():
    from ..summarizers.medium import summarizer
    from ..datasets.cmmlu.cmmlu_gen_c13365 import cmmlu_datasets

datasets = [
    *cmmlu_datasets,
]

models = [
    dict(
        type=OpenAI,
        openai_api_base="http://127.0.0.1:8080/v1/chat/completions",
        abbr="mymodel",
        path="mymodel",
        key='-',  # please give you key
        run_cfg=dict(num_gpus=0),
        query_per_second=1,
    ),
]

infer = dict(
    partitioner=dict(type=NaivePartitioner),
    runner=dict(
        type=LocalAPIRunner,
        max_num_workers=1,
        concurrent_users=1,
        task=dict(type=OpenICLInferTask)),
)

work_dir = "outputs/api_mymodel/"

After execute the command python3 run.py configs/api_examples/eval_api_mymodel.py, I got many ERROR logs from console.

# many lines like
04/27 20:44:22 - OpenCompass - ERROR - /data/homedir/work/opencompass/opencompass/opencompass/runners/local_api.py - launch - 131 - task OpenICLInfer[mymodel/cmmlu-virology] fail, see
outputs/api_myqwen/20240427_204403/logs/infer/mymodel/cmmlu-virology.out

And what's in that .out files is:

Traceback (most recent call last):it/s]
  File "/data/homedir/work/opencompass/opencompass/opencompass/runners/local_api.py", line 122, in launch
    inferencer.run(inferencer, tokens)
  File "/data/homedir/work/opencompass/opencompass/opencompass/runners/local_api.py", line 52, in monkey_run
    self._inference()
  File "/data/homedir/work/opencompass/opencompass/opencompass/tasks/openicl_infer.py", line 134, in _inference
    inferencer.inference(retriever,
  File "/data/homedir/work/opencompass/opencompass/opencompass/openicl/icl_inferencer/icl_gen_inferencer.py", line 133, in inference
    entry, golds = list(zip(*datum))
ValueError: not enough values to unpack (expected 2, got 1)

the server did not receive any request.

[Mor-Li]

Hi kratorado, I recommend running your command with the --debug option to enable more detailed error logging. This will provide more insight into what might be causing the errors. You can do this by modifying your command like so:

python3 run.py configs/api_examples/eval_api_mymodel.py --debug

After running with debugging enabled, check the detailed logs from console to better understand the errors.

Best of luck!

[kratorado]

Thanks for reply.
After adding --debug option, the logs are the same as previous ones.

Traceback (most recent call last):it/s]
  File "/data/homedir/work/opencompass/opencompass/opencompass/runners/local_api.py", line 122, in launch
    inferencer.run(inferencer, tokens)
  File "/data/homedir/work/opencompass/opencompass/opencompass/runners/local_api.py", line 52, in monkey_run
    self._inference()
  File "/data/homedir/work/opencompass/opencompass/opencompass/tasks/openicl_infer.py", line 134, in _inference
    inferencer.inference(retriever,
  File "/data/homedir/work/opencompass/opencompass/opencompass/openicl/icl_inferencer/icl_gen_inferencer.py", line 133, in inference
    entry, golds = list(zip(*datum))
ValueError: not enough values to unpack (expected 2, got 1)

The info is clear enough, the function list() returns 1 value but the code here is going to unpack 2 values.

[kratorado]

I changed some codes to skip this exception and met another one:

Traceback (most recent call last):
  File "/data/homedir/work/opencompass/opencompass/run.py", line 4, in <module>
    main()
  File "/data/homedir/work/opencompass/opencompass/opencompass/cli/main.py", line 309, in main
    runner(tasks)
  File "/data/homedir/work/opencompass/opencompass/opencompass/runners/base.py", line 38, in __call__
    status = self.launch(tasks)
  File "/data/homedir/work/opencompass/opencompass/opencompass/runners/local.py", line 103, in launch
    task.run()
  File "/data/homedir/work/opencompass/opencompass/opencompass/tasks/openicl_infer.py", line 84, in run
    self._inference()
  File "/data/homedir/work/opencompass/opencompass/opencompass/tasks/openicl_infer.py", line 134, in _inference
    inferencer.inference(retriever,
  File "/data/homedir/work/opencompass/opencompass/opencompass/openicl/icl_inferencer/icl_gen_inferencer.py", line 147, in inference
    results = self.model.generate_from_template(
  File "/data/homedir/work/opencompass/opencompass/opencompass/models/base.py", line 165, in generate_from_template
    return self.generate(inputs, max_out_len=max_out_len, **kwargs)
  File "/data/homedir/work/opencompass/opencompass/opencompass/models/openai_api.py", line 129, in generate
    results = list(
  File "/home/devops/miniconda3/envs/opencompass/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator
    yield _result_or_cancel(fs.pop())
  File "/home/devops/miniconda3/envs/opencompass/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel
    return fut.result(timeout)
  File "/home/devops/miniconda3/envs/opencompass/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/home/devops/miniconda3/envs/opencompass/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/home/devops/miniconda3/envs/opencompass/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/data/homedir/work/opencompass/opencompass/opencompass/models/openai_api.py", line 184, in _generate
    max_out_len, context_window - self.get_token_len(str(input)) - 100)
  File "/data/homedir/work/opencompass/opencompass/opencompass/models/openai_api.py", line 282, in get_token_len
    enc = self.tiktoken.encoding_for_model(self.path)
  File "/home/devops/miniconda3/envs/opencompass/lib/python3.10/site-packages/tiktoken/model.py", line 101, in encoding_for_model
    return get_encoding(encoding_name_for_model(model_name))
  File "/home/devops/miniconda3/envs/opencompass/lib/python3.10/site-packages/tiktoken/model.py", line 88, in encoding_name_for_model
    raise KeyError(
KeyError: 'Could not automatically map mymodel to a tokeniser. Please use `tiktoken.get_encoding` to explicitly get the tokeniser you expect.'

[qianchen94]

I met the same issue,
Have you solved it？

[rangehow]

一样，但是我是测的本地的模型gemma-2b

[rank0]: Traceback (most recent call last):
[rank0]:   File "/data/ruanjh/lmdeploy/opencompass/opencompass/tasks/openicl_infer.py", line 162, in <module>
[rank0]:     inferencer.run()
[rank0]:   File "/data/ruanjh/lmdeploy/opencompass/opencompass/tasks/openicl_infer.py", line 90, in run
[rank0]:     self._inference()
[rank0]:   File "/data/ruanjh/lmdeploy/opencompass/opencompass/tasks/openicl_infer.py", line 129, in _inference
[rank0]:     inferencer.inference(retriever,
[rank0]:   File "/data/ruanjh/lmdeploy/opencompass/opencompass/openicl/icl_inferencer/icl_gen_inferencer.py", line 133, in inference
[rank0]:     entry, golds = list(zip(*datum))
[rank0]: ValueError: not enough values to unpack (expected 2, got 1)

[starplatinum3]

[Bug] ValueError: too many values to unpack (expected 2) · Issue #673 · open-compass/opencompass · GitHub
#673