两种评测方式结果差别很大是怎么回事。

[mathCrazyy]

如采用intnernvl3-1b-instruct ，在docvqa, mustard, ocrbench, textvqa 上进行测试。
方式 1.
启动服务: CUDA_VISIBLE_DEVICES=6,7 vllm serve /vlm/pretrain_models/OpenGVLab/InternVL3-1B --dtype auto --port 8000 --gpu_memory_utilization 0.4 --tensor-parallel-size 2 --pipeline-parallel-size 1
python3 -m lmms_eval
--model openai_compatible
--model_args "api_base=http://localhost:8000,api_key=EMPTY,model_version=/vlm/pretrain_models/OpenGVLab/InternVL3-1B"
--tasks textvqa,docvqa,ocrbench,mmstar
--batch_size 1
--log_samples
--output_path ./logs/2025.07.02
--limit 100

方式 2.
export CUDA_VISIBLE_DEVICES=0,1,2,3
TASK=$1
CKPT_PATH=$2
echo $TASK
TASK_SUFFIX="${TASK//,/_}"
echo $TASK_SUFFIX

accelerate launch --num_processes 4 --main_process_port 12380 -m lmms_eval
--model internvl2
--model_args pretrained=$CKPT_PATH
--tasks $TASK
--batch_size 1
--log_samples
--log_samples_suffix $TASK_SUFFIX
--output_path ./2025.07.02/ \

我做了以下事情来排查原因：

1. 固定seed；
2. 统一图片读取为RGB
3. 修改internvl3-1b 下的config 去除template 中的system prompt
4. 同一batch_size 为 1.
   均没有拉近这个差距，请问是为什么呢

[Luodian]

Do you set --limit to second evaluation method?

[mathCrazyy]

Do you set --limit to second evaluation method?

The -limit I understand is to make a choice on the quantity of the evaluation data, isn't it? In fact, for both of my evaluation methods, I have set a -limit of 100

[Luodian]

Yes I just to confirm the setting is correct if both are evaluating the same quantity of data.

[mathCrazyy]

Yes I just to confirm the setting is correct if both are evaluating the same quantity of data.

And I also confirmed the output logs, which are 100 identical pieces of data, but their results are different, as follows:

[Luodian]

I was suspecting that if it's appropriate to use internvl2 to run internvl3?

The vllm serve method maybe more correct at least it's a standard way of hosting any model.

[mathCrazyy]

I was suspecting that if it's appropriate to use internvl2 to run internvl3?

The vllm serve method maybe more correct at least it's a standard way of hosting any model.

In fact, I also reached a similar conclusion using qwen2-vl. The results of offline and online are different. Just now I found that the maximum value of the cut image for online and offline is different, which will have a certain impact. For online, it is 12, and for offline, it is 6. After modifying online to 6, the docvqa results were similar, but there were still some differences in other datasets.

[mathCrazyy]

It was found that in the online mode, the chat_template under the internvl3 model file is ali, which leads to inconsistent subsequent queries and is the main reason for the inconsistent metrics.