internvl 2.5 8b 和qwen2.5vl的结果

[yunzhuzhang0918]

经多次测试，发现internvl 2.5 8b的videomme，longvideobench_val_v和mlvu的结果与官方结果差异巨大，其中mlvu64帧结果只有57.89，比官方的60低了2个点。
qwen2.5vl 的videomme 结果只有62.9%，与官方的65.1差异巨大
测试时直接使用lmms eval跑的，什么也没有修改
建议lmms eval团队可以重新测试一下这些模型，看看是不是什么setting设置的不对？
期待回复

[Luodian]

We also observed this issue in both Qwen-2-VL and Qwen-2.5-VL. In our evaluation, the performance on mmmu_val appears to be around 51–52 as well.

I was thinking they are using GPT4o or other LLM as judge to directly give the score for model output versus ground truth.

We think we should raise this issue in Qwen's repo and to see how to reproduce the exact results.

Evaluation results should be exact and we could check the following parts:

1. model input: how to organize the prompts.
2. sampling params: temperature, max_pixels, sampled frames, etc.
3. post-evaluation processing rule: how to extract answer or judge score for given model output.
4. post-evaluation aggregate rule: how to aggregate each item's score to produce final reported metric, such as accuracy, per-class average accuracy, recall, f1 score.

[yunzhuzhang0918]

I am evaluating some Multi-choice video benchmark, such as videomme, mlvu, longvideobench. These benchmark don't need gpt4o as judger.
I believe that some settings, like testing frames, fps, or image resolution are not aligned with official implematation.
I will raise an issue in Qwen's repo, too.

[Chenfeng1271]

Hi, have you met a bug with Qwen2.5vl on video benchmark like below? I try device_map to cuda:0, cuda, auto. It doesnot work

[Chenfeng1271]

Hi, have you met a bug with Qwen2.5vl on video benchmark like below? I try device_map to cuda:0, cuda, auto. It doesnot work

Sorry for bothering u. I find it is raised from transformers version...

[whu125]

蹲，感觉好多模型复现出来都低两个点左右。。。

[Luodian]

我经验来看就是 mmmu 和 videomme 复现比较困难😂，尤其是 mmmu 会差很大一截，不过我们会和 qwen 团队确认一下他们的 evaluation 细节

[AlexCo1d]

@hello-bluedog 请问你videomme qwenvl 2.5 使用的脚本是什么？可以分享一下吗，不确定model args参数是那些

[Luciennnnnnn]

@Luodian Any updates? I still can not reproduce mmmu_val

[yunzhuzhang0918]

@hello-bluedog 请问你videomme qwenvl 2.5 使用的脚本是什么？可以分享一下吗，不确定model args参数是那些
我是这个，就默认的参数

accelerate launch --num_processes 8 --main_process_port 12344 -m lmms_eval
--model qwen2_5_vl
--model_args "pretrained=models/Qwen2.5-VL-7B-Instruct,fps=2,use_flash_attention_2=True,max_num_frames=768"
--tasks videomme
--batch_size 1
--log_samples
--log_samples_suffix llava_video_7B
--output_path ./logs_qwen2_5vl_external

@Chenfeng1271 Hi, I want to know the version of Transformers for conducting Qwen evaluation. Thanks

[ZizhenWang]

same question, I use vllm to evaluate Qwen2.5VL-7B on mlvu_dev, it only achieves 48.49, could anyone help to check, my command is accelerate launch --num_processes=8 -m lmms_eval --model vllm --model_args model_version=Qwen/Qwen2.5-VL-7B-Instruct --tasks mlvu_dev --batch_size 1

[Couteaux123]

Hi, have you met a bug with Qwen2.5vl on video benchmark like below? I try device_map to cuda:0, cuda, auto. It doesnot work

请问您这边使用的transformers和torch版本是什么呢，我也遇到过这种问题

[yaoyaolove]

我使用qwen2.5vl7b在muirbench上的评测分数为54.27，远不及原论文的59.6