Request for Instructions on Evaluating Publicly Available Models

[ocean146]

Hi, thank you for your amazing work on the VideoChat-Online project!

Given that the checkpoints of VideoChat-Online is not publicly available now, could you please provide instructions on how to eval other publicly available models such as Qwen2-VL?

I see that MODEL_DIR_LIST is defined in line 24 of shell/eval/online_bench_sliding_window.sh, but it appears that the paths are not included in the project repository.

Could you please provide some guidance or instructions on how to adapt the evaluation script to work with other publicly available models? Any information would be greatly appreciated.

Looking forward to your reply.
Thank you!

[hhhhzp]

Hi, sorry for the late reply, and thanks again for your interest!

• VideoChatOnline-4B weights are now available on Hugging Face:
  https://huggingface.co/MCG-NJU/VideoChatOnline-4B

• The sliding_window evaluation script has been updated. It's meant to be a quick and simple way to evaluate your development models. You only need to focus on the frame indexing logic properly: at each time t (t is question timestamp), we just take the clip [t - sliding_window, t], and evaluate it using your existing offline evaluation pipeline.

• For official evaluation of other models like Qwen2-VL, we're currently adapting lmm-eval to support large-scale evaluations. This will be released in future updates.

Feel free to ask more questions!

[prachigarg23]

Hi @hhhhzp , is there any update on VideoChat-Online being integrated into lmms-eval?

[leexinhao]

Hi @hhhhzp , is there any update on VideoChat-Online being integrated into lmms-eval?

We will update it recently, stay tune!