Reproduction Issue

[jessyjinzy]

Hi!
Thanks for making your code & checkpoints available! But I encountered some issues while trying to reproduce the code.
This is my result using the merge.pkl, 720p video segments from MatchTime and downstream_commentary_all_open.pth. I only made some code change in "dir_path" things and used the default settings of hyper-parameters. Is there anything wrong?
infer_world_720p.csv

[jyrao]

As for this issue, you need to adjust two parameters to open both "visual_encoder" and "llm_decoder" to fit the checkpoint you used, unless you are using "none_open" version of checkpoint.

Also, to be noted, 224p video is enough since SigLIP preprocessor will resize the frames to 224*224.