Inference with downstream_commentary_none_open.pth

[IsmaelElsharkawi]

Hi,
Thanks for the awesome work you guys did in this project!
I was trying to reproduce your work, however, whenever I use the checkpoint downstream_commentary_none_open.pth, the commentary output is hallucination. I'm using this fork https://github.com/IsmaelElsharkawi/UniSoccer/tree/759dd400fc082e3c7054e37198c3c70b93935d65
Please, let me know if I'm doing anything wrong.
Thanks.

[jyrao]

Hi @IsmaelElsharkawi,

Thanks for messaging and following! What kind of hallucination you've met with that? You can give some of the exammples. As for the usage of checkpoint, you could refer to #5 and #3.

Best,
Jiayuan

[IsmaelElsharkawi]

Hi @jyrao,

Here is the inference for all the MatchTime test set. https://drive.google.com/file/d/1PWU_fnhXCG6jfA1JYZc1qaEOCTAF10kd/view?usp=sharing

[jyrao]

Hi @IsmaelElsharkawi,

Seeing your results, I believe there is something wrong with the file choosing, when you use downstream_commentary_none_open.pth as for commentary generation, have you use pretrained_both.pth as visual encoder? Or you can directly use downstream_commentary_all_open.pth while keeping the hyper-paremeters of followings all true:

parser.add_argument("--open_visual_encoder", type=bool, default=False)

parser.add_argument("--open_llm_decoder", type=bool, default=False)

Hopefully that helps.

Best,
Jiayuan

[IsmaelElsharkawi]

Hi @jyrao,
I have used downstream_commentary_none_open.pth and pretrained_both.pth, while keeping the hyperparameters set to False. This was what I used to generate the csv that I mentioned above.

In a separate experiment, I tried the downstream_commentary_all_open.pth, with pretrained_both.pth, and the generated comments make sense.

Also, AFAIU from the code, --open_visual_encoder does not affect inference, it only affects the requires_grad property for the visual encoder parameters during training.