How to reproduce result Table 7, for Chameleon 30B on COCO ?

[MaxHeuillet]

Hello,

We are aiming to reproduce the results from Table 7 in the Chameleon paper. We are specifically working on the evaluation for the COCO dataset using Chameleon 30b, reporting 120.2 CIDER score.

Here are some of the main design questions we have:

1. Regarding the N-shot selection of the images (N=2), should we select them:

• at random from other images in the train set
• top-N between current image in the val set, and images in the train set using cosine similarity on CLIP embeddings (“laion/CLIP-ViT-B-32-laion2B-s34B-b79K »)
• Was the entire train set used for selecting the N-shot observations, or only a subset?
• with COCO we have access to 5 captions per observations. Should we select only the first caption available, or were the captions selected randomly?

2. Prompt design:

<image1> Caption this image. Caption: <caption1> \n (…) <image2> Caption this image. Caption: <captionN> \n <image> Caption this image. Caption: <eot>

4. Max new tokens: 30

So far we have tried a combination of these strategies, and the best results we got are around 40-60 CIDER.

Thank you for your help!