The code implementation does not match the description in the paper.

[chenxn2020]

@zhangshaolei1998
Hi there, thanks for the great work. I attempted to reproduce your work. While reading your code, I found that the implementation differs from the description in the paper.
Specifically, first, the paper describes the Pre-fusion module as taking the original visual input $H^{V}$ (as shown in Figure 6 of the paper). However, in llavamini/model/llavamini_arch.py, at line 369 and 410, the compressed visual tokens (named as compressed_image_features) are also fed into the pre-fusion module. as shown in below:

#line 369
x=torch.cat([global_image_features,compressed_image_features,text_embedding],dim=1)
#line 410
# modality pre-fusion
for layer in self.get_model().prefusion_layers:
    x = layer(x,attention_mask=attention_mask,position_ids=position_ids)[0]

Secondly, the compressed visual tokens (named as compressed_image_features in your code) ultimately come from the $x$ of the pre-fusion module, which does not align with the description in the paper. See llavamini/model/llavamini_arch.py, at lines 410-417.

# modality pre-fusion
for layer in self.get_model().prefusion_layers:
    x = layer(x,attention_mask=attention_mask,position_ids=position_ids)[0]

fusion_text_features=x[:,-1*input_ids.size(1):,:]
compressed_image_features=x[:,-1*input_ids.size(1)-1*compressed_image_features.size(1):-1*input_ids.size(1),:]
fusion_text_features=fusion_text_features*(~padding_mask).unsqueeze(-1).int()+all_text_embedding*padding_mask.unsqueeze(-1)

return compressed_image_features,fusion_text_features

Is there anyone who can explain the reason for this inconsistency? Or did I misunderstand something?

[wuhuaijin]

I found this problem too. Even, when I use the compressed_image_features without coming through pre-fusion module (which aligns with the description in the paper), the results are significantly different. I sincerely request the authors to give a corresponding explanation.

[NieCha]

Additionally, I found that the paper did not mention the training dataset used, suspecting that a better configuration than the baseline model was used

[ChicyChen]

What's more, I think the author's implementation of encode_image_mini may always fuse all text tokens with image tokens - this is a problem during training, since the "answer" text part will also be fused together when calling prepare_inputs_labels_for_multimodal.

[LeslieZJC]

I also found this issue:all_text_embedding = self.get_input_embeddings()(input_ids.clamp(min=0)).detach(),​.Does all_text_embedding include embeddings of the answer? Is this correct?​

[mysticalwing]

I found another miss-match: in paper the compression is performed after vision-tokens projected to text-embedding space by projector(see Figure 6). but in code, the compression is performed before the projector. see

LLaVA-Mini/llavamini/model/llavamini_arch.py

Line 364 in 47da113

compressed_image_features,attn=self.get_model().compressor(image_features)

[mysticalwing]

I found this problem too. Even, when I use the compressed_image_features without coming through pre-fusion module (which aligns with the description in the paper), the results are significantly different. I sincerely request the authors to give a corresponding explanation.

It's better or worse ? and how much is the gap?

[sywang-xverse]

I found this problem too. Even, when I use the compressed_image_features without coming through pre-fusion module (which aligns with the description in the paper), the results are significantly different. I sincerely request the authors to give a corresponding explanation.

It's better or worse ? and how much is the gap?

worse