Model code configuration issue.

[shenxiaochenn]

Hello, I note you have open-sourced two implementations of toklip: transfommer_toklip.py and timm_model_toklip.py. During my debugging process, I observed that you utilised TimmModel. I attempted to adapt it to the VisionTransformer implementation, but encountered significant discrepancies between the two codebases. For instance, within TimmModel, the class token is implicitly absent, whereas it is explicitly defined in VisionTransformer. Additionally, there are issues concerning the attention pool. Might you be able to update the code or provide the latest version of transfommer_toklip.py?

Thanks!

[FelixMessi]

Hi, thanks for your interest!

Actually, transformer_toklip.py is designed for the OpenAI CLIP architecture, while timm_model_toklip.py follows the SigLIP implementation. SigLIP does not use an explicit [CLS] token, which explains the architectural differences you noticed.

If you plan to adapt the VisionTransformer implementation for SigLIP, some code modifications will indeed be required.

Regarding the attention pooling, I’m not entirely sure which specific issue you’re referring to — could you kindly share more details or point to the relevant code lines? That would help me better understand and assist you.