CLIP

[awsaf49]

I think CLIP (Contrastive Language-Image Pre-training) from Openai would be a great addition to this library. Also having a script for clip training (vision + language) on datasets like LAION would be even more sweet.

Here are some references,

1. https://github.com/openai/CLIP [Official]
2. https://github.com/mlfoundations/open_clip [Communiy, used by timm]
3. https://arxiv.org/abs/2111.02114 [LAION]

[leondgarse]

Ya, I would like that too, and tried the basic implementation in 05-08_clip.md, but still havn't got some training tests...

[leondgarse]

The basic framework is working now, but cannot guarantee the efficiency, as it's only tested on a subset of COCO caption... May refer Custom caption detaset for creating a dataset, and keras_cv_attention_models/clip for a basic usage, and kecam_caption_test.ipynb for a basic test in colab.

[leondgarse]

It's now a single script clip_train_script.py for both TF and PyTorch backend, and a basic usage keras_cv_attention_models/clip. They are using almost same model / loss / data, but still TF training results not satisfying. I think maybe it's their optimizers behaving different...