Pipeline for embedding with SignCLIP.

[cleong110]

My process for setting up and running SignCLIP to get embeddings is very manual, involving things like copying models to the expected location and renaming them.

It would be nice to have a Dockerfile, or some other method to make embedding with signCLIP easier.

Blocks #1 , which needs this.

[cleong110]

Some subproblems I ran into when doing this:

• SignCLIP requires Python 3.8.8
• models have to be in exactly the right place, named exactly the right thing.

My Setup Script: https://github.com/cleong110/semantic-sign-language-search/blob/main/setup_signCLIP/setup_signclip.sh

[cleong110]

It might be possible to update SignCLIP to use more modern Python versions, transformers, etc.

The original VideoCLIP README says:

The code is developed under Python=3.8.8, Pytorch=1.8, cuda=11.0 with fairseq=1.0.0a0+af0389f and tested under Python=3.8.8 pytorch=1.9 cuda=11.0 fairseq=1.0.0a0+8e7bc73 during code release. Most models require transformers==3.4 for API compatibility pip install transformers==3.4. In addition, some downstream tasks may need conda install pandas.

[cleong110]

See

• Support videoclip with newer version of dependencies facebookresearch/fairseq#5404, where someone updated VideoCLIP to use newer deps,
• and also Has anyone got the MMPT example to work? facebookresearch/fairseq#5538 where someone had dependency issues with omegaconf
• and Can't install transformers==3.4 and tokenizers==0.9.2 to reproduce VideoCLIP facebookresearch/fairseq#4735 where someone tries (unsuccessfully) to get it running with python 3.9
• and also How to get video features in VideoCLIP without any access to captions facebookresearch/fairseq#5429

[AmitMY]

@J22Melody I think it will be best if there will be an easy way to run SignCLIP from python (not API). I imagine it being necessary to evaluate models when training them

[J22Melody]

Sure I will look into it.

[J22Melody]

Sorry for the long delay, and thanks for the links!

I got a minimal environment running in Python 3.12:

https://github.com/J22Melody/fairseq/tree/main/examples/MMPT#inference

... with the following fixs:

• Changes for using Python>3.11:
  facebookresearch/fairseq@4480043

• Changes for using the latest transformers:
  Support videoclip with newer version of dependencies facebookresearch/fairseq#5404 (comment)

• Use pip==24.0 to allow the installation of omegaconf<2.1

[J22Melody]

The above should allow us to run SignCLIP inference/embedding in the same environment as pose-evaluation, assuming you have everything in place.

Next steps:

• Automatically collect the model checkpoints.
• Make SignCLIP inference an installable Python package.