Request: Documentation/Demo for ViT Prisma with CLIP + Implementation Guidance

[Lancelottery]

I've noticed the recent CLIP-related updates in the repo. Could you provide documentation or examples showing how to use ViT Prisma on CLIP, specifically with logit lense and activation patching?

I'm also working on a project where I'm trying to activation patch CLIP's vision encoder (ViT) using ViT Prisma while keeping the text encoder unchanged. I'm wondering whether I should hook the entire CLIP model or if just hooking the visual encoder would be sufficient?

Any guidance would be appreciated!

[soniajoseph]

Hi there,

Thanks for your request! It is noted.

CLIP logit lenses are actually a bit tricky b/c we've found that the intermediate activations don't align with the output space in a nice way that permits an easy logit lens. There's a scrappy notebook here if you want to check it out: https://colab.research.google.com/drive/1SliS1C8O1wQ9UwAVHkRdzgx8CnQ7halI#scrollTo=LoBwMA4hU3N3

Activation patching should remain the same as a vanilla vision transformer.

For CLIP, we only load in the vision encoder. Currently, the library is not adapted for the text encoder, although that is something we will make part of our workflow in the future. If you're just looking at the vision encoder, then that should be sufficient.

We'll add the tutorials/notebooks later in the year! Feel free to push any tutorials or code of your own if they solve this issue before then.

Best,
Sonia