Does Llama 3.2 vision use CLIP?

[vedantroy]

2 questions really:

1. In the [source code] (https://github.com/pytorch/torchtune/blob/main/torchtune/models/llama3_2_vision/_model_builders.py) for Llama 3.2 vision, there's a component called clip_vision_encoder. I assume this is not referencing OpenAI CLIP?

2. Is this a correct representation of the Llama 3.2 vision architecture? I read the paper + browsed the source code, and this is what I came up with:

flowchart LR
    vision["{image}"] --> clipT[CLIP Transformer]
    clipT --> t[t Transformer]
    t --> cat[cat along feature dim]
    
    clipT -- "hidden state 1" --> cat
    clipT -- "hidden state 2" --> cat
    clipT -- "hidden state 3" --> cat
    
    cat --> MLP
    
    MLP -- "cross attention" --> decoder[Decoder]
    text["{text}"] --> decoder

Loading