Handling Reduced Visual Tokens During Inference in LLMs

[naajeehxe]

I’m not sure if the LLM can handle a different number of visual tokens than what was used during training. If N visual tokens are discarded, is there a step to adjust the dimension before feeding them into the LLM?

[UnableToUseGit]

In their code, redundant visual tokens are pruned inside LLM (decode layer[2, 6, 15, 19]). There is no step to adjust the dimension. After pruned, these rest tokens(hidden_states) will be passed in the next layer.