partially-tune vision tower

[RaptorMai]

Hi, I want to fine-tune the vision tower. However, I saw there is some TODO in the code for partially-tune. I wonder if it may cause some problems?

[ZhangXJ199]

Hello, fully fine-tuning the vision tower is fine, but partial fine-tuning may have some issues.

[RaptorMai]

Thanks for the quick responses. If I want to partial fine-tune, is there any issue that I need to pay attention to?

[RaptorMai]

for example, if i zero1 or zero0, will it not resolve the potential problem?

[ZhangXJ199]

Sorry, we haven't had a chance to test partial fine-tuning, so we left a note here.