Are LLM and MM Projector also tuned during unlearning?

[herbwood]

I tried reproducing the results in your paper by training the unlearning methods (GA, GD, ICD, IDK, and KL) using the same experimental settings described in the paper. I used the released pretrained weights(https://huggingface.co/gray311/vlm_unlearning_ft_llava_phi_3_mini) and followed the configurations as closely as possible. However, I observed that the results differ from those reported in the paper.

I also checked the configuration files on Huggingface and noticed that both tune_mm_projector and tune_language_model are set to true during LoRA fine-tuning. Could you please clarify whether, during unlearning, you updated not only the LoRA adapters but also the MM projector and the LLM layers?

[gray311]

Thanks for your interest in our work.

1. All four methods are training-based, so the unlearning results largely depend on the training steps. In our paper, we mention that we use an early stopping strategy to select suitable checkpoints, aiming to maximize forget quality while maintaining a certain level of model utility. For example, in the case of the GA method, we keep its loss within -10 to ensure model utility.

2. During unlearning， we use LoRA to fine-tune LLM layers， while full fine-tuning the MM projector layer.