Will Embeding change?

[Z-Zili]

I have been studying LoRA recently and I noticed that during pre-training, the word vectors change as the training progresses. However, what about when using LoRA for fine-tuning? Do the word vectors still change, or is it only the attention weights.

[yezhengmao1]

Typically, fine-tuning with LoRA does not alter the embedding layer. However, in some models, the training process of the LoRA module may concurrently train the embedding layer (due to the need to add additional Chinese tokens).

[Z-Zili]

Typically, fine-tuning with LoRA does not alter the embedding layer. However, in some models, the training process of the LoRA module may concurrently train the embedding layer (due to the need to add additional Chinese tokens).

So, what you mean is that when fine-tuning, if there is a word in the dataset that the model has not seen before, we need to adjust the embedding layer?

[yezhengmao1]

Typically, fine-tuning with LoRA does not alter the embedding layer. However, in some models, the training process of the LoRA module may concurrently train the embedding layer (due to the need to add additional Chinese tokens).

So, what you mean is that when fine-tuning, if there is a word in the dataset that the model has not seen before, we need to adjust the embedding layer?

yes, the best way is to add the new token ids to the tokenizer model and then enable the embedding layer's grad to train the LoRA module.

[Z-Zili]

OK，Thank you for your answer

[owos]

THank you for the response @yezhengmao1 . What if I want to use lora to update the embedding layers?

[yezhengmao1]

THank you for the response @yezhengmao1 . What if I want to use lora to update the embedding layers?

the embedding is just lookup table, can not use the lora to it.