In experiments/train.py，training labels problem

[Jean-ZW]

As shown over here, the 2 labels creating method only masked the starting special tokens until the start of user content, not the whole query inputs until the start assistant content like what regular SFT does.

Does this make sense? train not just the answer but the query content as well? Or what it wants is not only, just train for instruction following, answer generation ability, but also a bit like pretrain for latent sentiment space alignment for the adapters?

And what does these comment notes means? "Not sure... Possibly what they want is...", "...Not 100% this is correct"
Aren't the authors of 《KBLaM》 provided this code for paper reproduction? What labels they used for training exactly? Does anyone have any ideas?

[xidulu]

The idea is to fine-tune only over the answer part, and that's what _create_labels_for_llama is about.

Does this make sense? train not just the answer but the query content as well?

I didn't know why this does not make sense, I don't see why we need to sft over the answer to learn the adapter.

And what does these comment notes means?

Because I am the one the wrote the code, these comments are left by my colleagues trying to understand the meaning of the code.

[ti250]

Thanks for clarifying Xi :) Indeed those comments were internal ones when we were going through the code and trying to understand it. We should remove these comments as we managed to convince ourselves. Thanks for raising this as well Jean-ZW :)