how to process prefix token again in decoding?

[ArtyZe]

hello，if prefix token produced again in decoding after prefix, how to process, just get kv from prefix kvcache？

[ice-tong]

I have the same question.

[lonleyodd]

@ArtyZe @ChenMnZ I noticed many models identify token [BOS] as prefix token according prefixquant，and pass it to kvcache。However，during prefill process，tokenizer will add a new [BOS] token at begining of a text. this will result 2 [BOS] token in generation process like follow:

but the second [BOS] token will result in outlier activation and cause scale not correct for other token during quantization like follow.

so I guess how about remove second [BOS] token in quantizaton?