Skip to content

Commit 6401e31

Browse files
committed
fix: remove off-by-one in inference prompt truncation
1 parent 587d5cf commit 6401e31

1 file changed

Lines changed: 1 addition & 1 deletion

File tree

fastchat/serve/inference.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -94,7 +94,7 @@ def generate_stream(
9494
if model.config.is_encoder_decoder:
9595
max_src_len = context_len
9696
else: # truncate
97-
max_src_len = context_len - max_new_tokens - 1
97+
max_src_len = context_len - max_new_tokens
9898

9999
input_ids = input_ids[-max_src_len:]
100100
output_ids = list(input_ids)

0 commit comments

Comments
 (0)