Skip to content

Commit 80da6a5

Browse files
authored
Add token 128011 to the Llama tokenizer (#2428)
1 parent 67a8706 commit 80da6a5

File tree

1 file changed

+2
-1
lines changed

1 file changed

+2
-1
lines changed

torchtune/models/llama3/_tokenizer.py

+2-1
Original file line numberDiff line numberDiff line change
@@ -31,9 +31,10 @@
3131
"<|python_tag|>": 128010,
3232
"<|image|>": 128256,
3333
"<|video|>": 128012,
34+
"<|reserved_special_token_244|>": 128011,
3435
}
3536

36-
NUM_RESERVED_SPECIAL_TOKENS = 256
37+
NUM_RESERVED_SPECIAL_TOKENS = 257
3738

3839
RESERVED_TOKENS = {
3940
f"<|reserved_special_token_{2 + i}|>": 128013 + i

0 commit comments

Comments
 (0)