Commit d2cb00e
committed
Add InternLM2Tokenizer support (BPE tokenizer)
InternLM2 models use the same BPE/LLaMA tokenizer format as Llama.
This registers InternLM2Tokenizer so models exported with
tokenizer_class: InternLM2Tokenizer in tokenizer_config.json
are recognized at runtime.
Ref: https://huggingface.co/internlm/internlm2-1_8b1 parent 7a21a38 commit d2cb00e
1 file changed
+1
-0
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
| 24 | + | |
24 | 25 | | |
25 | 26 | | |
26 | 27 | | |
| |||
0 commit comments