Skip to content

Commit f13ff6a

Browse files
authored
Update tokenizer dictionary entry name
1 parent 31ee9fe commit f13ff6a

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

operators/tokenizer/tokenizer_jsconfig.hpp

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ enum class TokenType {
1515
};
1616

1717
constexpr std::pair<const char*, TokenType> kTokenizerDict[] = {
18-
{"PreTrainedTokenizerFast", TokenType::kBPE},
18+
{"PreTrainedTokenizer", TokenType::kBPE},
1919
{"CLIPTokenizer", TokenType::kBPE},
2020
{"WhisperTokenizer", TokenType::kBPE},
2121
{"GemmaTokenizer", TokenType::kBPE},

0 commit comments

Comments
 (0)