Commit cff6f1e
Add gpt-5.3 support in TiktokenTokenizer
Co-authored-by: stephentoub <2642209+stephentoub@users.noreply.github.com>1 parent d46475b commit cff6f1e
File tree
2 files changed
+6
-0
lines changed- src/Microsoft.ML.Tokenizers/Model
- test/Microsoft.ML.Tokenizers.Tests
2 files changed
+6
-0
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1044 | 1044 | | |
1045 | 1045 | | |
1046 | 1046 | | |
| 1047 | + | |
1047 | 1048 | | |
1048 | 1049 | | |
1049 | 1050 | | |
| |||
1073 | 1074 | | |
1074 | 1075 | | |
1075 | 1076 | | |
| 1077 | + | |
1076 | 1078 | | |
1077 | 1079 | | |
1078 | 1080 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
38 | 38 | | |
39 | 39 | | |
40 | 40 | | |
| 41 | + | |
41 | 42 | | |
42 | 43 | | |
43 | 44 | | |
| |||
421 | 422 | | |
422 | 423 | | |
423 | 424 | | |
| 425 | + | |
| 426 | + | |
424 | 427 | | |
425 | 428 | | |
426 | 429 | | |
| |||
541 | 544 | | |
542 | 545 | | |
543 | 546 | | |
| 547 | + | |
544 | 548 | | |
545 | 549 | | |
546 | 550 | | |
| |||
0 commit comments