Check for end-of-turn tokens in tokenizer config top-level - #155

Merged

Lewis300 merged 3 commits into

Lewis300:extra-eot-check

Aug 7, 2026

Commits on Aug 5, 2026

Also look for end of turn tokens in the top level of the tokenizer_config.json, some models have them there.
Lewis300
committed
8becce2

Commits on Aug 6, 2026

Add unit tests for additionalStopTokens func
Lewis300
committed
6f9b6b8
Merge branch 'main' into extra-eot-check
Lewis300
authored
c6b38f3