fix(Tokenizers): avoid quadratic regex matching for added tokens without whitespace padding - #386

fix(Tokenizers): avoid quadratic regex matching for added tokens without whitespace padding#386

shubhransh-gupta wants to merge 1 commit into

huggingface:mainhuggingface/swift-transformers:mainfrom

shubhransh-gupta:fix/added-tokens-regex-perfshubhransh-gupta/swift-transformers:fix/added-tokens-regex-perfCopy head branch name to clipboard