Add InternLM2Tokenizer support (BPE tokenizer)#1023

baijumeswani merged 1 commit intomicrosoft:mainmicrosoft/onnxruntime-extensions:mainfrom

amdrajeevp1:add-internlm2-tokenizeramdrajeevp1/onnxruntime-extensions:add-internlm2-tokenizerCopy head branch name to clipboard