[Tokenizer] Add tiktoken format loader for OpenAI BPE vocabularies. (… #47404

Job	Run time
setup / setup	9s
runtime_small	3m 46s
runtime :: ubuntu-24.04-arm	2m 47s
runtime :: ubuntu-24.04	1m 36s
runtime :: windows-2022	9m 55s
runtime_tracing :: ubuntu-24.04-arm :: tracy	1m 57s
runtime :: macos-14	2m 18s
runtime_tracing :: windows-2022 :: tracy	9m 20s
runtime_tracing :: ubuntu-24.04 :: tracy	1m 30s
runtime_tracing :: macos-14 :: tracy	1m 59s
runtime_tracing :: ubuntu-24.04 :: console	1m 21s
runtime_tracing :: macos-14 :: console	1m 28s
linux_x64_bazel / linux_x64_bazel	36m 41s
runtime_tracing :: windows-2022 :: console	9m 37s
runtime_tracing :: ubuntu-24.04-arm :: console	2m 7s
linux_x64_clang_debug / linux_x64_clang_debug	34m 28s
windows_x64_msvc / windows_x64_msvc	25m 12s
linux_x64_clang_asan / linux_x64_clang_asan	11m 13s
linux_x64_clang_ubsan / linux_x64_clang_ubsan	7m 9s
linux_x64_clang / linux_x64_clang	26m 13s
linux_x64_gcc	0s
linux_x64_clang_byollvm	0s
macos_x64_clang	0s
linux_arm64_clang	0s
macos_arm64_clang	0s
linux_x64_clang_tsan	0s
ci_summary / summary	1m 36s
	3h 12m 22s

Provide feedback