what's the tokenizer.model?

[VincentZ-2020]

Hi there!
How to get ' tokenizer.model' in 'tools/prepare_c4_megatron_llama3.py'?

[VainF]

Hi @VincentZ-2020, We don't need the tokenizer.model. There is a script for llama-3 pre-tokenization: https://github.com/NVlabs/MaskLLM/blob/main/scripts/data/prepare_c4_megatron_llama3.sh.