data_prep format

[tsebaka]

Hello!
question:
in data_prep if I use --concat_tokens k, its divide into chunks of k tokens my all data, but if I want to just take sample from my data and truncate by max_tokens or add pad tokens to max_tokens (for each sample from my data)? How it can be done in llm-foundry?

--concat_tokens 2
["some", "text"] -> ["so", "me", "te", "xt"]
I want:
max_len=3
["some", "text", "h"] -> ["som", "tex", "h"]

I know in pretrain LLMs it's useless but in sft I also don't find this data_prep in llm-foundry