Data cleaning, preprocessing, transformation, and tokenization

[sxaxmz]

Hi, great work. I have a few questions as it seems that it was unclear how the data was preprocessed, cleaned, transformed, and tokenized for the purpose of fine-tuning, could you please elaborate or provide more information. Thanks.