Data Processing Steps

[xiami2019]

Hi, nice work!

I would like to try SFT. However, the data processing pipeline appears to be incomplete. The files referenced in sft_stage_1.sh: train_s12w24_with_seeks.jsonl, train_livecc_with_seeks.jsonl, valid_s12w24_with_seeks.jsonl, and valid_livecc_with_seeks.jsonl do not exist in the downloaded dataset. Could you please advise how I can obtain these files?

In addition, I would like to understand why train_s12w24_with_seeks.jsonl and valid_s12w24_with_seeks.jsonl appear twice.

[xiami2019]

Hi, I think there’s also a lack of annotations other than those from the LiveCC dataset.

[xrorrim]

Sorry about that! A previous large upload triggered a bug on the Hugging Face repo and some training files were missing. This has now been fixed—please pull the latest version of Inf-Stream-Train for all training files.
Please let us know if the issue persists. I’ll follow up promptly. Thanks!