[Feature] Sample packing for continued pretrain #4046
Description
Continued pretraining is probably the part where sample packing is essential.
The documents are very different in length, from 8k to 42k tokens and it's disastrously wasteful to pretrain them without packing.
Unfortunately, in the documentation it's said that CCE is not supported for CPT and the training auto-disables the backing because UNSLOTH_RETURN_LOGITS is passed as True.
(With the said reason of Unsloth: Sample packing skipped (custom data collator detected). when no custom collator was activated)
Is it possible for you to make sample packing compatible with continuous pretrain?
It's very frustrating 🥴