SFT sample packing #3566
Open
SFT sample packing #3566
+2,014
−474
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
djsaunde
force-pushed
the
auto-packing
branch
2 times, most recently
from
072de80 to
efe4424
Compare
djsaunde
force-pushed
the
auto-packing
branch
3 times, most recently
from
f352814 to
9cd0b26
Compare
djsaunde
force-pushed
the
auto-packing
branch
from
8f06935 to
50e5104
Compare
Collaborator
Author
|
I tested all the SFT notebooks on the main page README. I'd like to test a few more notebooks, including ones that we don't expect to use packing (multimodal training, RL training, ...). |
Collaborator
Author
|
Okay, I ran through all the notebooks displayed in the README. As expected, non of the non-text-only SFT notebooks utilized sample packing, and none of them hit any errors when switching to this branch. |
for more information, see https://pre-commit.ci
djsaunde
force-pushed
the
auto-packing
branch
from
585e26f to
e498912
Compare
djsaunde
force-pushed
the
auto-packing
branch
from
e498912 to
6c90169
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR patches the SFTTrainer class' constructor to auto-enable sample packing when applicable (i.e., when the user is not training a multi-modal model or passing in a custom data collator). Sample packing is a good default in SFT since we reduce the amount of zero-padding we see and increase the training token/s throughput as a result.
Followup to #3525; can merge after that PR is.I closed that PR, I think we should just merge this one.Needs to be extensively tested so we don't break any existing notebooks / scripts.