Raise error when minibatch is used in SPMD dataloading and per host b… #8205
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…atch size is not divisble by data mesh
jonb377
reviewed
Oct 2, 2024
|
ok let me resolve the merge conflicts |
|
@jonb377 can you take another look? |
jonb377
approved these changes
Oct 4, 2024
| if sharding and tensor.dim() > 0 and (tensor.size()[0] % | ||
| local_runtime_device_count) != 0: | ||
| raise RuntimeError( | ||
| "When minibatch is configured, batch dimension of the tensor " + |
There was a problem hiding this comment.
Maybe clarify the per-host batch size must be divisible...? These concepts are kind of confusing since we're mapping host-level sharding into device-level and representing it as a global tensor, so there's three different batch dimensions to consider.
There was a problem hiding this comment.
Thanks, let me update in a follow up pr.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
…atch size is not divisble by data mesh
When
minibatchis used in sharding spec, each device is loading it own part of the data and reassemble it during the runtime.Imagenine a case the mesh is (4,1) and the batch size we used is 3, each device will see a data like
and runtime will reassemble a incorrect final data.
If
mini batchis not used it is OK, because each device will seeand we will drop padding at the end.