enable Context Parallel #592
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[ghstack-poisoned]
XilunWu
added a commit
that referenced
this pull request
Sep 30, 2024
ghstack-source-id: b76e0d183826dad8c4c76426fe62abaf9ad43f2f Pull Request resolved: #592
facebook-github-bot
added
the
CLA Signed
[ghstack-poisoned]
XilunWu
added a commit
that referenced
this pull request
Sep 30, 2024
ghstack-source-id: 90f1bde378561c9bd1dee3ac82990f9d91ba59ab Pull Request resolved: #592
fegin
reviewed
Oct 1, 2024
| # (use 2x max sequence length to be safe) | ||
| self.model_args.max_seq_len * 2, | ||
| # Note: removed the 2x relaxing in CP enablement | ||
| self.model_args.max_seq_len, |
There was a problem hiding this comment.
cc., @tianyu-l Want to understand is this okay?
For a general use case, we can also expand the CP to support stride-like feature.
There was a problem hiding this comment.
Could you please elaborate a bit on why this change was needed by CP?
fegin
reviewed
Oct 1, 2024
| if parallel_dims.cp_enabled: | ||
| dp_dim_names = dp_mesh.mesh_dim_names | ||
| assert isinstance(dp_dim_names, Tuple) | ||
| dp_mesh = world_mesh[(*dp_dim_names, "cp")]._flatten() |
There was a problem hiding this comment.
I remember we want to initialize all the PG in the very beginning. Can we move this to parallel_dims.py and use mesh_dim_name to rename it to dp_shard_cp?
Closed
[ghstack-poisoned]
XilunWu
added a commit
that referenced
this pull request
Oct 3, 2024
ghstack-source-id: f9bc24ff92c0abce98dc3a0f847fc874fa77788c Pull Request resolved: #592
[ghstack-poisoned]
XilunWu
added a commit
that referenced
this pull request
Oct 3, 2024
ghstack-source-id: 51288d0a142c839291d6035e6dddcc915e5e5a08 Pull Request resolved: #592
[ghstack-poisoned]
XilunWu
added a commit
that referenced
this pull request
Oct 4, 2024
ghstack-source-id: 6126585b13e49131e8b2d9e05a5ef1f736a0c4d9 Pull Request resolved: #592
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Stack from ghstack (oldest at bottom):