Skip to content

update fsdp1/2 state dict - save replicated items on all shards - #1245

Merged
cbalioglu merged 2 commits into
facebookresearch:v0.5.0a1from
zyaoj:zyaoj/update-fsdp-state-dict
Aug 11, 2025
Merged

update fsdp1/2 state dict - save replicated items on all shards#1245
cbalioglu merged 2 commits into
facebookresearch:v0.5.0a1from
zyaoj:zyaoj/update-fsdp-state-dict

Conversation

@zyaoj

@zyaoj zyaoj commented Aug 11, 2025

Copy link
Copy Markdown
Contributor

What does this PR do? Please describe:

When using fsdp1/2, replicated items (e.g. model buffer) are only saved to rank 0 (fsdp1, fsdp2). However, when continue training from sharded model checkpoints from fsdp1/2, we don't retrieve and load those replicated items from sdp_00.pt. This PR fixes the subtle bug by retiring the small optimization of model buffer de-dup.

Does your PR introduce any breaking changes? If yes, please list them:
N/A

Check list:

  • Was the content of this PR discussed and approved via a GitHub issue? (no need for typos or documentation improvements)
  • Did you read the contributor guideline?
  • Did you make sure that your PR does only one thing instead of bundling different changes together?
  • Did you make sure to update the documentation with your changes? (if necessary)
  • Did you write any new necessary tests?
  • Did you verify new and existing tests pass locally with your changes?
  • Did you update the CHANGELOG? (no need for typos, documentation, or minor internal changes)

@zyaoj
zyaoj requested a review from cbalioglu as a code owner August 11, 2025 13:01
@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 11, 2025
@zyaoj
zyaoj requested a review from artemru August 11, 2025 20:26
Comment thread src/fairseq2/nn/data_parallel/fsdp1.py
@cbalioglu
cbalioglu merged commit cbee43e into facebookresearch:v0.5.0a1 Aug 11, 2025
13 checks passed
@zyaoj
zyaoj deleted the zyaoj/update-fsdp-state-dict branch August 12, 2025 08:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants