[spmd_types] qwen3.5 enablement#3895
Draft
pianpwk wants to merge 4 commits into
Draft
Conversation
meta-cla
Bot
added
the
CLA Signed
pianpwk
added a commit
that referenced
this pull request
Jul 10, 2026
Enable Qwen3.5 spmd_types parallelization path and sharding contracts for dense, vision, GatedDeltaNet, and MoE paths. Status: - Passing 1-step typechecking smokes: dense FSDP, TP, FSDP+TP, HSDP; MoE EP off, EP on, FSDP+TP+EP; varlen attention. - Passing 10-step bitwise default-vs-spmd_types loss and grad_norm: dense FSDP, dense HSDP, MoE FSDP with local dispatcher, MoE FSDP with standard A2A EP. - Blocked 10-step bitwise for TP-including default-backend shapes: default backend fails in Qwen3.5 vision rotary with mixed torch.Tensor/DTensor at torch.outer(seq, self.inv_freq). - Known remaining gaps: PP not tested; larger flavors not tested; flex_flash blocked by missing CUTE flash attention package; torch_native GatedDeltaNet fails kernel output type; fla_fused_recurrent backward is not implemented upstream. ghstack-source-id: 2d10476 Pull Request resolved: #3895
pianpwk
added a commit
that referenced
this pull request
Jul 10, 2026
Enable Qwen3.5 input annotations, local multimodal regions, sharding configs, and torch_native GDN coverage for the spmd_types backend. Status: FSDP=2 matched default backend bitwise over 10 debug steps. TP=2 and FSDP=2+TP=2 run, but are not bitwise against default backend; grad_norm diverges from step 1. TP=2 typechecking passes for qwen35_debugmodel and qwen35_debugmodel_gdn_torch_native. ghstack-source-id: 9115a56 Pull Request resolved: #3895
pianpwk
added a commit
that referenced
this pull request
Jul 10, 2026
Enable Qwen3.5 input annotations, local multimodal regions, sharding configs, and torch_native GDN coverage for the spmd_types backend. Status: FSDP=2 matched default backend bitwise over 10 debug steps. TP=2 and FSDP=2+TP=2 run, but are not bitwise against default backend; grad_norm diverges from step 1. TP=2 typechecking passes for qwen35_debugmodel and qwen35_debugmodel_gdn_torch_native. ghstack-source-id: 84201c7 Pull Request resolved: #3895
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack from ghstack (oldest at bottom):
Enable Qwen3.5 spmd_types parallelization path and sharding contracts for dense, vision, GatedDeltaNet, and MoE paths.
Status:
Passing 1-step typechecking smokes: dense FSDP, TP, FSDP+TP, HSDP; MoE EP off, EP on, FSDP+TP+EP; varlen attention.
Passing 10-step bitwise default-vs-spmd_types loss and grad_norm: dense FSDP, dense HSDP, MoE FSDP with local dispatcher, MoE FSDP with standard A2A EP.
Blocked 10-step bitwise for TP-including default-backend shapes: default backend fails in Qwen3.5 vision rotary with mixed torch.Tensor/DTensor at torch.outer(seq, self.inv_freq).
Known remaining gaps: PP not tested; larger flavors not tested; flex_flash blocked by missing CUTE flash attention package; torch_native GatedDeltaNet fails kernel output type; fla_fused_recurrent backward is not implemented upstream.