[ci] Skip incompatible spmd_types model integration test variants#3888
Open
wwwjn wants to merge 1 commit into
Open
[ci] Skip incompatible spmd_types model integration test variants#3888wwwjn wants to merge 1 commit into
wwwjn wants to merge 1 commit into
Conversation
The spmd_types backend variants are auto-generated for every model test, but two combinations can never pass and were red in model CI: - spmd_types + torch.compile: dynamo cannot trace spmd_types' tensor-subclass machinery (setattr of _local_type on tensors, custom all_to_all collectives), so the backend fails inside the spmd_types library regardless of typechecking. - qwen3_5 + spmd_types: qwen3_5 is not wired for spmd_types; its parallelize path resolves the legacy flattened `fsdp` mesh axis, which is absent under spmd_types (which exposes `dp_shard`/`cp`). Skip generating those variants. The 6 supported spmd_types variants and all base tests are unchanged.
meta-cla
Bot
added
the
CLA Signed
wwwjn
changed the title
wwwjn
changed the title
pianpwk
approved these changes
Jul 8, 2026
pianpwk
left a comment
pianpwk
left a comment
Contributor
There was a problem hiding this comment.
compile is unexpected, i'll follow up
tianyu-l
reviewed
Jul 8, 2026
Comment on lines
230
to
+231
| *model_tests, | ||
| *[_enable_spmd_backend(t, "spmd_types") for t in model_tests], | ||
| *[t for t in spmd_tests if t is not None], |
Contributor
There was a problem hiding this comment.
do we need to keep both? I think we could do
- compile / qwen3.5: non-spmd_types
- others: spmd_types
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Contents
The
spmd_typesbackend variants are auto-generated for every model test intests/integration_tests/models.py, but two combinations can never pass andwere red in model CI:
deepseek_v3,gpt_oss,qwen3helion):dynamo cannot trace spmd_types' tensor-subclass machinery (
setattrof_local_typeon tensors, customall_to_allcollectives), so the backendfails inside the
spmd_typeslibrary regardless of typechecking.parallelize path resolves the legacy flattened
fsdpmesh axis, which isabsent under spmd_types (which exposes
dp_shard/cp)._enable_spmd_backendnow skips those variants. The 6 supported spmd_typesvariants and all base tests are unchanged.
Tests
Reproduced and verified locally (8x H100, spmd_types installed):
deepseek_v3_fsdp+ep+compile_spmd_typesfails withtorch._dynamo.exc.Unsupported: setattr() on unsupported typeatdeepseek_v3/model.pyspmd.assert_type-> confirms spmd_types + compile isunfixable here.
Invalid mesh dim: 'fsdp').deepseek_v3_pp+fsdp+tp+ep,deepseek_v3_hsdp+ep,gpt_oss_pp+fsdp+ep+sacop,qwen3_fsdp+tp+cp_no_sp,qwen3_moe_fsdp+tp+ep_param_groups,qwen3_non_fused_qkv_fsdp+tp+cp.