Skip to content

Stabilize fork-sensitive nightly tests and AutoSP coverage#8172

Open
tohtana wants to merge 4 commits into
deepspeedai:masterfrom
tohtana:tohtana/fix-nightly-zero-nvme-hang
Open

Stabilize fork-sensitive nightly tests and AutoSP coverage#8172
tohtana wants to merge 4 commits into
deepspeedai:masterfrom
tohtana:tohtana/fix-nightly-zero-nvme-hang

Conversation

@tohtana

@tohtana tohtana commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

This PR fixes issues with the nightly full test:

  • Fork-sensitive tests: Run these tests in the non-forked sequential phase
    • Heavy two-rank NVMe cases contended in the forked parallel phase
    • The CPU-only FlopsProfiler formatting test could inherit initialized CUDA state and fail in a forked subprocess
  • AutoSP coverage: TestSDPANodesCompile inspected only the first FX graph produced from a Transformers model, but a compiled model can be split into multiple graphs and the first graph may not contain attention. The test now constructs an FX GraphModule with Q/K/V inputs and one torch.nn.functional.scaled_dot_product_attention call. The separate four-GPU tests still continue to use a tiny Transformers Llama model.
  • Transformers version output: Nightly tests install Transformers from Git, but the workflow incorrectly displayed the unused PyPI fallback version (4.50.0) as if it were installed. It now reports the actually installed version and Git commit, shows unused fields as N/A.

tohtana added 4 commits July 19, 2026 10:36
Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com>
Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com>
Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com>
Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com>
@tohtana
tohtana requested review from loadams and tjruwase as code owners July 24, 2026 05:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant