Skip to content

Commit 676cebe

Browse files
committed
[graph_trainer] Add precompile integration tests to CI
Precompile integration tests (run_precompile_tests.py) were never run in CI, which allowed the bucketing + CooR incompatibility from #2934 to go undetected. Add them to both CI workflows: - A10 workflow: Llama3 aot_fx_trace precompile test (FSDP+TP) - H100 workflow: DSv3 aot_fx_trace precompile test (FSDP+TP+EP) Also fix the DSv3 precompile test config from debugmodel to debugmodel_ep — EP tests require AllToAllTokenDispatcher which is only created when moe_comm_backend="standard". ghstack-source-id: 35a8421 Pull-Request: #3043
1 parent 5e67901 commit 676cebe

2 files changed

Lines changed: 6 additions & 0 deletions

File tree

.github/workflows/integration_test_8gpu_graph_trainer.yaml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -80,6 +80,9 @@ jobs:
8080
# Run precompile unit tests
8181
pytest torchtitan/experiments/graph_trainer/tests/test_precompile.py -v
8282
83+
# Run precompile integration tests (Llama3 only; DSv3 runs in H100 workflow)
84+
python -m torchtitan.experiments.graph_trainer.tests.run_precompile_tests $RUNNER_TEMP/artifacts-to-be-uploaded/precompile --ngpu 8 --test_name aot_fx_trace_llama3_precompile_fsdp_tp
85+
8386
# Run bitwise deterministic and SAC peak-memory guardrail tests
8487
pytest torchtitan/experiments/graph_trainer/tests/test_bitwise_deterministic.py -v
8588
pytest torchtitan/experiments/graph_trainer/tests/test_sac_peak_memory.py -v

.github/workflows/integration_test_8gpu_graph_trainer_h100.yaml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,9 @@ jobs:
6666
# Run the MoE numerics tests
6767
NCCL_NVLS_ENABLE=0 pytest torchtitan/experiments/graph_trainer/tests/test_numerics.py::TestGraphTrainerNumerics -v -k "moe"
6868
69+
# Run precompile integration tests (DSv3 with EP; Llama3 runs in default workflow)
70+
NCCL_NVLS_ENABLE=0 python -m torchtitan.experiments.graph_trainer.tests.run_precompile_tests $RUNNER_TEMP/artifacts-to-be-uploaded/precompile --ngpu 8 --test_name aot_fx_trace_deepseek_v3_precompile_fsdp_tp_ep
71+
6972
# Run bitwise deterministic and SAC peak-memory guardrail tests
7073
pytest torchtitan/experiments/graph_trainer/tests/test_bitwise_deterministic.py -v
7174
pytest torchtitan/experiments/graph_trainer/tests/test_sac_peak_memory.py -v

0 commit comments

Comments
 (0)