Skip to content

[qwen3.5] fix DTensor TP vision position caches#3899

Open
pianpwk wants to merge 1 commit into
gh/pianpwk/63/basefrom
gh/pianpwk/63/head
Open

[qwen3.5] fix DTensor TP vision position caches#3899
pianpwk wants to merge 1 commit into
gh/pianpwk/63/basefrom
gh/pianpwk/63/head

Conversation

@pianpwk

@pianpwk pianpwk commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Stack from ghstack (oldest at bottom):

Main TP was failing with FLA_TILELANG=0 MODULE=qwen3_5 CONFIG=qwen35_debugmodel NGPU=2 ./run_train.sh --parallelism.tensor_parallel_degree 2 --parallelism.data_parallel_shard_degree 1 --debug.seed 42 --debug.deterministic --training.steps 1

[rank0]:[rank0]:   File "/data/users/pianpwk/full_spmd_types_torchtitan/tt4_spmd_types/torchtitan/torchtitan/models/qwen3_5/vision_encoder.py", line 91, in _compute_learned_pos_embeds
[rank0]:[rank0]:     pos_hw = F.interpolate(
[rank0]:[rank0]:              ^^^^^^^^^^^^^^
[rank0]:[rank0]:   File "/data/users/pianpwk/full_spmd_types_torchtitan/pytorch/torch/nn/functional.py", line 5273, in interpolate
[rank0]:[rank0]:     )._upsample_linear_vec(input, output_size, align_corners, scale_factors)
[rank0]:[rank0]:       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:[rank0]:   File "/data/users/pianpwk/full_spmd_types_torchtitan/pytorch/torch/_decomp/decompositions.py", line 4507, in _upsample_linear_vec
[rank0]:[rank0]:     return _upsample_linear(input, osize, align_corners, scales)
[rank0]:[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:[rank0]:   File "/data/users/pianpwk/full_spmd_types_torchtitan/pytorch/torch/_decomp/decompositions.py", line 91, in inner
[rank0]:[rank0]:     r = f(*tree_map(increase_prec, args), **tree_map(increase_prec, kwargs))
[rank0]:[rank0]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:[rank0]:   File "/data/users/pianpwk/full_spmd_types_torchtitan/pytorch/torch/_decomp/decompositions.py", line 4627, in _upsample_linear
[rank0]:[rank0]:     v = aten._unsafe_index(input, idx)
[rank0]:[rank0]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:[rank0]:   File "/data/users/pianpwk/full_spmd_types_torchtitan/pytorch/torch/_ops.py", line 1350, in __call__
[rank0]:[rank0]:     return self._op(*args, **kwargs)
[rank0]:[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:[rank0]: RuntimeError: aten._unsafe_index.Tensor got mixed torch.Tensor and DTensor, need to convert all torch.Tensor to DTensor before calling distributed operators!
  • DTensor decomposition goes to unsafe_index which creates local plain tensors - fixes by local-mapping interpolate
  • upcoming spmd_types PR would parallelize inv_freq, get ahead of this by also wrapping positions as DTensors - previously that portion was done in plain tensor

[ghstack-poisoned]
@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Jul 10, 2026
@pianpwk pianpwk marked this pull request as ready for review July 10, 2026 18:09

@tianyu-l tianyu-l left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pls make sure numerics doesn't change

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ciflow/8gpu CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants