RL Training RuntimeError in vLLM rollout when videos have different frame lengths

[JoeYangRL]

Hi, thanks for the great work on this project.
While reproducing RL training, we encountered a runtime error during the vllm.generate() stage after a tool call.

Description

When the tool returns videos with different numbers of frames, the preprocessing step inside the Qwen2-VL video processor fails due to a torch.stack size mismatch.

It appears that group_videos_by_shape only groups videos by (H, W) but later stacks tensors that may have different temporal lengths T.

This happens when multiple videos with the same resolution but different frame counts are passed in the same batch.

sample info

videos_count=2 videos=[{'i': 0, 'shape': [76, 3, 364, 644], 'dtype': 'torch.float32', 'device': 'cpu'}, {'i': 1, 'shape': [12, 3
, 364, 644], 'dtype': 'torch.float32', 'device': 'cpu'}]

Error Log

File "/data/anaconda/anaconda3/envs/verl/lib/python3.11/site-packages/transformers/models/qwen2_5_vl/processing_qwen2_5_vl.py", line 152, in __call__
    videos_inputs = self.video_processor(videos=videos, **output_kwargs["videos_kwargs"])
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/anaconda/anaconda3/envs/verl/lib/python3.11/site-packages/transformers/video_processing_utils.py", line 197, in __call__
    return self.preprocess(videos, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/anaconda/anaconda3/envs/verl/lib/python3.11/site-packages/transformers/video_processing_utils.py", line 278, in preprocess
    return self._preprocess(videos=videos, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/anaconda/anaconda3/envs/verl/lib/python3.11/site-packages/transformers/models/qwen2_vl/video_processing_qwen2_vl.py", line 133, in _preprocess
    grouped_videos, grouped_videos_index = group_videos_by_shape(videos)
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/anaconda/anaconda3/envs/verl/lib/python3.11/site-packages/transformers/video_utils.py", line 704, in group_videos_by_shape
    grouped_videos = {shape: torch.stack(videos, dim=0) for shape, videos in grouped_videos.items()}
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/anaconda/anaconda3/envs/verl/lib/python3.11/site-packages/transformers/video_utils.py", line 704, in <dictcomp>
    grouped_videos = {shape: torch.stack(videos, dim=0) for shape, videos in grouped_videos.items()}
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: stack expects each tensor to be equal size, but got [76, 3, 364, 644] at entry 0 and [12, 3, 364, 644] at entry 1

[ZZQ987]

Have you tried training with the Qwen2.5-VL model?
Our experiments were conducted based on Qwen2.5-VL, and we haven’t tested whether the Qwen2-VL model can be trained normally.

[JoeYangRL]

I am training with Qwen2.5-VL-7B-Instruct. qwen2.5-vl series share the same video processor with qwen2-vl series.

[LanXingXuan]

Hi, thank you for providing such a detailed report.

This issue stems from using an older version of the transformers library (specifically v4.52.0). In this version, the group_videos_by_shape function groups videos using only the spatial resolution (H, W) as the key. In more recent releases of the transformers library, this grouping key has been updated to include the temporal dimension: (T, H, W).

To resolve this, you have two options:

• Manually patch the library: You can manually modify the transformers source code in your current environment so that the grouping key uses (T, H, W). It should run normally after this adjustment.

• Upgrade transformers: Alternatively, you can try upgrading your transformers version. However, please note that doing so might introduce additional environment conflicts or dependency issues.

If you encounter any further issues or have any questions, please feel free to reach out!

[JoeYangRL]

Thank you for the quick response!
Can you specify the transformers version you used when training? Or did you fix the bug manually?
I installed RL environment with the requirements.txt you provided and it's written transformers==4.52.0:

Video-o3/RL/requirements.txt

Line 291 in 59e67cf

transformers==4.52.0

[LanXingXuan]

In our previous pipeline, we manually patched the transformers library to ensure the grouping key supported (T, H, W).

We plan to provide updated, stable environment dependencies in our upcoming releases. This will allow the code to run smoothly out-of-the-box without requiring any manual modifications. Stay tuned!