fix(reasoner): preserve VideoPhy2 timeline after frame limiting - #200
Conversation
Signed-off-by: PeiPei233 <2503585731@qq.com>
There was a problem hiding this comment.
Pull request overview
Fixes VideoPhy2 video timeline/FPS reporting after applying the 32-frame cap so Qwen timestamp tokens remain aligned with the source video duration, and adds a regression test covering a long-video capped case.
Changes:
- Update
_decode_video_to_pil_framesto computeeffective_fpsfrom the final sampled-frame count relative to the original video duration. - Add a unit test that simulates a 900-frame, 30 FPS video capped to 32 frames and asserts the corrected
effective_fps. - Minor formatting adjustment to a
TypeErrorraise.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| cosmos_framework/configs/base/reasoner/experiment/videophy2_dataflow_roles.py | Computes effective_fps based on final indices to preserve the correct timeline after frame limiting. |
| cosmos_framework/configs/base/reasoner/experiment/videophy2_dataflow_roles_test.py | Adds regression coverage for capped-frame videos and validates the updated FPS calculation. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| decoders_module = ModuleType("torchcodec.decoders") | ||
| decoders_module.VideoDecoder = FakeVideoDecoder # type: ignore[attr-defined] | ||
| torchcodec_module = ModuleType("torchcodec") | ||
| torchcodec_module.decoders = decoders_module # type: ignore[attr-defined] | ||
| monkeypatch.setitem(sys.modules, "torchcodec", torchcodec_module) | ||
| monkeypatch.setitem(sys.modules, "torchcodec.decoders", decoders_module) |
pengcuo
left a comment
There was a problem hiding this comment.
Thanks for the fix! Verified the logic: the old source_fps / stride always reported ~2.0 fps regardless of the 32-frame cap, compressing the perceived timeline of any video longer than 16s (a 30s clip showed up as 16s downstream, where fps feeds the Qwen3-VL video_metadata timestamps and the Nemotron bridge duration). The new len(indices) / total_frames * source_fps matches the qwen_vl_utils / i4 video_decoder_qwen convention exactly, and degenerates to the old value when the cap is not hit — confirmed both cases by running the decode against a fake 900-frame/30fps and 300-frame/30fps decoder. LGTM.
Problem
VideoPhy2 first samples videos toward 2 FPS, then caps clips with more than 32 sampled frames. After that second downsampling step, the processor still reports
source_fps / stride, so Qwen timestamp tokens see a compressed timeline for long videos.Fix
Compute the effective FPS from the final number of sampled frames:
This matches
_decode_reasoner_videoand keeps timestamp tokens aligned with the source video duration after the 32-frame cap. A regression test covers a 900-frame, 30 FPS video capped to 32 frames.Testing
uvx ruff==0.12.7 check <changed files>uvx ruff==0.12.7 format --check <changed files>uvx pre-commit run -c ci/.pre-commit-config-base.yaml --files <changed files>pytest cosmos_framework/configs/base/reasoner/experiment/videophy2_dataflow_roles_test.py(1 passed)