fix(mm_plugin): regularize images before processor in MiniCPMV4_6Plugin#10619
Open
ZMXJJ wants to merge 1 commit into
Open
fix(mm_plugin): regularize images before processor in MiniCPMV4_6Plugin#10619ZMXJJ wants to merge 1 commit into
ZMXJJ wants to merge 1 commit into
Conversation
MiniCPMV4_6Plugin._get_mm_inputs passed raw image inputs (paths/bytes) straight to the image_processor, unlike every other plugin (e.g. MiniCPMVPlugin) which first calls _regularize_images to load them into PIL objects and apply image_max_pixels/image_min_pixels resizing. Two consequences: - image_max_pixels / image_min_pixels were silently ignored for v4.6. - With transformers >= 5.7 the (torchvision-backed) image_processor decodes path inputs via torchvision, which fails on builds without libjpeg (e.g. the ROCm torchvision wheels) with: "decode_jpeg: torchvision not compiled with libjpeg support". Pre-loading to PIL makes the processor pass images through unchanged, restoring the resize behavior and avoiding the torchvision decode path. The video branch is left untouched (videos use PyAV, not torchvision). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Code Review
This pull request updates src/llamafactory/data/mm_plugin.py to pre-load images into PIL objects using _regularize_images before passing them to the image processor. This prevents torchvision from decoding image paths directly, which can fail on environments like ROCm where torchvision might be built without libjpeg support. There are no review comments, and I have no feedback to provide.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
MiniCPMV4_6Plugin._get_mm_inputsfeeds raw image inputs (file paths / bytes / dicts)directly into
image_processor, unlike every other plugin — including the olderMiniCPMVPlugin— which first calls_regularize_images()to load them into PILobjects and apply
image_max_pixels/image_min_pixelsresizing.This has two consequences for MiniCPM-V-4.6:
image_max_pixels/image_min_pixelsare silently ignored, so images arenever resized according to the configured pixel bounds.
transformers>=5.7, the (torchvision-backed) image processor decodes path inputsthrough torchvision. On wheels built without libjpeg (e.g. the ROCm torchvision
builds) this fails during preprocessing with:
Pre-loading the images to PIL via
_regularize_images()makes the processor passthem through unchanged, which both restores the resize behavior and avoids the
torchvision decode path. The video branch is intentionally left untouched, since
videos are decoded via PyAV (
_regularize_videos), not torchvision.How was this tested?
Reproduced while LoRA fine-tuning MiniCPM-V-4.6 (
template: minicpm_v_4_6) on anAMD ROCm setup (
transformers==5.7.0, ROCm torchvision without libjpeg). Before thischange,
llamafactory-cli traincrashed in dataset preprocessing with thedecode_jpegerror above. After the change, SFT runs to completion and the trainedadapter produces correct image-grounded answers.
Fixes # (no issue)
Before submitting