Skip to content

fix(mm_plugin): regularize images before processor in MiniCPMV4_6Plugin#10619

Open
ZMXJJ wants to merge 1 commit into
hiyouga:mainfrom
ZMXJJ:fix/minicpmv4_6-regularize-images
Open

fix(mm_plugin): regularize images before processor in MiniCPMV4_6Plugin#10619
ZMXJJ wants to merge 1 commit into
hiyouga:mainfrom
ZMXJJ:fix/minicpmv4_6-regularize-images

Conversation

@ZMXJJ

@ZMXJJ ZMXJJ commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

MiniCPMV4_6Plugin._get_mm_inputs feeds raw image inputs (file paths / bytes / dicts)
directly into image_processor, unlike every other plugin — including the older
MiniCPMVPlugin — which first calls _regularize_images() to load them into PIL
objects and apply image_max_pixels / image_min_pixels resizing.

This has two consequences for MiniCPM-V-4.6:

  1. image_max_pixels / image_min_pixels are silently ignored, so images are
    never resized according to the configured pixel bounds.
  2. Training/inference crashes on torchvision builds without libjpeg. With
    transformers>=5.7, the (torchvision-backed) image processor decodes path inputs
    through torchvision. On wheels built without libjpeg (e.g. the ROCm torchvision
    builds) this fails during preprocessing with:
    RuntimeError: decode_jpeg: torchvision not compiled with libjpeg support
    

Pre-loading the images to PIL via _regularize_images() makes the processor pass
them through unchanged, which both restores the resize behavior and avoids the
torchvision decode path. The video branch is intentionally left untouched, since
videos are decoded via PyAV (_regularize_videos), not torchvision.

How was this tested?

Reproduced while LoRA fine-tuning MiniCPM-V-4.6 (template: minicpm_v_4_6) on an
AMD ROCm setup (transformers==5.7.0, ROCm torchvision without libjpeg). Before this
change, llamafactory-cli train crashed in dataset preprocessing with the
decode_jpeg error above. After the change, SFT runs to completion and the trained
adapter produces correct image-grounded answers.

Fixes # (no issue)

Before submitting

MiniCPMV4_6Plugin._get_mm_inputs passed raw image inputs (paths/bytes)
straight to the image_processor, unlike every other plugin (e.g.
MiniCPMVPlugin) which first calls _regularize_images to load them into
PIL objects and apply image_max_pixels/image_min_pixels resizing.

Two consequences:
- image_max_pixels / image_min_pixels were silently ignored for v4.6.
- With transformers >= 5.7 the (torchvision-backed) image_processor
  decodes path inputs via torchvision, which fails on builds without
  libjpeg (e.g. the ROCm torchvision wheels) with:
  "decode_jpeg: torchvision not compiled with libjpeg support".

Pre-loading to PIL makes the processor pass images through unchanged,
restoring the resize behavior and avoiding the torchvision decode path.
The video branch is left untouched (videos use PyAV, not torchvision).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates src/llamafactory/data/mm_plugin.py to pre-load images into PIL objects using _regularize_images before passing them to the image processor. This prevents torchvision from decoding image paths directly, which can fail on environments like ROCm where torchvision might be built without libjpeg support. There are no review comments, and I have no feedback to provide.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant