Idefics3/SmolVLM: num_patches=0 for non-tiled images mis-sizes pixel_values while the prompt still reserves image_seq_len placeholder tokens

[ErenAta16]

Description

For Idefics3/SmolVLM, when an image doesn't get tiled (do_image_splitting=False, or splitting enabled but the image is already small enough that no split is needed), Idefics3ProcessingInfo computes num_patches=0 for that image and uses it to size the image's pixel_values/pixel_attention_mask slice in the multimodal field config, while the prompt still correctly reserves image_seq_len placeholder tokens for it. The zero-size slice doesn't match the real, non-empty pixel data HF's own preprocessing produces for that image, which should surface as a _merge_multimodal_embeddings token-count mismatch (0 actual vs image_seq_len expected) at inference time.

Where the numbers diverge

get_number_of_image_patches in HF's Idefics3ImageProcessor (checked directly against transformers==5.5.3, the minimum vLLM currently requires per requirements/common.txt):

# transformers/models/idefics3/image_processing_idefics3.py, v5.5.3
def get_number_of_image_patches(self, height, width, images_kwargs):
    ...
    num_patches = num_rows = num_cols = 0
    if do_image_splitting:
        ...
        if resized_height > max_height or resized_width > max_width:
            num_rows = math.ceil(resized_height / max_height)
            num_cols = math.ceil(resized_width / max_width)
            num_patches = num_rows * num_cols + 1
    return num_patches, num_rows, num_cols

num_patches stays 0 whenever do_image_splitting=False, or whenever splitting is enabled but the image doesn't actually exceed max_image_size. This is consistent with the real preprocessing path, split_images() in the same class, which returns num_splits_h = num_splits_w = 0 in exactly the same "no split needed" case, but still returns frames = [image], one real frame, the global image itself.

vllm/model_executor/models/idefics3.py uses this num_patches value in two places that end up disagreeing with each other:

1. Idefics3MultiModalProcessor._call_hf_processor (line ~331-340):

   num_patches = [
       self.info.get_num_patches(image_width=size.width, image_height=size.height,
                                  processor=hf_processor, mm_kwargs=mm_kwargs)
       for size in image_sizes
   ]
   processed_outputs["num_patches"] = torch.tensor(num_patches)

   This directly reuses HF's num_patches (0 in the no-split case) as size_per_item for MultiModalFieldConfig.flat_from_sizes("image", num_patches) in _get_mm_fields_config (line ~356-359), which slices the flattened pixel_values/pixel_attention_mask tensor per image using exactly these sizes. A 0 here means vLLM will slice a zero-row chunk for that image, even though HF's actual preprocessing produced one real frame of pixel data for it.

2. Idefics3ProcessingInfo.get_image_repl (line ~204-247) builds the prompt text for the same image independently, and has its own explicit special case for this:

   _, grid_h, grid_w = self._get_image_feature_grid_size(...)
   if grid_w == 0 and grid_h == 0:
       return global_img_placeholder + fake_image_token   # image_seq_len <image> tokens

   So the prompt correctly ends up with image_seq_len placeholder tokens for this image, that part is right, it's specifically the pixel_values sizing in (1) that doesn't have an equivalent special case.

Why this should be user-visible

MultiModalFieldConfig.flat_from_sizes slices a flattened tensor using size_per_item as consecutive chunk sizes (vllm/multimodal/inputs.py, flat_from_sizes). A 0 entry for an image whose actual pixel data isn't empty means either that image's real pixel rows get silently reattributed to a neighboring image's slice in the same batch (in a multi-image request mixing split and non-split images), or, in the single-image case, the vision tower ends up running on a 0-row input and producing 0 embeddings for that image. Either way, is_multimodal.sum() (from the correctly-sized prompt, image_seq_len positions) won't match len(mm_embeds_flat) (0, or misattributed from another image) when _merge_multimodal_embeddings runs (vllm/model_executor/models/utils.py, ~line 545-550), which is exactly the shape-mismatch RuntimeError that function's except branch is written to catch and report.

Reproduction

Any Idefics3/SmolVLM checkpoint with do_image_splitting=False set (either via --mm-processor-kwargs '{"do_image_splitting": false}', or a preprocessor_config.json that sets that default), or any image small enough that splitting doesn't trigger even with the default do_image_splitting=True, should hit this on the very first request containing that image. smolvlm.py inherits this class unmodified, so it's affected the same way.

Notes

tests/models/multimodal/processing/test_smolvlm.py doesn't appear to cover the no-split path (this is inferred from the file layout in the repo, I didn't run the suite), which is consistent with this not having been caught yet, dummy/test images there are sized to force splitting.

Happy to help narrow down the exact runtime error text if useful, wanted to get the root-cause diff (the num_patches=0-but-real-pixel-data-exists mismatch, specifically in the _get_mm_fields_config sizing, not the prompt string which is already correct) written up precisely first.