Skip to content

Add LFM2.5-VL-3B benchmark handling - #1663

Open
rshube wants to merge 1 commit into
open-compass:mainfrom
Liquid4All:wip/lfm25-vl-benchmark-handling
Open

Add LFM2.5-VL-3B benchmark handling#1663
rshube wants to merge 1 commit into
open-compass:mainfrom
Liquid4All:wip/lfm25-vl-benchmark-handling

Conversation

@rshube

@rshube rshube commented Aug 28, 2026

Copy link
Copy Markdown

Summary

  • Add benchmark-specific prompting and ScreenSpot response normalization for LFM2.5-VL-3B.
  • Preserve existing LFM2-VL registrations without custom prompting.
  • Preserve chat roles and interleaved inputs, decode completion tokens only, and use deterministic generation.
  • Fix MM-IFEval verifier lookup and localized image paths.

Validation

  • Reproduced the affected public benchmark scores with the LFM2.5-VL-3B checkpoint.
  • Confirmed raw predictions remain identical after the model processor configuration update.
  • The full pre-commit hook suite passes across all files.

@rshube

rshube commented Aug 28, 2026

Copy link
Copy Markdown
Author

Checkpoint reproduction for affected benchmarks

Only benchmarks with dataset-specific prompting or response handling in this PR are listed. Published values are from the release table; scores are normalized to 0–100.

Benchmark Published Reproduced Delta Handling
MME 73.1 73.89 +0.79 Category-specific answer formats
SimpleVQA 35.4 35.67 +0.27 No appended instruction
MM-IFEval 60.6 59.04 -1.56 Preserve image-first ordering; judge-dependent score
LogicVista 37.4 37.58 +0.18 No appended instruction
MathVista (mini) 68.5 68.50 0.00 Step-by-step reasoning with boxed final answer
MMMU (validation) 48.4 48.22 -0.18 No appended instruction
OCRBench v1 84.2 84.10 -0.10 Concise-answer instruction
BLINK 61.5 61.55 +0.05 No redundant instruction
MUIRBench 58.3 58.27 -0.03 Numbered images and explicit final option format
HallusionBench 47.2 47.25 +0.05 Yes/no instruction; judge-dependent score
POPE 88.7 88.75 +0.05 No appended instruction
RefCOCO average 87.9 88.86 +0.96 Strict normalized bounding-box JSON
ScreenSpot-v2 Desktop 78.7 78.44 -0.26 Click-target prompt and response normalization
ScreenSpot-v2 Mobile 81.2 80.64 -0.56 Click-target prompt and response normalization
ScreenSpot-v2 Web 82.2 82.15 -0.05 Click-target prompt and response normalization

@rshube
rshube force-pushed the wip/lfm25-vl-benchmark-handling branch from 515d609 to 3c560e9 Compare August 28, 2026 19:04
@rshube

rshube commented Aug 28, 2026

Copy link
Copy Markdown
Author

Hello @mzr1996 , I am an MTS at Liquid AI and I made this PR to add the necessary pre-/post-processing for our latest release (LFM2.5-VL-3B) benchmark scores to be reproducible with VLMEvalKit

I see that you have reviewed a few of the most recent PRs, I would greatly appreciate if you could take a look at this one please

Comment thread vlmeval/smp/file.py
if 'image_path' not in data:
data['image_path'] = [x[0] if len(x) == 1 else x for x in ret]
data['image_path'] = [x[0] if len(x) == 1 else x for x in ret]
return data

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Potential regression: This change removes the if 'image_path' not in data guard and unconditionally overwrites a caller-provided image_path.

localize_df already treats an existing image_path as the authoritative filename mapping (img_paths = list(data['image_path'])). However, decode_img_omni returns osp.join(root, p) for every entry, so
relative paths are rewritten under LMUData/images/<dname>/. It also skips decoding short reference values, meaning this can produce a new path that was never created when the original image_path pointed to
an already-localized file.

This affects every caller of the shared localization helper, not only LFM2.5-VL. Could we keep the existing guard and handle any LFM-specific path normalization at the relevant caller, or add a regression test
covering pre-existing relative/absolute image_path values and short image references?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants