Skip to content

Add LFM2.5-VL-3B benchmark handling - #1

Open
rshube wants to merge 4 commits into
mainfrom
wip/lfm25-vl-benchmark-handling
Open

Add LFM2.5-VL-3B benchmark handling#1
rshube wants to merge 4 commits into
mainfrom
wip/lfm25-vl-benchmark-handling

Conversation

@rshube

@rshube rshube commented Aug 27, 2026

Copy link
Copy Markdown

Summary

  • Add benchmark-specific prompting and ScreenSpot response normalization for LFM2.5-VL-3B.
  • Preserve existing LFM2-VL registrations without custom prompting.
  • Preserve chat roles, interleaved inputs, completion-only decoding, and deterministic generation.
  • Fix MM-IFEval verifier lookup and localized image paths.

Validation

  • pytest -q: 27 passed
  • Python compilation and liquid.py lint passed
  • ScreenSpot-v2 and MMMU-Pro scores reproduced exactly
  • MMMU validation predictions reproduced exactly

@rshube

rshube commented Aug 27, 2026

Copy link
Copy Markdown
Author

Checkpoint reproduction for affected benchmarks

Only benchmarks with dataset-specific prompting or response handling in this PR are listed. Published values are from the release table; scores are normalized to 0–100.

Benchmark Published Reproduced Delta Handling
MME 73.1 73.89 +0.79 Category-specific answer formats
SimpleVQA 35.4 35.67 +0.27 No appended instruction
MM-IFEval 60.6 59.04 -1.56 Preserve image-first ordering; judge-dependent score
LogicVista 37.4 37.58 +0.18 No appended instruction
MathVista (mini) 68.5 68.50 0.00 Step-by-step reasoning with boxed final answer
MMMU (validation) 48.4 48.22 -0.18 No appended instruction
OCRBench v1 84.2 84.10 -0.10 Concise-answer instruction
BLINK 61.5 61.55 +0.05 No redundant instruction
MUIRBench 58.3 58.27 -0.03 Numbered images and explicit final option format
HallusionBench 47.2 47.25 +0.05 Yes/no instruction; judge-dependent score
POPE 88.7 88.75 +0.05 No appended instruction
RefCOCO average 87.9 88.86 +0.96 Strict normalized bounding-box JSON
ScreenSpot-v2 Desktop 78.7 78.44 -0.26 Click-target prompt and response normalization
ScreenSpot-v2 Mobile 81.2 80.64 -0.56 Click-target prompt and response normalization
ScreenSpot-v2 Web 82.2 82.15 -0.05 Click-target prompt and response normalization

TianhaoLiang2000 and others added 3 commits August 28, 2026 06:31
…py (open-compass#1532)

* Centralize default judge model selection

* Move judge defaults to benchmark classes

* Support benchmark-specific judge defaults

* Simplify judge default resolution

* Fix dataset instance default judge resolution

* Use class attributes for fixed judge defaults

* Simplify default judge model resolution

* Refactor default judge model resolution
* Add OpenAI Responses API wrapper

* Support selecting inference API wrapper

* Log non-stop Responses finish reason

* Improve OpenAI Responses request handling

* Rename API model class argument

* Apply pre-commit fixes for MolRecBench Wild
* Fix LMDeploy response charset handling

* Document response charset options
@rshube
rshube force-pushed the wip/lfm25-vl-benchmark-handling branch from 469597a to 515d609 Compare August 28, 2026 18:52
@rshube
rshube force-pushed the wip/lfm25-vl-benchmark-handling branch from 515d609 to 3c560e9 Compare August 28, 2026 19:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants