Add LFM2.5-VL-3B benchmark handling - #1663
Conversation
Checkpoint reproduction for affected benchmarksOnly benchmarks with dataset-specific prompting or response handling in this PR are listed. Published values are from the release table; scores are normalized to 0–100.
|
515d609 to
3c560e9
Compare
|
Hello @mzr1996 , I am an MTS at Liquid AI and I made this PR to add the necessary pre-/post-processing for our latest release (LFM2.5-VL-3B) benchmark scores to be reproducible with VLMEvalKit I see that you have reviewed a few of the most recent PRs, I would greatly appreciate if you could take a look at this one please |
| if 'image_path' not in data: | ||
| data['image_path'] = [x[0] if len(x) == 1 else x for x in ret] | ||
| data['image_path'] = [x[0] if len(x) == 1 else x for x in ret] | ||
| return data |
There was a problem hiding this comment.
Potential regression: This change removes the if 'image_path' not in data guard and unconditionally overwrites a caller-provided image_path.
localize_df already treats an existing image_path as the authoritative filename mapping (img_paths = list(data['image_path'])). However, decode_img_omni returns osp.join(root, p) for every entry, so
relative paths are rewritten under LMUData/images/<dname>/. It also skips decoding short reference values, meaning this can produce a new path that was never created when the original image_path pointed to
an already-localized file.
This affects every caller of the shared localization helper, not only LFM2.5-VL. Could we keep the existing guard and handle any LFM-specific path normalization at the relevant caller, or add a regression test
covering pre-existing relative/absolute image_path values and short image references?
Summary
Validation