feat: add Video-MME-v2 benchmark task#1289
Merged
Merged
Conversation
mwxely
force-pushed
the
feat/video-mme-v2
branch
3 times, most recently
from
272c0d2 to
a98c362
Compare
- Dataset: MME-Benchmarks/Video-MME-v2 (800 videos, 3200 questions) - 8-option MCQ (A-H) with grouped non-linear scoring - Generation config: max_new_tokens=64, temperature=0
- Grouped non-linear scoring: relevance (quadratic) + logic (chain-based) - 3 group structures: [1,2,3,4], [1,[2,3],4], [[1,2],3,4] - Answer extraction with 11 prefix patterns (A-H range) - Per-level, per-category, per-group-type breakdown reporting - Prompt aligned with official INSTRUCT_PROMPT - Verified against VLMEvalKit implementation
- Load word-level JSONL subtitles and prepend to prompt - Graceful fallback when subtitle file is missing - Task: videomme_v2_w_subtitle
- Chain-of-thought prompt requiring Final Answer: <letter> format - max_new_tokens=4096 for reasoning space - Task: videomme_v2_reasoning
mwxely
force-pushed
the
feat/video-mme-v2
branch
from
a98c362 to
73603b4
Compare
kcz358
reviewed
Apr 9, 2026
Collaborator
There was a problem hiding this comment.
The aggregation can revise a bit to report sub category score better. Just like #1285 does. Can let agent refer and change a bit. Other LGTM
Per review feedback (ref: PR #1285 pattern): - Report relevance/logic group-type scores separately - Report per-level (1/2/3) scores separately - Refactor aggregate logic into _compute_all_subscores helper - process_results returns same entry under all 6 metric keys - Detailed second_head/third_head breakdowns still logged
Collaborator
Author
|
Thanks @kcz358 for the review! Updated in 81e162e — now reports 6 separate metrics following the PR #1285 pattern:
Detailed second_head/third_head breakdowns are still logged via |
kcz358
approved these changes
Apr 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Tasks added
videomme_v2videomme_v2_w_subtitlevideomme_v2_reasoningDetails
Test plan
--limit 8)INSTRUCT_PROMPTandTHINK_PROMPTcal_relevance+cal_logicPlanned follow-ups
--subtitle-interleave). In lmms-eval, this requires coordination with the model adapter layer (each model handles frame/text interleaving differently), so it cannot be implemented purely at the task level. Will need model-specific support.level_1/2/3,relevance_score,logic_scoreas independent metrics. Currently these are logged viaeval_logger.info()during aggregation. Exposing them as separate reportable metrics requires adding multiplemetric_listentries with dedicated aggregation functions. Current implementation reports the overall grouped score as the primary metric.