[Task] Report sub category score for 3DSRBench and Viewspatial#1285
Merged
Conversation
Contributor
Author
Contributor
Author
|
Added sub category metrics for Embspatial as well: |
Contributor
Author
|
Also added vsibench_debiased by frames: |
Contributor
Author
|
Added sub category metrics for Sparbench as well: |
kcz358
approved these changes
Apr 7, 2026
11 tasks
mwxely
added a commit
that referenced
this pull request
Apr 9, 2026
Per review feedback (ref: PR #1285 pattern): - Report relevance/logic group-type scores separately - Report per-level (1/2/3) scores separately - Refactor aggregate logic into _compute_all_subscores helper - process_results returns same entry under all 6 metric keys - Detailed second_head/third_head breakdowns still logged
kcz358
pushed a commit
that referenced
this pull request
Apr 9, 2026
* feat(videomme_v2): add task config and default template - Dataset: MME-Benchmarks/Video-MME-v2 (800 videos, 3200 questions) - 8-option MCQ (A-H) with grouped non-linear scoring - Generation config: max_new_tokens=64, temperature=0 * feat(videomme_v2): add scoring, prompts, and evaluation logic - Grouped non-linear scoring: relevance (quadratic) + logic (chain-based) - 3 group structures: [1,2,3,4], [1,[2,3],4], [[1,2],3,4] - Answer extraction with 11 prefix patterns (A-H range) - Per-level, per-category, per-group-type breakdown reporting - Prompt aligned with official INSTRUCT_PROMPT - Verified against VLMEvalKit implementation * feat(videomme_v2): add subtitle variant (concatenated mode) - Load word-level JSONL subtitles and prepend to prompt - Graceful fallback when subtitle file is missing - Task: videomme_v2_w_subtitle * feat(videomme_v2): add reasoning mode variant - Chain-of-thought prompt requiring Final Answer: <letter> format - max_new_tokens=4096 for reasoning space - Task: videomme_v2_reasoning * fix(videomme_v2): report sub-category scores as separate metrics Per review feedback (ref: PR #1285 pattern): - Report relevance/logic group-type scores separately - Report per-level (1/2/3) scores separately - Refactor aggregate logic into _compute_all_subscores helper - process_results returns same entry under all 6 metric keys - Detailed second_head/third_head breakdowns still logged --------- Co-authored-by: mwxely <mwxely@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Updated 3DSRBench and Viewspatial bench to report sub category metric scores
test run (3DSRBench):
test run (Viewspatial):
