Skip to content

[Benchmark] Add ActiveVision benchmark - #1646

Open
lzpp2598 wants to merge 1 commit into
open-compass:mainfrom
lzpp2598:add-activevision-benchmark
Open

[Benchmark] Add ActiveVision benchmark#1646
lzpp2598 wants to merge 1 commit into
open-compass:mainfrom
lzpp2598:add-activevision-benchmark

Conversation

@lzpp2598

Copy link
Copy Markdown
Contributor

Description

Adds ActiveVision (arXiv:2607.16165), an exam for active observers: 85 single-image QA items across 17 tasks / 3 categories, rule-based exact-match scoring.

  • Dataset class: ActiveVisionDataset (TYPE='VQA') in vlmeval/dataset/activevision.py
  • Data: auto-downloaded from HF activevisionai/ActiveVision (metadata.jsonl + images), converted to a cached TSV with base64-embedded images on first run
  • Prompt: question text is sent verbatim (it already instructs the model to answer in <answer>...</answer> tags), no extra instruction appended
  • Scoring: faithful port of the official eval/lib/scoring.py — answer-tag extraction, int/string normalisation, exact match; reports Overall / per-category / per-task accuracy

Validation

  • Smoke-tested: build_dataset('ActiveVision') loads 85 samples; build_prompt yields image+text messages; evaluate on synthetic predictions produces the expected Overall / 3 category / 17 task columns
  • Scoring functions verified against the official repo's implementation

@lzpp2598
lzpp2598 force-pushed the add-activevision-benchmark branch from d3de5a4 to ddf6be2 Compare August 20, 2026 04:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant