Skip to content

[Feature] Support evaluating selected dataset indices - #1634

Open
reacher-z wants to merge 1 commit into
open-compass:mainfrom
reacher-z:feat/data-index-subsets
Open

[Feature] Support evaluating selected dataset indices#1634
reacher-z wants to merge 1 commit into
open-compass:mainfrom
reacher-z:feat/data-index-subsets

Conversation

@reacher-z

@reacher-z reacher-z commented Aug 16, 2026

Copy link
Copy Markdown

Summary

  • add --data-indices as a JSON mapping from dataset names to sample indices
  • apply the subset consistently in local, distributed, and async API modes
  • keep video pack metadata and reused prediction/checkpoint files scoped to the selected samples
  • avoid reusing full-dataset evaluation auxiliary files for subset runs
  • document the option in the English and Chinese quickstarts

This lets users rerun a few problematic samples without rebuilding a custom dataset. Unknown datasets, missing indices, invalid values, and duplicates fail explicitly instead of being silently skipped.

python run.py --data MMBench_DEV_EN --data-indices '{"MMBench_DEV_EN": [1203, 63384]}' --model qwen_chat

Closes #1474.

Tests

  • python -m unittest discover -s tests -p 'test_dataset_subset.py'
  • python tests/test_inference_api.py TestInferenceApiProcessHelpers.test_load_checkpoint_ignores_results_outside_dataset_subset
  • pre-commit run --files run.py vlmeval/inference_api.py vlmeval/utils/dataset_subset.py tests/test_dataset_subset.py tests/test_inference_api.py docs/en/Quickstart.md docs/zh-CN/Quickstart.md

AI assistance

Implementation and tests were prepared with OpenAI Codex assistance and manually validated before submission.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

是否支持数据集指定索引的评测?

1 participant