Skip to content

[Benchmark] Add support for PixmoPoints benchmark#1523

Open
rshube wants to merge 1 commit into
open-compass:mainfrom
rshube:benchmark/add_pixmopoints_support
Open

[Benchmark] Add support for PixmoPoints benchmark#1523
rshube wants to merge 1 commit into
open-compass:mainfrom
rshube:benchmark/add_pixmopoints_support

Conversation

@rshube
Copy link
Copy Markdown

@rshube rshube commented Apr 22, 2026

Summary

This PR adds support for the PixmoPoints benchmark in VLMEvalKit.

Changes

  • adds a dedicated PixmoPointsDataset
  • registers PixmoPoints in the dataset registry
  • adds benchmark-specific point localization evaluation
  • uses the repo-standard point_2d JSON prediction format

Dataset format

Expected TSV columns:

  • index
  • image
  • question
  • answer
  • width
  • height
  • category

Evaluation

The evaluator performs point matching against normalized ground-truth points and reports:

  • precision
  • recall
  • f1

Notes

  • this PR only adds benchmark support code
  • hosted DATASET_URL / DATASET_MD5 can be filled once the final TSV hosting location is settled (sending email to maintainers)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant