Skip to content

[Dataset] Add downloadable OmniMat data URLs - #1639

Open
yansumm wants to merge 1 commit into
open-compass:mainfrom
yansumm:codex/omnimat-data-url
Open

[Dataset] Add downloadable OmniMat data URLs#1639
yansumm wants to merge 1 commit into
open-compass:mainfrom
yansumm:codex/omnimat-data-url

Conversation

@yansumm

@yansumm yansumm commented Aug 18, 2026

Copy link
Copy Markdown

Summary

  • populate the published TSV URLs for OmniMat_QA and OmniMat_CAL
  • fall back to prepare_tsv() when no local OmniMat TSV is available
  • preserve the existing local-file precedence and ID normalization
  • add focused regression tests for the URL contract and download fallback

This PR intentionally does not change the OmniMat prompts or scoring logic. Those concerns are deferred to a separate follow-up.

Dataset scope and publication status

The linked Hugging Face repository currently contains the OmniMat1K release:

  • 498 QA examples
  • 502 CAL examples
  • 1,000 examples in total

This is not the full 3,171-example dataset described in the paper. The Hugging Face repository is currently public and ungated, but its API metadata does not yet declare a license or dataset owner. This PR is therefore opened as a draft pending confirmation from the maintainers/data owner about the intended upstream naming and publication metadata.

Addresses the OmniMat portion of #1614.

Verification

  • pytest -q tests/test_omnimat_data_url.py — 2 passed
  • fresh temporary LMUData download — 498 QA + 502 CAL loaded successfully
  • both published URLs returned HTTP 206 and downloaded successfully
  • uvx pre-commit run --files vlmeval/dataset/omnimat.py tests/test_omnimat_data_url.py — passed
  • uvx pre-commit run --all-files — blocked by two pre-existing E226 violations on upstream main:
    • vlmeval/dataset/utils/memlens_utils.py:848
    • vlmeval/dataset/utils/mmlongbench_metrics.py:109

@yansumm
yansumm marked this pull request as ready for review August 18, 2026 03:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant