Skip to content

Add embedding and tiling diagnostics evals#573

Draft
Hgherzog wants to merge 3 commits into
mainfrom
henryh/embedding-tiling-diagnostics
Draft

Add embedding and tiling diagnostics evals#573
Hgherzog wants to merge 3 commits into
mainfrom
henryh/embedding-tiling-diagnostics

Conversation

@Hgherzog

@Hgherzog Hgherzog commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add embedding geometry diagnostics and tiling artifact metrics for eval-time representation checks.
  • Extend pretrain-subset eval adapters so diagnostics can run on fixed-size pretraining samples and optional pinned subsets.
  • Wire diagnostics through downstream evaluator logging and document the workflows.

Test plan

  • uv run pytest -vv tests/unit/eval/test_embedding_diagnostics.py tests/unit/eval/datasets/test_pretrain_subset.py
  • uv run python - <<'PY'
    from olmoearth_pretrain.evals.datasets.configs import dataset_to_config, get_eval_mode
    from olmoearth_pretrain.internal.all_evals import TILING_DIAG_TASKS
    print(get_eval_mode(dataset_to_config('pretrain_subset').task_type))
    print(get_eval_mode(dataset_to_config('pretrain_subset_128').task_type))
    print(sorted(TILING_DIAG_TASKS))
    PY

Notes

Made with Cursor

Add diagnostic eval modes and pretrain-subset adapters so representation geometry and patch-grid artifacts can be tracked separately from downstream probe scores.

Co-authored-by: Cursor <cursoragent@cursor.com>
gabrieltseng and others added 2 commits July 17, 2026 10:25
# Conflicts:
#	olmoearth_pretrain/evals/datasets/__init__.py
#	olmoearth_pretrain/evals/datasets/configs.py
#	olmoearth_pretrain/evals/eval_wrapper.py
#	olmoearth_pretrain/evals/task_types.py
#	olmoearth_pretrain/train/callbacks/evaluator_callback.py
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants