Skip to content

bench: load committed Tier-A CMNN datasets from LFS (SOURCES lfs + runners use require_dataset) #144

Description

@davorrunje

Context

The CMNN paper-reproduction datasets (auto, heart, compas, blog, loan)
are now registered as Tier A in the honest-scholar registry datasets.yml
and their redistributable copies are committed via Git LFS under
benchmarks/data/<name>/{train,test}_<name>.csv.gz (author sign-off 2026-07-22;
see the PR that closed #138). honest-scholar dataset verify <id> resolves and
checks those committed bytes at their repo path.

However, the benchmark harness still loads these five datasets from the
runtime download cache
, not from the committed copies. Every runner resolves
their data_dir via benchmarks.datasets.download.default_dest() (the
~/.cache/mononet/datasets / $MONONET_DATA_DIR cache populated by
benchmarks/datasets/download.py from Zenodo 7968969), and
benchmarks/datasets/sources.py::SOURCES has no entries for these five — so
resolve_dir("auto") / require_dataset("auto") would KeyError. The large-screen
datasets (adult, german, lc, polish, taiwan) already do the right thing:
they are in SOURCES with hosting="lfs" and load from benchmarks/data/ via
require_dataset. This issue makes the CMNN five consistent, so a fresh clone
reproduces CMNN results with zero network setup (git lfs pull only).

Goal

Make the committed Tier-A LFS copies the harness's canonical source for auto,
heart, compas, blog, loan, mirroring the existing large-screen datasets —
so the benchmark runners load them from benchmarks/data/<name>/ with no runtime
Zenodo fetch.

Where

  • benchmarks/datasets/sources.pySOURCES, resolve_dir, require_dataset.
    Add the five with hosting="lfs", license="CC-BY-4.0 (Zenodo)",
    url="https://zenodo.org/records/7968969", and a prep_hint pointing at
    benchmarks/datasets/download.py.
  • The runners that currently pass data_dir=default_dest() for these datasets:
    • benchmarks/run.py:159
    • benchmarks/sensitivity.py:81
    • benchmarks/flavor_ablation.py:136
    • benchmarks/loan_size_ladder_run.py:244
    • benchmarks/_common/search.py:445
  • benchmarks/large_screen_run.py:79 is the reference pattern (already uses
    require_dataset).
  • Leave benchmarks/datasets/download.py + benchmarks/datasets/manifest.toml
    as the regeneration/provenance path (manifest.toml pins the raw upstream
    .csv SHA-256; datasets.yml pins the committed .csv.gz SHA-256).

Proposed approach

Add a single resolver that returns require_dataset(name) when name in SOURCES
(committed Tier A) and falls back to default_dest() otherwise (generator-backed
synth_* specs and any cache-only dataset), then route the runner call sites
through it. This keeps synthetic datasets (which ignore data_dir) and the
download path working. The committed .csv.gz decompresses to the exact Zenodo
raw .csv (verified when #138 landed), so loaded bundles are byte-identical to
the previous cache path — no result changes expected.

Acceptance criteria

  • python -m benchmarks.run --dataset heart ... (and the other four) loads
    from benchmarks/data/<name>/ on a fresh clone with no prior download
    and no MONONET_DATA_DIR set (after git lfs pull).
  • require_dataset("auto"|"heart"|"compas"|"blog"|"loan") resolves to
    benchmarks/data/<name>/ instead of raising KeyError.
  • Loaded DatasetBundles are byte-identical to the pre-change cache path
    (spot-check X/y shapes + a checksum of the arrays for one dataset).
  • uv run pytest -m "not slow" and the benchmark unit tests pass.
  • honest-scholar dataset verify <id> still passes for all five.

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestfollow-upDeferred work captured as a self-contained issue

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions