Context
The CMNN paper-reproduction datasets (auto, heart, compas, blog, loan)
are now registered as Tier A in the honest-scholar registry datasets.yml
and their redistributable copies are committed via Git LFS under
benchmarks/data/<name>/{train,test}_<name>.csv.gz (author sign-off 2026-07-22;
see the PR that closed #138). honest-scholar dataset verify <id> resolves and
checks those committed bytes at their repo path.
However, the benchmark harness still loads these five datasets from the
runtime download cache, not from the committed copies. Every runner resolves
their data_dir via benchmarks.datasets.download.default_dest() (the
~/.cache/mononet/datasets / $MONONET_DATA_DIR cache populated by
benchmarks/datasets/download.py from Zenodo 7968969), and
benchmarks/datasets/sources.py::SOURCES has no entries for these five — so
resolve_dir("auto") / require_dataset("auto") would KeyError. The large-screen
datasets (adult, german, lc, polish, taiwan) already do the right thing:
they are in SOURCES with hosting="lfs" and load from benchmarks/data/ via
require_dataset. This issue makes the CMNN five consistent, so a fresh clone
reproduces CMNN results with zero network setup (git lfs pull only).
Goal
Make the committed Tier-A LFS copies the harness's canonical source for auto,
heart, compas, blog, loan, mirroring the existing large-screen datasets —
so the benchmark runners load them from benchmarks/data/<name>/ with no runtime
Zenodo fetch.
Where
benchmarks/datasets/sources.py — SOURCES, resolve_dir, require_dataset.
Add the five with hosting="lfs", license="CC-BY-4.0 (Zenodo)",
url="https://zenodo.org/records/7968969", and a prep_hint pointing at
benchmarks/datasets/download.py.
- The runners that currently pass
data_dir=default_dest() for these datasets:
benchmarks/run.py:159
benchmarks/sensitivity.py:81
benchmarks/flavor_ablation.py:136
benchmarks/loan_size_ladder_run.py:244
benchmarks/_common/search.py:445
benchmarks/large_screen_run.py:79 is the reference pattern (already uses
require_dataset).
- Leave
benchmarks/datasets/download.py + benchmarks/datasets/manifest.toml
as the regeneration/provenance path (manifest.toml pins the raw upstream
.csv SHA-256; datasets.yml pins the committed .csv.gz SHA-256).
Proposed approach
Add a single resolver that returns require_dataset(name) when name in SOURCES
(committed Tier A) and falls back to default_dest() otherwise (generator-backed
synth_* specs and any cache-only dataset), then route the runner call sites
through it. This keeps synthetic datasets (which ignore data_dir) and the
download path working. The committed .csv.gz decompresses to the exact Zenodo
raw .csv (verified when #138 landed), so loaded bundles are byte-identical to
the previous cache path — no result changes expected.
Acceptance criteria
References
Context
The CMNN paper-reproduction datasets (
auto,heart,compas,blog,loan)are now registered as Tier A in the honest-scholar registry
datasets.ymland their redistributable copies are committed via Git LFS under
benchmarks/data/<name>/{train,test}_<name>.csv.gz(author sign-off 2026-07-22;see the PR that closed #138).
honest-scholar dataset verify <id>resolves andchecks those committed bytes at their repo path.
However, the benchmark harness still loads these five datasets from the
runtime download cache, not from the committed copies. Every runner resolves
their
data_dirviabenchmarks.datasets.download.default_dest()(the~/.cache/mononet/datasets/$MONONET_DATA_DIRcache populated bybenchmarks/datasets/download.pyfrom Zenodo 7968969), andbenchmarks/datasets/sources.py::SOURCEShas no entries for these five — soresolve_dir("auto")/require_dataset("auto")wouldKeyError. The large-screendatasets (
adult,german,lc,polish,taiwan) already do the right thing:they are in
SOURCESwithhosting="lfs"and load frombenchmarks/data/viarequire_dataset. This issue makes the CMNN five consistent, so a fresh clonereproduces CMNN results with zero network setup (
git lfs pullonly).Goal
Make the committed Tier-A LFS copies the harness's canonical source for
auto,heart,compas,blog,loan, mirroring the existing large-screen datasets —so the benchmark runners load them from
benchmarks/data/<name>/with no runtimeZenodo fetch.
Where
benchmarks/datasets/sources.py—SOURCES,resolve_dir,require_dataset.Add the five with
hosting="lfs",license="CC-BY-4.0 (Zenodo)",url="https://zenodo.org/records/7968969", and aprep_hintpointing atbenchmarks/datasets/download.py.data_dir=default_dest()for these datasets:benchmarks/run.py:159benchmarks/sensitivity.py:81benchmarks/flavor_ablation.py:136benchmarks/loan_size_ladder_run.py:244benchmarks/_common/search.py:445benchmarks/large_screen_run.py:79is the reference pattern (already usesrequire_dataset).benchmarks/datasets/download.py+benchmarks/datasets/manifest.tomlas the regeneration/provenance path (manifest.toml pins the raw upstream
.csvSHA-256;datasets.ymlpins the committed.csv.gzSHA-256).Proposed approach
Add a single resolver that returns
require_dataset(name)whenname in SOURCES(committed Tier A) and falls back to
default_dest()otherwise (generator-backedsynth_*specs and any cache-only dataset), then route the runner call sitesthrough it. This keeps synthetic datasets (which ignore
data_dir) and thedownload path working. The committed
.csv.gzdecompresses to the exact Zenodoraw
.csv(verified when #138 landed), so loaded bundles are byte-identical tothe previous cache path — no result changes expected.
Acceptance criteria
python -m benchmarks.run --dataset heart ...(and the other four) loadsfrom
benchmarks/data/<name>/on a fresh clone with no prior downloadand no
MONONET_DATA_DIRset (aftergit lfs pull).require_dataset("auto"|"heart"|"compas"|"blog"|"loan")resolves tobenchmarks/data/<name>/instead of raisingKeyError.DatasetBundles are byte-identical to the pre-change cache path(spot-check X/y shapes + a checksum of the arrays for one dataset).
uv run pytest -m "not slow"and the benchmark unit tests pass.honest-scholar dataset verify <id>still passes for all five.References
datasets.yml,benchmarks/datasets/sources.py,benchmarks/datasets/download.pybenchmarks/datasets/manifest.toml