bench(results): large-dataset screen — monotone depth is neutral across all 5 datasets#115
Draft
davorrunje wants to merge 6 commits into
Draft
bench(results): large-dataset screen — monotone depth is neutral across all 5 datasets#115davorrunje wants to merge 6 commits into
davorrunje wants to merge 6 commits into
Conversation
Small batch sizes make 50-epoch search intractable on 24k+ row datasets under n_jobs=1 (Optuna kept picking batch 8-64 -> tens of thousands of steps/fit). Add lc/adult/taiwan to _LARGE_BATCH_DATASETS (the existing tractability mechanism, same as loan/blog); polish/german stay small-batch (genuinely small).
Theory follow-up to the depth-null screen result. Synthetic monotone targets with a complexity knob (additive control; monotone teacher-depth sweep; max/min-lattice teacher) separate expressivity/data-simplicity from optimization, and test H-strong (monotone class is depth-insensitive; DLN contrast) vs H-weak (null is data-driven). Reuses run_ladder deep/shallow arms; MSE regression to dodge the accuracy-ceiling artifact. Benchmark-only.
…datasets Max-size deep/shallow screen (coarse-gate budget: 15 trials/2 seeds/8 final/30 ep, n_jobs=1). Every dataset (german 800 -> lc 829k) lands 'standard': monotone depth never clears the CI+practical-floor gate. adult is the only near-signal (Δ=+0.0048, sub-margin); taiwan/lc collapse to base rate (accuracy ceiling). Fills docs/benchmarks/large-dataset-screen.md (table + Δ plot PNG+PDF + gate interpretation) and commits the per-dataset + merged result JSONs. Corroborates the loan size-ladder across new domains/scales. Theory follow-up: the synthetic-probe spec.
…roved spec) 4 TDD tasks: synthetic monotone-target generator (additive/teacher/lattice with a complexity knob), the deep/shallow probe runner (MSE, task-aware), the Δ(c) report (shallow-deep sign, positive=depth helps), and docs/RUNBOOK. Reuses run search + final_eval + the size-ladder bootstrap; benchmark-only.
This was referenced Jul 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Results of the max-size deep/shallow screen across the 5-dataset roster (adult + the four onboarded in #85), plus the tractability fix that made the large datasets runnable and the theory-probe design the result motivates. Benchmark-only (no
mononetchange).Result: monotone depth is neutral across every dataset
Coarse-gate budget (15 trials / 2 search-seeds / 8 test-seeds / 30 epochs,
n_jobs=1); Δ = IQM(deep) − IQM(shallow), 95% seed-bootstrap band; gate = advance to a size-ladder iff Δ_lo > 0 and Δ ≥ 0.005.Every dataset routes to
standard— depth never clears the gate, from 800 to 829k rows. This corroborates the loan size-ladder (#81, no dose-response in N) across new domains/scales.adultis the only near-signal (Δ=+0.0048, misses both criteria);polishshows the CI+floor gate correctly rejecting a significant-but-trivial gap;taiwan/lccollapse to base rate on both arms (accuracy ceiling on imbalanced targets).Also in this PR
_LARGE_BATCH_DATASETS+= lc/adult/taiwan (search_spaces.py) — small batches made 50-epoch search on 24k+ rows intractable undern_jobs=1; the large-batch band (the existing loan/blog tractability lever) cut per-fit steps ~10–100× and made the screen finish. Small sets (polish/german) stay small-batch.docs/benchmarks/large-dataset-screen.mdfilled: table + Δ-per-dataset plot (PNG + LFS PDF) + gate interpretation (incl. the collapse notes).docs/superpowers/specs/2026-07-12-monotone-depth-synthetic-probe-design.md: a synthetic-monotone-target experiment to explain why depth doesn't help (separating expressivity vs data-simplicity vs optimization; the Deep-Lattice-Networks contrast). Approved; implementation to follow.Follow-ups (tracked)
🤖 Generated with Claude Code
Fresh draft superseding the closed #90 (reopen was blocked after the
mainLFS history rewrite; branch content unchanged).