Grouped and pipeline-level resampling for selection stability - #22
Merged
Conversation
Row resampling measures the wrong uncertainty whenever rows are not independent observations. Two cases from #17: - Rows are grouped (several rows per condition, curve, subject). A row bootstrap leaks: a replicate that drops half a group still trains on the rest, so the reported spread is far narrower than the real between-group variability. - Rows are outputs of an upstream fit (spline evaluations, estimated derivatives, simulation output). Resampling those rows perturbs nothing about the stage that produced them, so the dominant error source is invisible to the bootstrap entirely. Adds, all backwards compatible: - bootstrap_model_selection(..., groups=...) resamples whole groups with replacement instead of rows. - bootstrap_model_selection(..., resample_fn=...) hands each replicate back to the caller as resample_fn(rng) -> (X_b, y_b), so the upstream stage can be re-run per replicate. X/y may then be None. - summarize_selection_replicates() is the reporting half, made public: point it at replicates you produced yourself (fitted models, mappings with a "features" key, or plain name lists) and get the same summary. It now also reports "structures" (each distinct selected feature set with its frequency) and "n_distinct_structures", which is what makes "nine distinct structural forms" reportable at all. - The result records which level was resampled ("rows", "groups", "pipeline") and how many replicate fits failed. - cross_validate(..., groups=..., strategy=...) supports "group-kfold" and "leave-one-group-out"; passing groups promotes the default strategy to "group-kfold". Results carry per-group scores and flag the lowest/highest numeric group labels as "edge_groups", since those folds are extrapolation and carry different risk than the mean. Replicate cloning now goes through _clone_estimator, so options the old hand-written clone dropped (param_optimizer, constraint_enforcement, prune_tol, ...) are carried into every replicate. Guides, skill decision tables and the review checklists are updated to route on the resampling level first. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vp45J9ECvQ9he31ALBdBFz
jkitchin
pushed a commit
that referenced
this pull request
Aug 12, 2026
Main's #22 rewrote bootstrap_model_selection into a generator plus a public summarize_selection_replicates, and added row/group/pipeline resampling. This branch's fix is re-expressed inside that architecture rather than alongside it: - Canonical keying moves into _replicate_terms, so both the summariser and the bootstrap key model replicates by basis identity. Callers passing their own replicates (mappings, name sequences) are unaffected. - Parameter distributions are accumulated in summarize_selection_replicates, so pipeline- and group-level replicates report them too, not just rows. - The per-replicate library copy moves into _clone_estimator, which #22 made the single clone path. That also fixes MultiOutputSymbolicRegressor, whose per-output clones shared one parametric library. - Dropped this branch's "n_successful" (main's n_replicates/n_failed say the same thing) and its unfitted-template check (main deliberately accepts an unfitted template and reports the modal structure instead). test_clone_symbolic_regressor asserted the clone shares the template's library; that identity is exactly what had to change, so it now asserts an equivalent but independent library. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ARRqz2SjSoYH5N3t66HDjm
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #17.
Row resampling measures the wrong uncertainty whenever rows are not independent observations. This adds the two missing resampling levels and makes the reporting half usable on replicates the caller produced. All additions are backwards compatible — existing calls behave as before.
What's new
bootstrap_model_selection(model, X, y, groups=...)— resamples whole groups with replacement instead of rows, so a replicate never trains on part of a group it also holds out. This is the isotherm/condition/subject case: a row bootstrap that drops half a group still sees the rest, and reports a spread far narrower than the real between-group variability.bootstrap_model_selection(model, None, None, resample_fn=...)— the pipeline-level hook.resample_fn(rng) -> (X_b, y_b)is called once per replicate with the bootstrap'sRandomState, so whatever stage produced the rows (smoother, derivative estimate, simulation) is re-run per replicate instead of being frozen.X/ymay beNone;groupsandresample_fnare mutually exclusive.summarize_selection_replicates(replicates, reference=..., resampling=...)— the reporting half, made public. Accepts fitted models,{"features": [...], "expression": ...}mappings, or plain lists of term names, so replicates orchestrated entirely outsideSymbolicRegressorget the same summary. It addsstructures(each distinct selected feature set with its count and frequency) andn_distinct_structures, which is what makes "nine distinct structural forms" reportable at all. Without areference,stability_scorefalls back to the modal structure's frequency.Bootstrap results now also record
resampling("rows"/"groups"/"pipeline") andn_failed.cross_validate(..., groups=..., strategy=...)— adds"group-kfold"(whole groups balanced acrosscvfolds) and"leave-one-group-out". Passinggroupspromotes the default strategy to"group-kfold". Results carryper_group_scores, andedge_groupsflags the lowest and highest group labels when labels are numeric — those folds are extrapolation, not interpolation, and averaging them intomean_test_scorehides the failure most likely to be asked about. On the demo data the edge groups score roughly 4x worse than the interior ones.Incidental fix
Replicate cloning now goes through
_clone_estimatorrather than a hand-written constructor call that listed eight parameters. Options the old clone silently dropped —param_optimizer,param_optimization_budget,constraint_enforcement,constraint_selection_weight,prune_tol— now reach every replicate.Docs
guides/uncertainty.mdgains a "Resample at the level your data actually varies" section with a routing table and worked examples for all three levels;guides/model-fitting.mdgains a grouped cross-validation section covering edge groups. The skill decision table now routes on resampling level before method, and both review checklists (CLAUDE.md,jaxsr-review) gained rows for the new signatures.src/jaxsr/skill/re-synced from.claude/skills/jaxsr/.Testing
scripts/test_under_numpy.py) — no new JAX API calls, so the shim is untouchedblack --check src/ tests/andruff check src/ tests/cleanNot addressed
feature_frequenciesstill keys on rendered names, so a grouped bootstrap over a parametric basis will still fail to aggregate. The two issues compose but are independent.SymbolicRegressor.cv_foldsturns out to be stored but never used (selection is information-criterion driven), so there was no internal CV path to make group-aware.cross_validate(groups=...)is the entire selection-scoring surface.🤖 Generated with Claude Code
https://claude.ai/code/session_01Vp45J9ECvQ9he31ALBdBFz
Generated by Claude Code