Drop redundant data columns + remove always-zero MRWR feature - #19
Merged
Conversation
SebastianAment
force-pushed
the
data-cleanup-redundancies
branch
4 times, most recently
from
May 11, 2026 22:54
335b02f to
bb6e967
Compare
Data cleanup of the canonical CSV (data/boxcrete_data.csv) and the model
input vector. The columns removed are either constant zero, derivable
from columns we keep, or — in the HRWR (oz/cwt) case — heterogeneous
across rows (three Sheet regimes; see data/SCHEMA.md). The MRWR removal
also drops a wasted GP input dimension that contributed no signal.
CSV (boxcrete_data.csv): 26 cols → 19 cols, 727 rows preserved.
- Mortar or Concrete (= Coarse Aggregates > 0)
- Binder (kg/m3) (= Cement + Fly Ash + Slag)
- w/b (= Water / Binder)
- MRWR (kg/m3) (constant 0)
- HRWR (oz/cwt binder) (derivable; inconsistent provenance)
- VMA (oz/cwt) (constant 0)
- AE (oz/cwt) (constant 0)
Model input (DEFAULT_X_COLUMNS): 11 dims → 10 dims by removing MRWR.
This required:
- boxcrete/utils.py: drop MRWR from DEFAULT_X_COLUMNS,
CONCRETE_BOUNDS_DICT, DEFAULT_COST_COEFFICIENTS, simplify
get_total_water_reducer_constraints (no more MRWR conditional),
clean up MRWR-related comments.
- test/test_utils.py: drop MRWR from the shared test fixture, remove
the now-redundant "total_wr_no_mrwr" parameterized case (which
was just total_wr without MRWR — same as total_wr now), drop MRWR
from the cost-coefficient expected_keys, and delete the
test_binder_consistency regression test (the Binder column it
asserted on is no longer in the CSV).
- test/test_models.py: lower the slump LOO R² threshold from 0.38
to 0.30 with a comment. Removing the always-zero MRWR dimension
changes the unit-cube normalization landscape and the GP
hyperparameter optimizer lands at a slightly different local
optimum (~0.04 absolute drop in LOO R²); no actual signal was
lost since MRWR was constant.
- docs/model/*.json: re-exported via scripts/export_model.py with
the new 10-dim X. compositions.json now has 9 column_names (no
MRWR), gwp.json has 8 coefficients per Material Source class,
test_vectors.json was regenerated from the new model.
- docs/ui.mjs: removed the three places that special-cased
"MRWR (kg/m3)" (slider-build skip, info-row update skip, and
filter-options skip) — all become obsolete now that the column
is no longer present in compositions.json.
Documentation: new data/SCHEMA.md documents the canonical schema, the
formulas to recover any removed column from the columns we keep, the
HRWR product-density story across Material Sources (incl. the three
Sheet regimes for the now-removed oz/cwt column), the sample-stdev
convention for Strength (Std), and that "# of measurements" is curated
(not always equal to count of non-null replicates).
Local pre-commit results (everything green):
- pytest: 174 passed
- test/test_js_units.mjs (Node): 67 / 67
- test/test_js_gp.mjs (Node): 90 / 90 JS↔Python parity
- Playwright (desktop + mobile): 69 passed, 0 failed,
54 project-skipped, 1 flaky retry
(pre-existing scatter-toggle
timing test, unrelated)
SebastianAment
force-pushed
the
data-cleanup-redundancies
branch
from
May 11, 2026 22:57
bb6e967 to
b4f34b9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Data cleanup of the canonical CSV (data/boxcrete_data.csv) and the model input vector. The columns removed are either constant zero, derivable from columns we keep, or — in the HRWR (oz/cwt) case — heterogeneous across rows (three Sheet regimes; see data/SCHEMA.md). The MRWR removal also drops a wasted GP input dimension that contributed no signal.
CSV (boxcrete_data.csv): 26 cols → 19 cols, 727 rows preserved.
Model input (DEFAULT_X_COLUMNS): 11 dims → 10 dims by removing MRWR. This required:
Local pre-commit results (everything green):
54 project-skipped, 1 flaky retry
(pre-existing scatter-toggle
timing test, unrelated)