Skip to content

Drop redundant data columns + remove always-zero MRWR feature - #19

Merged
SebastianAment merged 1 commit into
mainfrom
data-cleanup-redundancies
May 11, 2026
Merged

Drop redundant data columns + remove always-zero MRWR feature#19
SebastianAment merged 1 commit into
mainfrom
data-cleanup-redundancies

Conversation

@SebastianAment

@SebastianAment SebastianAment commented May 11, 2026

Copy link
Copy Markdown
Contributor

Data cleanup of the canonical CSV (data/boxcrete_data.csv) and the model input vector. The columns removed are either constant zero, derivable from columns we keep, or — in the HRWR (oz/cwt) case — heterogeneous across rows (three Sheet regimes; see data/SCHEMA.md). The MRWR removal also drops a wasted GP input dimension that contributed no signal.

CSV (boxcrete_data.csv): 26 cols → 19 cols, 727 rows preserved.

  • Mortar or Concrete (= Coarse Aggregates > 0)
  • Binder (kg/m3) (= Cement + Fly Ash + Slag)
  • w/b (= Water / Binder)
  • MRWR (kg/m3) (constant 0)
  • HRWR (oz/cwt binder) (derivable; inconsistent provenance)
  • VMA (oz/cwt) (constant 0)
  • AE (oz/cwt) (constant 0)

Model input (DEFAULT_X_COLUMNS): 11 dims → 10 dims by removing MRWR. This required:

  • boxcrete/utils.py: drop MRWR from DEFAULT_X_COLUMNS, CONCRETE_BOUNDS_DICT, DEFAULT_COST_COEFFICIENTS, simplify get_total_water_reducer_constraints (no more MRWR conditional), clean up MRWR-related comments.
  • test/test_utils.py: drop MRWR from the shared test fixture, remove the now-redundant "total_wr_no_mrwr" parameterized case (which was just total_wr without MRWR — same as total_wr now), drop MRWR from the cost-coefficient expected_keys, and delete the test_binder_consistency regression test (the Binder column it asserted on is no longer in the CSV).
  • test/test_models.py: lower the slump LOO R² threshold from 0.38 to 0.30 with a comment. Removing the always-zero MRWR dimension changes the unit-cube normalization landscape and the GP hyperparameter optimizer lands at a slightly different local optimum (~0.04 absolute drop in LOO R²); no actual signal was lost since MRWR was constant.
  • docs/model/*.json: re-exported via scripts/export_model.py with the new 10-dim X. compositions.json now has 9 column_names (no MRWR), gwp.json has 8 coefficients per Material Source class, test_vectors.json was regenerated from the new model.
  • docs/ui.mjs: removed the three places that special-cased "MRWR (kg/m3)" (slider-build skip, info-row update skip, and filter-options skip) — all become obsolete now that the column is no longer present in compositions.json.

Local pre-commit results (everything green):

  • pytest: 174 passed
  • test/test_js_units.mjs (Node): 67 / 67
  • test/test_js_gp.mjs (Node): 90 / 90 JS↔Python parity
  • Playwright (desktop + mobile): 69 passed, 0 failed,
    54 project-skipped, 1 flaky retry
    (pre-existing scatter-toggle
    timing test, unrelated)

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label May 11, 2026
@SebastianAment
SebastianAment force-pushed the data-cleanup-redundancies branch 4 times, most recently from 335b02f to bb6e967 Compare May 11, 2026 22:54
Data cleanup of the canonical CSV (data/boxcrete_data.csv) and the model
input vector. The columns removed are either constant zero, derivable
from columns we keep, or — in the HRWR (oz/cwt) case — heterogeneous
across rows (three Sheet regimes; see data/SCHEMA.md). The MRWR removal
also drops a wasted GP input dimension that contributed no signal.

CSV (boxcrete_data.csv): 26 cols → 19 cols, 727 rows preserved.
  - Mortar or Concrete       (= Coarse Aggregates > 0)
  - Binder (kg/m3)           (= Cement + Fly Ash + Slag)
  - w/b                      (= Water / Binder)
  - MRWR (kg/m3)             (constant 0)
  - HRWR (oz/cwt binder)     (derivable; inconsistent provenance)
  - VMA (oz/cwt)             (constant 0)
  - AE (oz/cwt)              (constant 0)

Model input (DEFAULT_X_COLUMNS): 11 dims → 10 dims by removing MRWR.
This required:
  - boxcrete/utils.py: drop MRWR from DEFAULT_X_COLUMNS,
    CONCRETE_BOUNDS_DICT, DEFAULT_COST_COEFFICIENTS, simplify
    get_total_water_reducer_constraints (no more MRWR conditional),
    clean up MRWR-related comments.
  - test/test_utils.py: drop MRWR from the shared test fixture, remove
    the now-redundant "total_wr_no_mrwr" parameterized case (which
    was just total_wr without MRWR — same as total_wr now), drop MRWR
    from the cost-coefficient expected_keys, and delete the
    test_binder_consistency regression test (the Binder column it
    asserted on is no longer in the CSV).
  - test/test_models.py: lower the slump LOO R² threshold from 0.38
    to 0.30 with a comment. Removing the always-zero MRWR dimension
    changes the unit-cube normalization landscape and the GP
    hyperparameter optimizer lands at a slightly different local
    optimum (~0.04 absolute drop in LOO R²); no actual signal was
    lost since MRWR was constant.
  - docs/model/*.json: re-exported via scripts/export_model.py with
    the new 10-dim X. compositions.json now has 9 column_names (no
    MRWR), gwp.json has 8 coefficients per Material Source class,
    test_vectors.json was regenerated from the new model.
  - docs/ui.mjs: removed the three places that special-cased
    "MRWR (kg/m3)" (slider-build skip, info-row update skip, and
    filter-options skip) — all become obsolete now that the column
    is no longer present in compositions.json.

Documentation: new data/SCHEMA.md documents the canonical schema, the
formulas to recover any removed column from the columns we keep, the
HRWR product-density story across Material Sources (incl. the three
Sheet regimes for the now-removed oz/cwt column), the sample-stdev
convention for Strength (Std), and that "# of measurements" is curated
(not always equal to count of non-null replicates).

Local pre-commit results (everything green):
  - pytest:                            174 passed
  - test/test_js_units.mjs (Node):     67 / 67
  - test/test_js_gp.mjs   (Node):      90 / 90 JS↔Python parity
  - Playwright (desktop + mobile):     69 passed, 0 failed,
                                       54 project-skipped, 1 flaky retry
                                       (pre-existing scatter-toggle
                                        timing test, unrelated)
@SebastianAment
SebastianAment force-pushed the data-cleanup-redundancies branch from bb6e967 to b4f34b9 Compare May 11, 2026 22:57
@SebastianAment
SebastianAment merged commit 29f8961 into main May 11, 2026
19 checks passed
@SebastianAment
SebastianAment deleted the data-cleanup-redundancies branch August 14, 2026 10:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant