Skip to content

Commit ddc6877

Browse files
SebastianAmentSebastianAment
committed
explorer + ci: V2 strength GP web hookup + artifact / parity gates
Hooks the V2 strength GP up to the in-browser BOxCrete explorer and wires up the CI gates that enforce coherence between the deployed boxcrete model and the published model artifacts. JS-side V2 implementation (docs/explorer): * docs/gp_v2_fast.mjs — pure-JS implementation of the V2 gated-kernel posterior, optimised for the in-browser explorer. * docs/feature_registry.mjs — JS port of the F5_alllog feature builders to keep the JS path byte-identical to the Python path. * docs/gp.mjs / docs/ui.mjs / docs/units.mjs — explorer integration. * docs/generate_mix_analyses.py — produces per-mix analysis pages consumed by the explorer. * docs/model/README.md — documents the docs/model/ artifact layout. * docs/model/strength.json + compositions.json + test_vectors.json — refreshed model artifacts produced by the V2 fit. CI gates: * .github/workflows/strength-parity.yml — guards that the V2 fit factory produces byte-equivalent posteriors to the research-side catalog (test_strength_model_parity.py). * .github/workflows/model-artifacts-coherence.yml — guards that the published docs/model/ artifacts match the boxcrete fit output (catches stale artifacts after model code changes). JS-side regression tests: * test/test_js_strength_v2.mjs — V2 posterior parity (Python vs JS). * test/test_js_physical_constraints.mjs — JS-side f(x, t=0) = 0 guard. * test/test_lengthscales_v2.mjs — JS-side lengthscale parity. * test/test_curve_monotonicity.mjs — strength-curve monotonicity. * test/test_data_freshness.mjs — guards that test_vectors.json is derived from the same data the deployed model was fit on. * test/test_js_ui_smoke.mjs — explorer UI smoke test. * experiments/regenerate_compositions_strength_predictions.mjs + augment_test_vectors_with_gwp_cost.mjs — JS-side artifact regeneration scripts.
1 parent ab13237 commit ddc6877

27 files changed

Lines changed: 4201 additions & 523 deletions

‎.github/workflows/model-artifacts-coherence.yml‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
# Verifies that docs/model/*.json artifacts are coherent with the
2+
# Python V2 strength GP fit. Runs the regen pipeline and asserts no
3+
# diff — any uncommitted drift in strength.json / test_vectors.json /
4+
# compositions.json (e.g., from a manual edit, a partial regen, or
5+
# upstream model code that wasn't followed by a regen run) fails CI.
6+
#
7+
# See docs/model/README.md for the artifact schema + regen workflow.
8+
9+
name: Model Artifacts Coherence
10+
11+
permissions:
12+
contents: read
13+
14+
on:
15+
push:
16+
branches: [main, master]
17+
paths:
18+
- 'boxcrete/strength_model.py'
19+
- 'boxcrete/kernels.py'
20+
- 'boxcrete/likelihoods.py'
21+
- 'boxcrete/priors.py'
22+
- 'boxcrete/features.py'
23+
- 'boxcrete/utils.py'
24+
- 'boxcrete/__init__.py'
25+
- 'data/**'
26+
- 'experiments/regenerate_strength_json.py'
27+
- 'experiments/regenerate_compositions_strength_predictions.mjs'
28+
- 'experiments/augment_test_vectors_with_gwp_cost.mjs'
29+
- 'experiments/regenerate_all_artifacts.sh'
30+
- 'docs/model/**'
31+
- 'docs/feature_registry.mjs'
32+
- 'docs/gp.mjs'
33+
- 'docs/gp_v2_fast.mjs'
34+
- '.github/workflows/model-artifacts-coherence.yml'
35+
pull_request:
36+
branches: [main, master]
37+
paths:
38+
- 'boxcrete/strength_model.py'
39+
- 'boxcrete/kernels.py'
40+
- 'boxcrete/likelihoods.py'
41+
- 'boxcrete/priors.py'
42+
- 'boxcrete/features.py'
43+
- 'boxcrete/utils.py'
44+
- 'boxcrete/__init__.py'
45+
- 'data/**'
46+
- 'experiments/regenerate_strength_json.py'
47+
- 'experiments/regenerate_compositions_strength_predictions.mjs'
48+
- 'experiments/augment_test_vectors_with_gwp_cost.mjs'
49+
- 'experiments/regenerate_all_artifacts.sh'
50+
- 'docs/model/**'
51+
- 'docs/feature_registry.mjs'
52+
- 'docs/gp.mjs'
53+
- 'docs/gp_v2_fast.mjs'
54+
- '.github/workflows/model-artifacts-coherence.yml'
55+
workflow_dispatch: {}
56+
57+
jobs:
58+
regen-idempotency:
59+
runs-on: ubuntu-latest
60+
steps:
61+
- uses: actions/checkout@v4
62+
63+
- uses: actions/setup-python@v5
64+
with:
65+
python-version: '3.12'
66+
cache: 'pip'
67+
68+
- uses: actions/setup-node@v4
69+
with:
70+
node-version: '20'
71+
72+
- name: Install Python deps
73+
run: |
74+
python -m pip install --upgrade pip
75+
pip install -e .
76+
77+
- name: Save committed docs/model/ for later comparison
78+
# Snapshot the JSON artifacts BEFORE regen overwrites them, so
79+
# the post-regen comparison can diff against the committed copy.
80+
# We snapshot only the JSONs (not the .pt) because cross-arch
81+
# determinism on binary state_dicts requires bit-equality which
82+
# we can't expect from a multi-modal MLL fit; the JSON-level
83+
# checks cover the same coverage surface (lengthscales +
84+
# prediction surface) via experiments/check_artifacts_drift.py.
85+
run: |
86+
mkdir -p /tmp/committed_docs_model
87+
cp docs/model/strength.json /tmp/committed_docs_model/
88+
cp docs/model/test_vectors.json /tmp/committed_docs_model/
89+
cp docs/model/compositions.json /tmp/committed_docs_model/
90+
91+
- name: Run regen pipeline
92+
run: bash experiments/regenerate_all_artifacts.sh
93+
94+
- name: Assert artifacts agree with committed copy within tolerance
95+
# Replaces the legacy ``git diff --exit-code docs/model/`` check,
96+
# which was over-strict: it required bit-equality of JSON output
97+
# across architectures, but the V2 strength GP fit goes through
98+
# scipy's L-BFGS-B against a multi-modal MLL surface, and
99+
# different CPU architectures land in different local optima
100+
# (Apple Silicon via qemu-emulated amd64 vs GitHub-runner native
101+
# x86_64 produce ~2x different lengthscales while predicting
102+
# nearly the same surface). The new check tolerates this
103+
# cross-architecture basin divergence (10x ratio band on
104+
# internal hyperparameters) while still catching the original
105+
# failure mode (a stale export typically shifts predictions
106+
# by 100s of psi at OOT compositions). See the docstring of
107+
# ``experiments/check_artifacts_drift.py`` for the full rationale.
108+
run: |
109+
python experiments/check_artifacts_drift.py \
110+
--committed-dir /tmp/committed_docs_model \
111+
--fresh-dir docs/model

‎.github/workflows/notebooks.yml‎

Lines changed: 113 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@ jobs:
4646
- name: Set up Python 3.10
4747
uses: actions/setup-python@v5
4848
with:
49-
python-version: '3.10'
49+
python-version: '3.12'
5050

5151
- name: Cache pip dependencies
5252
uses: actions/cache@v4
@@ -66,34 +66,99 @@ jobs:
6666
run: |
6767
python -m ipykernel install --user --name python3
6868
69+
- name: Diagnostic — kernel + import sanity checks
70+
# Bisect what's hanging in the heavy notebook by running
71+
# progressively-richer probes, each with a tight timeout. If
72+
# the trivial nbconvert (just `print('hello')`) hangs, the
73+
# Jupyter kernel itself is broken on this runner. If the
74+
# imports-only probe hangs, something in the import chain
75+
# (torch, botorch, boxcrete) is hanging on this runner. If
76+
# both succeed quickly but the heavy notebook still cancels,
77+
# the issue is in a specific cell of the notebook.
78+
run: |
79+
set -x
80+
echo "--- jupyter kernelspec list ---"
81+
jupyter kernelspec list --json | python -m json.tool
82+
echo
83+
echo "--- trivial nbconvert (just print) ---"
84+
cat > /tmp/trivial.ipynb << 'NBEOF'
85+
{
86+
"cells": [
87+
{"cell_type": "code", "metadata": {}, "outputs": [], "source": ["print('hello from kernel')"], "execution_count": null}
88+
],
89+
"metadata": {"kernelspec": {"display_name": "Python 3", "language": "python", "name": "python3"}},
90+
"nbformat": 4, "nbformat_minor": 5
91+
}
92+
NBEOF
93+
time timeout 90 python -m jupyter nbconvert \
94+
--to notebook \
95+
--execute \
96+
--inplace \
97+
--ExecutePreprocessor.timeout=60 \
98+
--ExecutePreprocessor.kernel_name=python3 \
99+
/tmp/trivial.ipynb
100+
echo "--- trivial.ipynb output cells ---"
101+
python -c "import json; nb=json.load(open('/tmp/trivial.ipynb')); print([o for c in nb['cells'] for o in c.get('outputs', [])])"
102+
echo
103+
echo "--- imports-only probe (heavy notebook's cell 1) ---"
104+
time timeout 90 python -c "
105+
import os
106+
import botorch
107+
import matplotlib.pyplot as plt
108+
import pandas as pd
109+
import torch
110+
from boxcrete.units import STRENGTH_DISPLAY_SCALE, strength_label
111+
print('imports OK; torch=', torch.__version__, 'botorch=', botorch.__version__)
112+
"
113+
echo
114+
echo "--- imports + first-fit probe (boxcrete fit, timed) ---"
115+
time timeout 120 python -c "
116+
import time
117+
t0 = time.perf_counter()
118+
from boxcrete.strength_data import default_strength_data
119+
from boxcrete.strength_model import fit_strength_gp
120+
data = default_strength_data()
121+
gp, *_ = fit_strength_gp(data.X, data.Y, X_bounds=data.X_bounds)
122+
print(f'fit OK in {time.perf_counter()-t0:.1f}s')
123+
"
124+
69125
- name: Execute mode-dependent notebooks (${{ matrix.optimization-mode }}, cost=${{ matrix.include-cost }})
70126
env:
71127
BOXCRETE_OPTIMIZATION_MODE: ${{ matrix.optimization-mode }}
72128
BOXCRETE_INCLUDE_COST: ${{ matrix.include-cost }}
73129
run: |
130+
# Diagnostics so we can see *why* nbconvert dies on the GitHub
131+
# Linux runner. Local Mac runs of this notebook complete cleanly
132+
# in ~3 min; CI silently kills the kernel mid-execution. The
133+
# block below prints memory + disk before, captures nbconvert
134+
# stdout AND stderr explicitly (the previous wrapper relied on
135+
# inherited streams which were getting truncated when the
136+
# kernel was SIGKILL'd), and on failure dumps OOM evidence
137+
# from dmesg and the negative returncode signal name.
138+
echo "=== runner resources before nbconvert ==="
139+
free -h || true
140+
df -h /home/runner/work || true
141+
uname -a || true
142+
python -c "import torch; print('torch:', torch.__version__, 'threads:', torch.get_num_threads())" || true
143+
74144
python - << 'PYEOF'
75-
import subprocess, sys, os
145+
import os, signal, subprocess, sys
76146
from pathlib import Path
77147
78-
# Only notebooks that use BOXCRETE_OPTIMIZATION_MODE
79-
MODE_DEPENDENT = [
80-
"notebooks/prediction_and_optimization_tutorial.ipynb",
81-
]
82-
148+
MODE_DEPENDENT = ["notebooks/prediction_and_optimization_tutorial.ipynb"]
83149
mode = os.environ.get("BOXCRETE_OPTIMIZATION_MODE", "concrete")
84150
failed = []
85151
86152
for nb in MODE_DEPENDENT:
87-
nb_path = Path(nb)
88-
if not nb_path.exists():
153+
if not Path(nb).exists():
89154
print(f"⚠️ Skipping (not found): {nb}")
90155
continue
91-
92156
print(f"{'=' * 40}")
93157
print(f"Executing ({mode}): {nb}")
94158
print(f"{'=' * 40}")
159+
sys.stdout.flush()
95160
96-
result = subprocess.run(
161+
proc = subprocess.run(
97162
[
98163
sys.executable, "-m", "jupyter", "nbconvert",
99164
"--to", "notebook",
@@ -103,15 +168,48 @@ jobs:
103168
"--ExecutePreprocessor.kernel_name=python3",
104169
str(nb),
105170
],
171+
# Stream stdout/stderr live to the action log instead
172+
# of buffering them in memory. The previous attempt
173+
# used capture_output=True with --debug, which made
174+
# nbconvert emit a huge debug stream that filled the
175+
# wrapper's RSS until the wrapper itself got killed
176+
# mid-subprocess.run before it could print any
177+
# diagnostic. Streaming means whatever nbconvert
178+
# prints (including cell tracebacks) shows up in the
179+
# action log even if the kernel is killed mid-cell.
106180
capture_output=False,
107181
)
108-
109-
if result.returncode != 0:
182+
# Returncode is whatever the child reports; -N means
183+
# killed by signal N (e.g., -9 = SIGKILL = OOM-killer).
184+
print(f"--- nbconvert returncode: {proc.returncode} ---")
185+
if proc.returncode < 0:
186+
try:
187+
sig = signal.Signals(-proc.returncode)
188+
print(f" nbconvert was terminated by {sig.name} (signal {-proc.returncode})")
189+
except (ValueError, AttributeError):
190+
print(f" nbconvert was terminated by signal {-proc.returncode}")
191+
192+
if proc.returncode != 0:
110193
failed.append(str(nb))
111194
print(f"❌ Failed ({mode}): {nb}")
195+
# OOM / kernel-killed evidence (Linux runner only).
196+
print("--- dmesg | tail (OOM evidence if any) ---")
197+
try:
198+
d = subprocess.run(
199+
["sudo", "dmesg", "--ctime"],
200+
capture_output=True, text=True, timeout=5,
201+
)
202+
tail = d.stdout.splitlines()[-50:]
203+
print("\n".join(tail) if tail else "(dmesg empty)")
204+
except Exception as e:
205+
print(f"(dmesg unavailable: {e})")
206+
print("--- runner resources after failure ---")
207+
subprocess.run(["free", "-h"], check=False)
208+
subprocess.run(["df", "-h", "/home/runner/work"], check=False)
112209
else:
113210
print(f"✅ Successfully executed ({mode}): {nb}")
114211
print()
212+
sys.stdout.flush()
115213
116214
if failed:
117215
print(f"\n{len(failed)} notebook(s) failed in {mode} mode:")
@@ -144,7 +242,7 @@ jobs:
144242
- name: Set up Python 3.10
145243
uses: actions/setup-python@v5
146244
with:
147-
python-version: '3.10'
245+
python-version: '3.12'
148246

149247
- name: Cache pip dependencies
150248
uses: actions/cache@v4
@@ -232,7 +330,7 @@ jobs:
232330
- name: Set up Python
233331
uses: actions/setup-python@v5
234332
with:
235-
python-version: '3.10'
333+
python-version: '3.12'
236334

237335
- name: Install dependencies
238336
run: |

‎.github/workflows/strength-parity.yml‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
# Verifies that boxcrete.fit_strength_gp (production) and the
2+
# experiments.model_variant_study variant catalog produce byte-equivalent
3+
# posteriors for the V2 strength GP configuration, AND that
4+
# boxcrete.load_pretrained_strength_gp() faithfully reconstructs the
5+
# deployed V2 strength GP from docs/model/strength_model.pt.
6+
#
7+
# Runs:
8+
# - test/test_strength_model_parity.py (atol=1e-8 / rtol=1e-6, ~3s)
9+
# - test/test_pretrained_loader_fidelity.py (atol=1e-5 / 1e-3, ~3s)
10+
#
11+
# See those test files for the rationale.
12+
13+
name: Strength GP Parity
14+
15+
permissions:
16+
contents: read
17+
18+
on:
19+
push:
20+
branches: [main, master]
21+
paths:
22+
- 'boxcrete/strength_model.py'
23+
- 'boxcrete/kernels.py'
24+
- 'boxcrete/likelihoods.py'
25+
- 'boxcrete/priors.py'
26+
- 'boxcrete/features.py'
27+
- 'boxcrete/__init__.py'
28+
- 'experiments/model_variant_study.py'
29+
- 'experiments/_research_features.py'
30+
- 'test/test_strength_model_parity.py'
31+
- 'test/test_pretrained_loader_fidelity.py'
32+
- 'docs/model/strength.json'
33+
- 'docs/model/strength_model.pt'
34+
- 'docs/model/test_vectors.json'
35+
- '.github/workflows/strength-parity.yml'
36+
pull_request:
37+
branches: [main, master]
38+
paths:
39+
- 'boxcrete/strength_model.py'
40+
- 'boxcrete/kernels.py'
41+
- 'boxcrete/likelihoods.py'
42+
- 'boxcrete/priors.py'
43+
- 'boxcrete/features.py'
44+
- 'boxcrete/__init__.py'
45+
- 'experiments/model_variant_study.py'
46+
- 'experiments/_research_features.py'
47+
- 'test/test_strength_model_parity.py'
48+
- 'test/test_pretrained_loader_fidelity.py'
49+
- 'docs/model/strength.json'
50+
- 'docs/model/strength_model.pt'
51+
- 'docs/model/test_vectors.json'
52+
- '.github/workflows/strength-parity.yml'
53+
workflow_dispatch: {}
54+
55+
jobs:
56+
parity:
57+
runs-on: ubuntu-latest
58+
steps:
59+
- uses: actions/checkout@v4
60+
61+
- uses: actions/setup-python@v5
62+
with:
63+
python-version: '3.12'
64+
cache: 'pip'
65+
66+
- name: Install Python deps
67+
run: |
68+
python -m pip install --upgrade pip
69+
pip install -e .
70+
pip install pytest
71+
72+
- name: Run parity test
73+
run: python -m pytest test/test_strength_model_parity.py -v --tb=short
74+
75+
- name: Run pretrained-loader fidelity test
76+
run: python -m pytest test/test_pretrained_loader_fidelity.py -v --tb=short

0 commit comments

Comments
 (0)