explorer + ci: V2 strength GP web hookup + artifact / parity gates

Hooks the V2 strength GP up to the in-browser BOxCrete explorer and
wires up the CI gates that enforce coherence between the deployed
boxcrete model and the published model artifacts.

JS-side V2 implementation (docs/explorer):

  * docs/gp_v2_fast.mjs — pure-JS implementation of the V2 gated-kernel
    posterior, optimised for the in-browser explorer.
  * docs/feature_registry.mjs — JS port of the F5_alllog feature
    builders to keep the JS path byte-identical to the Python path.
  * docs/gp.mjs / docs/ui.mjs / docs/units.mjs — explorer integration.
  * docs/generate_mix_analyses.py — produces per-mix analysis pages
    consumed by the explorer.
  * docs/model/README.md — documents the docs/model/ artifact layout.
  * docs/model/strength.json + compositions.json + test_vectors.json —
    refreshed model artifacts produced by the V2 fit.

CI gates:

  * .github/workflows/strength-parity.yml — guards that the V2 fit
    factory produces byte-equivalent posteriors to the research-side
    catalog (test_strength_model_parity.py).
  * .github/workflows/model-artifacts-coherence.yml — guards that the
    published docs/model/ artifacts match the boxcrete fit output
    (catches stale artifacts after model code changes).

JS-side regression tests:

  * test/test_js_strength_v2.mjs — V2 posterior parity (Python vs JS).
  * test/test_js_physical_constraints.mjs — JS-side f(x, t=0) = 0 guard.
  * test/test_lengthscales_v2.mjs — JS-side lengthscale parity.
  * test/test_curve_monotonicity.mjs — strength-curve monotonicity.
  * test/test_data_freshness.mjs — guards that test_vectors.json is
    derived from the same data the deployed model was fit on.
  * test/test_js_ui_smoke.mjs — explorer UI smoke test.
  * experiments/regenerate_compositions_strength_predictions.mjs +
    augment_test_vectors_with_gwp_cost.mjs — JS-side artifact
    regeneration scripts.

Author: SebastianAment

.github/workflows/model-artifacts-coherence.yml

@@ -0,0 +1,111 @@
+# Verifies that docs/model/*.json artifacts are coherent with the
+# Python V2 strength GP fit. Runs the regen pipeline and asserts no
+# diff — any uncommitted drift in strength.json / test_vectors.json /
+# compositions.json (e.g., from a manual edit, a partial regen, or
+# upstream model code that wasn't followed by a regen run) fails CI.
+#
+# See docs/model/README.md for the artifact schema + regen workflow.
+
+name: Model Artifacts Coherence
+
+permissions:
+  contents: read
+
+on:
+  push:
+    branches: [main, master]
+    paths:
+      - 'boxcrete/strength_model.py'
+      - 'boxcrete/kernels.py'
+      - 'boxcrete/likelihoods.py'
+      - 'boxcrete/priors.py'
+      - 'boxcrete/features.py'
+      - 'boxcrete/utils.py'
+      - 'boxcrete/__init__.py'
+      - 'data/**'
+      - 'experiments/regenerate_strength_json.py'
+      - 'experiments/regenerate_compositions_strength_predictions.mjs'
+      - 'experiments/augment_test_vectors_with_gwp_cost.mjs'
+      - 'experiments/regenerate_all_artifacts.sh'
+      - 'docs/model/**'
+      - 'docs/feature_registry.mjs'
+      - 'docs/gp.mjs'
+      - 'docs/gp_v2_fast.mjs'
+      - '.github/workflows/model-artifacts-coherence.yml'
+  pull_request:
+    branches: [main, master]
+    paths:
+      - 'boxcrete/strength_model.py'
+      - 'boxcrete/kernels.py'
+      - 'boxcrete/likelihoods.py'
+      - 'boxcrete/priors.py'
+      - 'boxcrete/features.py'
+      - 'boxcrete/utils.py'
+      - 'boxcrete/__init__.py'
+      - 'data/**'
+      - 'experiments/regenerate_strength_json.py'
+      - 'experiments/regenerate_compositions_strength_predictions.mjs'
+      - 'experiments/augment_test_vectors_with_gwp_cost.mjs'
+      - 'experiments/regenerate_all_artifacts.sh'
+      - 'docs/model/**'
+      - 'docs/feature_registry.mjs'
+      - 'docs/gp.mjs'
+      - 'docs/gp_v2_fast.mjs'
+      - '.github/workflows/model-artifacts-coherence.yml'
+  workflow_dispatch: {}
+
+jobs:
+  regen-idempotency:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+          cache: 'pip'
+
+      - uses: actions/setup-node@v4
+        with:
+          node-version: '20'
+
+      - name: Install Python deps
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e .
+
+      - name: Save committed docs/model/ for later comparison
+        # Snapshot the JSON artifacts BEFORE regen overwrites them, so
+        # the post-regen comparison can diff against the committed copy.
+        # We snapshot only the JSONs (not the .pt) because cross-arch
+        # determinism on binary state_dicts requires bit-equality which
+        # we can't expect from a multi-modal MLL fit; the JSON-level
+        # checks cover the same coverage surface (lengthscales +
+        # prediction surface) via experiments/check_artifacts_drift.py.
+        run: |
+          mkdir -p /tmp/committed_docs_model
+          cp docs/model/strength.json /tmp/committed_docs_model/
+          cp docs/model/test_vectors.json /tmp/committed_docs_model/
+          cp docs/model/compositions.json /tmp/committed_docs_model/
+
+      - name: Run regen pipeline
+        run: bash experiments/regenerate_all_artifacts.sh
+
+      - name: Assert artifacts agree with committed copy within tolerance
+        # Replaces the legacy ``git diff --exit-code docs/model/`` check,
+        # which was over-strict: it required bit-equality of JSON output
+        # across architectures, but the V2 strength GP fit goes through
+        # scipy's L-BFGS-B against a multi-modal MLL surface, and
+        # different CPU architectures land in different local optima
+        # (Apple Silicon via qemu-emulated amd64 vs GitHub-runner native
+        # x86_64 produce ~2x different lengthscales while predicting
+        # nearly the same surface). The new check tolerates this
+        # cross-architecture basin divergence (10x ratio band on
+        # internal hyperparameters) while still catching the original
+        # failure mode (a stale export typically shifts predictions
+        # by 100s of psi at OOT compositions). See the docstring of
+        # ``experiments/check_artifacts_drift.py`` for the full rationale.
+        run: |
+          python experiments/check_artifacts_drift.py \
+            --committed-dir /tmp/committed_docs_model \
+            --fresh-dir docs/model

.github/workflows/notebooks.yml

@@ -46,7 +46,7 @@ jobs:
       - name: Set up Python 3.10
         uses: actions/setup-python@v5
         with:
-          python-version: '3.10'
+          python-version: '3.12'
 
       - name: Cache pip dependencies
         uses: actions/cache@v4
@@ -66,34 +66,120 @@ jobs:
         run: |
           python -m ipykernel install --user --name python3
 
+      - name: Diagnostic — kernel + import sanity checks
+        # Bisect what's hanging in the heavy notebook by running
+        # progressively-richer probes, each with a tight timeout. If
+        # the trivial nbconvert (just `print('hello')`) hangs, the
+        # Jupyter kernel itself is broken on this runner. If the
+        # imports-only probe hangs, something in the import chain
+        # (torch, botorch, boxcrete) is hanging on this runner. If
+        # both succeed quickly but the heavy notebook still cancels,
+        # the issue is in a specific cell of the notebook.
+        run: |
+          set -x
+          echo "--- jupyter kernelspec list ---"
+          jupyter kernelspec list --json | python -m json.tool
+          echo
+          echo "--- trivial nbconvert (just print) ---"
+          cat > /tmp/trivial.ipynb << 'NBEOF'
+          {
+           "cells": [
+            {"cell_type": "code", "metadata": {}, "outputs": [], "source": ["print('hello from kernel')"], "execution_count": null}
+           ],
+           "metadata": {"kernelspec": {"display_name": "Python 3", "language": "python", "name": "python3"}},
+           "nbformat": 4, "nbformat_minor": 5
+          }
+          NBEOF
+          time timeout 90 python -m jupyter nbconvert \
+              --to notebook \
+              --execute \
+              --inplace \
+              --ExecutePreprocessor.timeout=60 \
+              --ExecutePreprocessor.kernel_name=python3 \
+              /tmp/trivial.ipynb
+          echo "--- trivial.ipynb output cells ---"
+          python -c "import json; nb=json.load(open('/tmp/trivial.ipynb')); print([o for c in nb['cells'] for o in c.get('outputs', [])])"
+          echo
+          echo "--- imports-only probe (heavy notebook's cell 1) ---"
+          time timeout 90 python -c "
+          import os
+          import botorch
+          import matplotlib.pyplot as plt
+          import pandas as pd
+          import torch
+          from boxcrete.units import STRENGTH_DISPLAY_SCALE, strength_label
+          print('imports OK; torch=', torch.__version__, 'botorch=', botorch.__version__)
+          "
+          echo
+          echo "--- cell-by-cell exec of heavy notebook (no Jupyter kernel) ---"
+          # Run every code cell sequentially in a single Python process,
+          # printing markers + timings between cells. This bypasses
+          # ipykernel entirely; whichever cell takes >> 30s or hangs
+          # will surface as the last `=== cell N ===` line printed
+          # before the step is canceled. `python -u` for unbuffered
+          # stdout so each cell's marker hits the action log live.
+          time timeout 300 python -u - << 'PYEOF'
+          import json, sys, time
+          nb = json.load(open('notebooks/prediction_and_optimization_tutorial.ipynb'))
+          ns = {'__name__': '__main__'}
+          for i, cell in enumerate(nb['cells']):
+              if cell['cell_type'] != 'code':
+                  continue
+              src = ''.join(cell['source'])
+              if not src.strip() or src.lstrip().startswith('%'):
+                  # Skip empty cells and IPython magics (e.g., %matplotlib).
+                  continue
+              first_line = src.lstrip().splitlines()[0][:80]
+              print(f'=== cell {i}: {first_line!r} ===', flush=True)
+              t0 = time.perf_counter()
+              try:
+                  exec(compile(src, f'<cell {i}>', 'exec'), ns)
+              except SystemExit:
+                  raise
+              except BaseException as e:
+                  print(f'    cell {i} RAISED {type(e).__name__}: {e}', flush=True)
+                  raise
+              print(f'    cell {i} done in {time.perf_counter()-t0:.2f}s', flush=True)
+          print('=== ALL CELLS DONE ===', flush=True)
+          PYEOF
+
       - name: Execute mode-dependent notebooks (${{ matrix.optimization-mode }}, cost=${{ matrix.include-cost }})
         env:
           BOXCRETE_OPTIMIZATION_MODE: ${{ matrix.optimization-mode }}
           BOXCRETE_INCLUDE_COST: ${{ matrix.include-cost }}
         run: |
+          # Diagnostics so we can see *why* nbconvert dies on the GitHub
+          # Linux runner. Local Mac runs of this notebook complete cleanly
+          # in ~3 min; CI silently kills the kernel mid-execution. The
+          # block below prints memory + disk before, captures nbconvert
+          # stdout AND stderr explicitly (the previous wrapper relied on
+          # inherited streams which were getting truncated when the
+          # kernel was SIGKILL'd), and on failure dumps OOM evidence
+          # from dmesg and the negative returncode signal name.
+          echo "=== runner resources before nbconvert ==="
+          free -h || true
+          df -h /home/runner/work || true
+          uname -a || true
+          python -c "import torch; print('torch:', torch.__version__, 'threads:', torch.get_num_threads())" || true
+
           python - << 'PYEOF'
-          import subprocess, sys, os
+          import os, signal, subprocess, sys
           from pathlib import Path
 
-          # Only notebooks that use BOXCRETE_OPTIMIZATION_MODE
-          MODE_DEPENDENT = [
-              "notebooks/prediction_and_optimization_tutorial.ipynb",
-          ]
-
+          MODE_DEPENDENT = ["notebooks/prediction_and_optimization_tutorial.ipynb"]
           mode = os.environ.get("BOXCRETE_OPTIMIZATION_MODE", "concrete")
           failed = []
 
           for nb in MODE_DEPENDENT:
-              nb_path = Path(nb)
-              if not nb_path.exists():
+              if not Path(nb).exists():
                   print(f"⚠️  Skipping (not found): {nb}")
                   continue
-
               print(f"{'=' * 40}")
               print(f"Executing ({mode}): {nb}")
               print(f"{'=' * 40}")
+              sys.stdout.flush()
 
-              result = subprocess.run(
+              proc = subprocess.run(
                   [
                       sys.executable, "-m", "jupyter", "nbconvert",
                       "--to", "notebook",
@@ -103,15 +189,48 @@ jobs:
                       "--ExecutePreprocessor.kernel_name=python3",
                       str(nb),
                   ],
+                  # Stream stdout/stderr live to the action log instead
+                  # of buffering them in memory. The previous attempt
+                  # used capture_output=True with --debug, which made
+                  # nbconvert emit a huge debug stream that filled the
+                  # wrapper's RSS until the wrapper itself got killed
+                  # mid-subprocess.run before it could print any
+                  # diagnostic. Streaming means whatever nbconvert
+                  # prints (including cell tracebacks) shows up in the
+                  # action log even if the kernel is killed mid-cell.
                   capture_output=False,
               )
-
-              if result.returncode != 0:
+              # Returncode is whatever the child reports; -N means
+              # killed by signal N (e.g., -9 = SIGKILL = OOM-killer).
+              print(f"--- nbconvert returncode: {proc.returncode} ---")
+              if proc.returncode < 0:
+                  try:
+                      sig = signal.Signals(-proc.returncode)
+                      print(f"    nbconvert was terminated by {sig.name} (signal {-proc.returncode})")
+                  except (ValueError, AttributeError):
+                      print(f"    nbconvert was terminated by signal {-proc.returncode}")
+
+              if proc.returncode != 0:
                   failed.append(str(nb))
                   print(f"❌ Failed ({mode}): {nb}")
+                  # OOM / kernel-killed evidence (Linux runner only).
+                  print("--- dmesg | tail (OOM evidence if any) ---")
+                  try:
+                      d = subprocess.run(
+                          ["sudo", "dmesg", "--ctime"],
+                          capture_output=True, text=True, timeout=5,
+                      )
+                      tail = d.stdout.splitlines()[-50:]
+                      print("\n".join(tail) if tail else "(dmesg empty)")
+                  except Exception as e:
+                      print(f"(dmesg unavailable: {e})")
+                  print("--- runner resources after failure ---")
+                  subprocess.run(["free", "-h"], check=False)
+                  subprocess.run(["df", "-h", "/home/runner/work"], check=False)
               else:
                   print(f"✅ Successfully executed ({mode}): {nb}")
               print()
+              sys.stdout.flush()
 
           if failed:
               print(f"\n{len(failed)} notebook(s) failed in {mode} mode:")
@@ -144,7 +263,7 @@ jobs:
       - name: Set up Python 3.10
         uses: actions/setup-python@v5
         with:
-          python-version: '3.10'
+          python-version: '3.12'
 
       - name: Cache pip dependencies
         uses: actions/cache@v4
@@ -232,7 +351,7 @@ jobs:
       - name: Set up Python
         uses: actions/setup-python@v5
         with:
-          python-version: '3.10'
+          python-version: '3.12'
 
       - name: Install dependencies
         run: |

.github/workflows/strength-parity.yml

@@ -0,0 +1,76 @@
+# Verifies that boxcrete.fit_strength_gp (production) and the
+# experiments.model_variant_study variant catalog produce byte-equivalent
+# posteriors for the V2 strength GP configuration, AND that
+# boxcrete.load_pretrained_strength_gp() faithfully reconstructs the
+# deployed V2 strength GP from docs/model/strength_model.pt.
+#
+# Runs:
+#   - test/test_strength_model_parity.py (atol=1e-8 / rtol=1e-6, ~3s)
+#   - test/test_pretrained_loader_fidelity.py (atol=1e-5 / 1e-3, ~3s)
+#
+# See those test files for the rationale.
+
+name: Strength GP Parity
+
+permissions:
+  contents: read
+
+on:
+  push:
+    branches: [main, master]
+    paths:
+      - 'boxcrete/strength_model.py'
+      - 'boxcrete/kernels.py'
+      - 'boxcrete/likelihoods.py'
+      - 'boxcrete/priors.py'
+      - 'boxcrete/features.py'
+      - 'boxcrete/__init__.py'
+      - 'experiments/model_variant_study.py'
+      - 'experiments/_research_features.py'
+      - 'test/test_strength_model_parity.py'
+      - 'test/test_pretrained_loader_fidelity.py'
+      - 'docs/model/strength.json'
+      - 'docs/model/strength_model.pt'
+      - 'docs/model/test_vectors.json'
+      - '.github/workflows/strength-parity.yml'
+  pull_request:
+    branches: [main, master]
+    paths:
+      - 'boxcrete/strength_model.py'
+      - 'boxcrete/kernels.py'
+      - 'boxcrete/likelihoods.py'
+      - 'boxcrete/priors.py'
+      - 'boxcrete/features.py'
+      - 'boxcrete/__init__.py'
+      - 'experiments/model_variant_study.py'
+      - 'experiments/_research_features.py'
+      - 'test/test_strength_model_parity.py'
+      - 'test/test_pretrained_loader_fidelity.py'
+      - 'docs/model/strength.json'
+      - 'docs/model/strength_model.pt'
+      - 'docs/model/test_vectors.json'
+      - '.github/workflows/strength-parity.yml'
+  workflow_dispatch: {}
+
+jobs:
+  parity:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+          cache: 'pip'
+
+      - name: Install Python deps
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e .
+          pip install pytest
+
+      - name: Run parity test
+        run: python -m pytest test/test_strength_model_parity.py -v --tb=short
+
+      - name: Run pretrained-loader fidelity test
+        run: python -m pytest test/test_pretrained_loader_fidelity.py -v --tb=short