Skip to content

Commit c430ebf

Browse files
committed
Hot-path simplify: cancel -> together (~200x speedup at L=3) + experiment registry
The headline change is one line in nbody/exact_growth_nbody.py simplify_generator(): swap cancel(expr) for together(expr). Cancel was eagerly expanding L=3 brackets to 100k-500k summands then forcing every downstream op (next bracket, lambdify, SVD eval) to walk that tree. Together returns mathematically equivalent factored form with 1-12 summands. Schwarzschild composite L<=3 dropped from ">5h, killed at 16 GB" to "90s, <0.2 GB" with the canonical [3, 6, 17, 116] dim sequence. Validated by an 8-stage paranoid suite (bench_flint/validation_summary.md): 8 potentials, 3 spatial dimensions, mass invariance, charged Coulomb, N=4, plus cross-check against the project's pinned dataset. Every numerically clean case matches canonical bit-for-bit; remaining mismatches are the documented float64-SVD precision wall at extreme mass ratios (cancel hits identical walls). Other changes in this commit: test_regression.py - Repaired (3 of 4 tests were silently failing on a non-existent compute_exact_growth method) - Extended with a Schwarzschild composite L<=3 case that guards future regressions of the simplify swap nbody/exact_growth_nbody.py - Intra-level checkpointing for crash resilience: save_checkpoint gains a partial=True path and load_checkpoint restores computed_pairs so a crash mid-L3 loses at most checkpoint_every brackets instead of the whole level nbody/run_schwarzschild.py - New: Schwarzschild effective-potential dimension-sequence runner with per-(M,L) checkpoint isolation, atomic per-point JSON, --resume support, and --checkpoint-every results/schwarzschild/ - L2 full grid (30 (M,L) points): all give [3, 6, 17]; first GR result for the project at L=2 registry/, scripts/registry_*.py, docs/registry_status.md - New machine-readable index of every script in the project (164 entries). YAML source of truth, JSON Schema validation, atomic bootstrap/curate/new CLIs, markdown + JSON renderers, CI lint job in .github/workflows/regression.yml docs/supplemental_triage.md - Triage of the 3body-supplemental folder: which suggestions are DONE, PENDING, REFERENCE, or NOT WORTH bench_flint/ - Validation infrastructure: watchdog (psutil hard timeout + RSS cap), validate_simplify_patch.py orchestrator, validate_simplify_worker.py per-case runner, stage7_dataset_crosscheck.py - Per-case JSON results, summaries, the FLINT investigation bench/probe that proved python-flint doesn't help here yet (DMP_Flint multivariate is still in-development upstream) .gitignore - bench_flint/_*, bench_flint/venv/, intermediate logs, and nbody/checkpoints_schwarzschild/ excluded as regeneratable Reversion path: revert this commit, or just revert nbody/exact_growth_nbody.py if only the engine change needs to back out. Made-with: Cursor
1 parent 3cc0381 commit c430ebf

42 files changed

Lines changed: 9435 additions & 31 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/workflows/regression.yml

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,8 @@ on:
99
- "3d/exact_growth_nd.py"
1010
- "requirements.txt"
1111
- "test_regression.py"
12+
- "registry/**"
13+
- "scripts/registry_*.py"
1214
- ".github/workflows/**"
1315
pull_request:
1416
branches: [main]
@@ -34,3 +36,20 @@ jobs:
3436

3537
- name: Run regression tests
3638
run: python test_regression.py
39+
40+
registry-lint:
41+
runs-on: ubuntu-latest
42+
timeout-minutes: 5
43+
steps:
44+
- uses: actions/checkout@v4
45+
46+
- name: Set up Python
47+
uses: actions/setup-python@v5
48+
with:
49+
python-version: "3.12"
50+
51+
- name: Install registry dependencies
52+
run: pip install pyyaml jsonschema
53+
54+
- name: Validate registry/experiments.yaml
55+
run: python -m registry.loader --check

.gitignore

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -138,3 +138,27 @@ legacy_figures_archive/
138138

139139
# New canonical rendered figures (regenerate via website/figures_render.py)
140140
figures_v2/
141+
142+
# Benchmark working dir: per-case caches, the python-flint test venv,
143+
# and ad-hoc throwaway scripts / intermediate logs from iterating on the
144+
# experiment. Canonical artifacts (writeups, results JSONs, the orchestrator,
145+
# the worker, the watchdog) are committed; everything else regenerates by
146+
# re-running the orchestrator.
147+
bench_flint/_*
148+
bench_flint/venv/
149+
bench_flint/p1_*.log
150+
bench_flint/p2_*.log
151+
bench_flint/p3_*.log
152+
bench_flint/probe_*.log
153+
bench_flint/simp2_*.log
154+
bench_flint/simplify_exp_*.log
155+
bench_flint/simplify_phases_orchestrator.log
156+
bench_flint/validation_orchestrator.log
157+
bench_flint/simplify_experiment.py
158+
bench_flint/simplify_experiment2.py
159+
bench_flint/simplify_experiment2_results.json
160+
bench_flint/simplify_experiment2_results.txt
161+
bench_flint/results.jsonl
162+
163+
# Per-(M,L) checkpoints from nbody/run_schwarzschild.py (regeneratable)
164+
nbody/checkpoints_schwarzschild/

bench_flint/bench.py

Lines changed: 237 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,237 @@
1+
#!/usr/bin/env python3
2+
"""python-flint vs default ground-types benchmark on the actual project workload.
3+
4+
Run with::
5+
6+
# Default Python ground types (in either venv)
7+
python bench_flint/bench.py
8+
9+
# FLINT ground types (in bench_flint/venv with python-flint installed)
10+
SYMPY_GROUND_TYPES=flint python bench_flint/bench.py
11+
12+
The script does three things:
13+
14+
1. Micro: time a single hard ``cancel()`` on a polynomial expression
15+
structurally similar to the Schwarzschild L3 brackets that take
16+
100s+ in the live run.
17+
18+
2. End-to-end (small): run the Schwarzschild composite at L<=2 and time
19+
the full ``compute_growth(max_level=2)`` call. Repeats N times for
20+
noise reduction.
21+
22+
3. End-to-end (medium): same at L<=3 with N=3, d=2, low samples. This is
23+
the most representative single-run benchmark; takes a few minutes
24+
under stock Python and should be much faster under FLINT.
25+
26+
Output is plain text plus a JSON line at the end suitable for
27+
diffing across runs.
28+
"""
29+
30+
from __future__ import annotations
31+
32+
import json
33+
import os
34+
import platform
35+
import sys
36+
import time
37+
from pathlib import Path
38+
39+
# Identify backend BEFORE importing sympy heavy bits.
40+
import sympy # noqa: E402
41+
from sympy.polys.domains import GROUND_TYPES # noqa: E402
42+
43+
import sympy as sp # noqa: E402
44+
45+
REPO_ROOT = Path(__file__).resolve().parent.parent
46+
sys.path.insert(0, str(REPO_ROOT / "nbody"))
47+
48+
49+
def banner(title: str) -> None:
50+
print()
51+
print("=" * 72)
52+
print(f" {title}")
53+
print("=" * 72)
54+
55+
56+
def print_env() -> dict:
57+
env = {
58+
"python": sys.version.split()[0],
59+
"platform": platform.platform(),
60+
"sympy": sympy.__version__,
61+
"ground_types": GROUND_TYPES,
62+
"env_var": os.environ.get("SYMPY_GROUND_TYPES", "<unset>"),
63+
}
64+
try:
65+
import flint # type: ignore
66+
env["python_flint"] = flint.__version__
67+
except ImportError:
68+
env["python_flint"] = "<not installed>"
69+
banner("Environment")
70+
for k, v in env.items():
71+
print(f" {k:18s} {v}")
72+
return env
73+
74+
75+
# --------------------------------------------------------------------------- #
76+
# Bench 1: hard cancel() on a structurally-realistic polynomial
77+
# --------------------------------------------------------------------------- #
78+
79+
def make_hard_polynomial() -> sp.Expr:
80+
"""Build a multivariate rational polynomial sized like a typical
81+
Schwarzschild L3 intermediate. 15 variables, lots of cross products,
82+
nontrivial denominator.
83+
"""
84+
syms = sp.symbols("x1 y1 x2 y2 x3 y3 px1 py1 px2 py2 px3 py3 u12 u13 u23")
85+
x1, y1, x2, y2, x3, y3, px1, py1, px2, py2, px3, py3, u12, u13, u23 = syms
86+
87+
# Numerator: a deliberately ugly polynomial in 15 variables.
88+
num = (
89+
(px1 ** 2 + py1 ** 2) * u12 ** 3 * u13
90+
+ (px2 ** 2 + py2 ** 2) * u12 ** 3 * u23
91+
+ (px3 ** 2 + py3 ** 2) * u13 ** 3 * u23
92+
+ (x1 - x2) ** 2 * (y2 - y3) ** 2 * u12 ** 2 * u23 ** 2
93+
+ (x1 - x3) ** 2 * (y1 - y3) ** 2 * u13 ** 2 * u23 ** 2
94+
+ (px1 * px2 + py1 * py2) * (x1 - x3) * (y2 - y3) * u12 ** 2 * u13 ** 2
95+
+ (px2 * px3 + py2 * py3) * (x2 - x1) * (y3 - y1) * u13 ** 2 * u23 ** 2
96+
- sp.Rational(7, 4) * u12 ** 4 * u13 ** 2 * u23
97+
+ sp.Rational(11, 8) * u12 ** 2 * u13 ** 4 * u23 ** 2
98+
- sp.Rational(3, 16) * u12 * u13 * u23 ** 5
99+
)
100+
101+
# Denominator: small but nontrivial - exercises GCD path.
102+
den = u12 ** 2 + u13 ** 2 + u23 ** 2
103+
104+
return num / den
105+
106+
107+
def bench_micro(n_iter: int = 5) -> dict:
108+
banner(f"Bench 1: cancel() on a hard rational polynomial (n_iter={n_iter})")
109+
expr = make_hard_polynomial()
110+
print(f" num/den built: {len(sp.Add.make_args(sp.expand(expr.as_numer_denom()[0])))} terms in numerator")
111+
112+
times = []
113+
for i in range(n_iter):
114+
# Re-create from scratch each iteration to defeat any caches.
115+
e = make_hard_polynomial()
116+
t0 = time.perf_counter()
117+
result = sp.cancel(e)
118+
elapsed = time.perf_counter() - t0
119+
nterms = len(sp.Add.make_args(result.as_numer_denom()[0]))
120+
times.append(elapsed)
121+
print(f" iter {i+1}/{n_iter}: {elapsed:8.3f}s ({nterms} terms in numerator)")
122+
123+
avg = sum(times) / len(times)
124+
best = min(times)
125+
print(f" -> mean: {avg:.3f}s, best: {best:.3f}s")
126+
return {"name": "micro_cancel", "iters": n_iter, "times_s": times,
127+
"mean_s": avg, "best_s": best}
128+
129+
130+
# --------------------------------------------------------------------------- #
131+
# Bench 2: end-to-end Schwarzschild L2 (small, fast, noise-free)
132+
# --------------------------------------------------------------------------- #
133+
134+
def bench_schwarzschild_l2(n_iter: int = 3) -> dict:
135+
from exact_growth_nbody import NBodyAlgebra
136+
137+
banner(f"Bench 2: Schwarzschild composite L<=2 dimseq (n_iter={n_iter})")
138+
times = []
139+
for i in range(n_iter):
140+
# Use a unique cache dir per iter to avoid loading prior pickle.
141+
ckpt = REPO_ROOT / "bench_flint" / f"_cache_l2_iter{i}_{GROUND_TYPES}"
142+
params = [
143+
(-sp.Integer(1), 1),
144+
(sp.Rational(1, 2), 2),
145+
(-sp.Integer(1), 3),
146+
]
147+
alg = NBodyAlgebra(
148+
n_bodies=3, d_spatial=2, potential="composite",
149+
potential_params=params, checkpoint_dir=str(ckpt),
150+
)
151+
t0 = time.perf_counter()
152+
dims = alg.compute_growth(max_level=2, n_samples=50, seed=42)
153+
elapsed = time.perf_counter() - t0
154+
seq = [int(dims[lv]) for lv in range(3)]
155+
times.append(elapsed)
156+
print(f" iter {i+1}/{n_iter}: {elapsed:8.3f}s sequence = {seq}")
157+
# Cleanup
158+
import shutil
159+
if ckpt.exists():
160+
shutil.rmtree(ckpt, ignore_errors=True)
161+
162+
avg = sum(times) / len(times)
163+
best = min(times)
164+
print(f" -> mean: {avg:.3f}s, best: {best:.3f}s")
165+
return {"name": "schwarzschild_l2", "iters": n_iter, "times_s": times,
166+
"mean_s": avg, "best_s": best}
167+
168+
169+
# --------------------------------------------------------------------------- #
170+
# Bench 3: end-to-end Schwarzschild L3 (slow, single iter)
171+
# --------------------------------------------------------------------------- #
172+
173+
def bench_schwarzschild_l3() -> dict:
174+
"""Time one full Schwarzschild L<=3 run. Single iter (this takes
175+
minutes-to-hours)."""
176+
from exact_growth_nbody import NBodyAlgebra
177+
178+
banner("Bench 3: Schwarzschild composite L<=3 dimseq (1 iter, can take minutes)")
179+
ckpt = REPO_ROOT / "bench_flint" / f"_cache_l3_{GROUND_TYPES}"
180+
params = [
181+
(-sp.Integer(1), 1),
182+
(sp.Rational(1, 2), 2),
183+
(-sp.Integer(1), 3),
184+
]
185+
alg = NBodyAlgebra(
186+
n_bodies=3, d_spatial=2, potential="composite",
187+
potential_params=params, checkpoint_dir=str(ckpt),
188+
)
189+
t0 = time.perf_counter()
190+
dims = alg.compute_growth(max_level=3, n_samples=100, seed=42)
191+
elapsed = time.perf_counter() - t0
192+
seq = [int(dims[lv]) for lv in range(4)]
193+
print(f"\n L<=3 elapsed: {elapsed:.1f}s sequence = {seq}")
194+
195+
# Don't auto-clean cache - caller may want to inspect it
196+
return {"name": "schwarzschild_l3", "iters": 1, "times_s": [elapsed],
197+
"mean_s": elapsed, "best_s": elapsed, "sequence": seq}
198+
199+
200+
# --------------------------------------------------------------------------- #
201+
# Main
202+
# --------------------------------------------------------------------------- #
203+
204+
def main() -> int:
205+
import argparse
206+
ap = argparse.ArgumentParser(description=__doc__)
207+
ap.add_argument("--skip-l3", action="store_true",
208+
help="Skip the slow L3 benchmark")
209+
ap.add_argument("--micro-iter", type=int, default=5)
210+
ap.add_argument("--l2-iter", type=int, default=3)
211+
ap.add_argument("--out", default=None,
212+
help="Append JSON results line to this file")
213+
args = ap.parse_args()
214+
215+
env = print_env()
216+
results = {"env": env, "benches": []}
217+
218+
results["benches"].append(bench_micro(args.micro_iter))
219+
results["benches"].append(bench_schwarzschild_l2(args.l2_iter))
220+
if not args.skip_l3:
221+
results["benches"].append(bench_schwarzschild_l3())
222+
223+
banner("SUMMARY")
224+
print(f" ground_types: {env['ground_types']}")
225+
for b in results["benches"]:
226+
print(f" {b['name']:30s} mean={b['mean_s']:8.2f}s best={b['best_s']:8.2f}s")
227+
228+
if args.out:
229+
with open(args.out, "a", encoding="utf-8") as f:
230+
f.write(json.dumps(results) + "\n")
231+
print(f"\n Appended to: {args.out}")
232+
233+
return 0
234+
235+
236+
if __name__ == "__main__":
237+
sys.exit(main())

bench_flint/diagnose_1r4.json

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
[
2+
{
3+
"d": 1,
4+
"n_samples": 500,
5+
"elapsed_s": 7.092368199999328,
6+
"sequence": [
7+
3,
8+
6,
9+
17,
10+
116
11+
]
12+
},
13+
{
14+
"d": 2,
15+
"n_samples": 1000,
16+
"elapsed_s": 25.419266299999435,
17+
"sequence": [
18+
3,
19+
6,
20+
17,
21+
116
22+
]
23+
},
24+
{
25+
"d": 2,
26+
"n_samples": 2000,
27+
"elapsed_s": 25.183044699995662,
28+
"sequence": [
29+
3,
30+
6,
31+
17,
32+
116
33+
]
34+
}
35+
]

bench_flint/p1_cancel.json

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
{
2+
"strategy": "cancel",
3+
"case": "schwarz_l2",
4+
"max_level": 2,
5+
"n_samples": 50,
6+
"expected": [
7+
3,
8+
6,
9+
17
10+
],
11+
"started_at": "2026-04-19T15:33:29",
12+
"elapsed_s": 4.913747500002501,
13+
"completed_at": "2026-04-19T15:33:34",
14+
"status": "done",
15+
"sequence": [
16+
3,
17+
6,
18+
17
19+
],
20+
"match": true
21+
}

bench_flint/p1_identity.json

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
{
2+
"strategy": "identity",
3+
"case": "schwarz_l2",
4+
"max_level": 2,
5+
"n_samples": 50,
6+
"expected": [
7+
3,
8+
6,
9+
17
10+
],
11+
"started_at": "2026-04-19T15:33:44",
12+
"elapsed_s": 1.6643433000062942,
13+
"completed_at": "2026-04-19T15:33:45",
14+
"status": "done",
15+
"sequence": [
16+
3,
17+
6,
18+
17
19+
],
20+
"match": true
21+
}

0 commit comments

Comments
 (0)