Skip to content

ENH: trace_s_curve: sparse corrector Jacobian for large clusters - #345

Merged
jameskermode merged 3 commits into
masterfrom
26_continuation_sparse
Oct 23, 2026
Merged

jameskermode merged 3 commits into
masterfrom
26_continuation_sparse

Conversation

@jameskermode

Copy link
Copy Markdown
Member

Stacked on #344 (base 26_continuation_tangent, so the diff shows only this PR's commits); it retargets to master when #344 merges.

What

trace_s_curve can now keep its corrector Jacobian sparse, so clamped clusters of several thousand atoms trace with modest memory. TraceSettings(surrogate="sparse") is opt-in; the default stays "dense".

  • LowRankJacobian: B = S + U Vᵀ.
    • S is the surrogate Jacobian as a scipy.sparse matrix.
    • U Vᵀ is a limited-memory good-Broyden correction, compressed by truncated SVD to max_rank (default 120).
    • refresh keeps the three rules from BUG: trace_s_curve: tangent predictor and projected surrogate refresh (large-cluster stall) #344 (projected / carry / reset). The projected form adds −(Sn − S) Q Qᵀ to the low-rank part.
    • The bordered system is solved by block elimination on one splu of S per refresh, Woodbury for the low-rank part (S⁻¹U cached) and one step of iterative refinement.
    • The same factorisation serves the tangent, the Broyden iterations and the Krylov preconditioner.
  • Correctors and solves: bordered_solve, BroydenCorrector (update, refresh, tangent) and the KrylovCorrector preconditioner accept either a dense B or a LowRankJacobian.
  • HessianSurrogate.jac_sparse / hessian_sparse: these keep the calculator's sparse hessian property sparse and add the X-H cap springs as sparse blocks. The dense jac and hessian are unchanged. calibrate now uses the sparse Jacobian in both modes, so its scale changes only at round-off.
  • Optional hook: if the problem provides hvp(x, v), an exact dF/dx·v, the Krylov corrector uses it instead of finite differences. The module stays calculator-agnostic; the hook is a seam for application code.
  • BUG fix: a turn in k at the first point after the start is no longer counted as a fold. A start relaxed to ftol can sit further off the curve than the first step is long, so k turned once and a fold was logged.
  • ChangeLog (ENH and BUG entries) and a "large clusters" section in docs/applications/fracture_mechanics.rst.

Validation

SW, unpassivated 110_110 clamped_cluster, start k 0.8, ftol 2e-3, dr_cap 0.05, alpha 2, run to just past K−. M3 Pro laptop, single process.

atoms B points K+ / K− force calls per point rejected steps wall peak RSS
864 dense 164 1.575250 / 0.601770 1.45 3 80 s 0.57 GB
864 sparse 164 identical 1.45 3 64 s 0.33 GB
1536 dense 192 1.574234 / 0.590595 1.35 2 236 s 1.36 GB
1536 sparse 192 identical 1.35 2 134 s 0.48 GB
3456 sparse 239 1.575392 / 0.580519 1.29 3 456 s 0.90 GB

Dense and sparse give the same points, force calls and rejected steps. The same design in the crackcell driver traced 9600 and 18816 atoms.

Tests

Nine new tests in tests/test_clamped_crack.py, each written to fail first:

  • sparse vs dense surrogate Jacobian (unpassivated and H-capped, to 1e-12);
  • low-rank update and bordered solve vs dense;
  • refresh vs the dense rules for projected / carry / reset;
  • compression keeps the leading terms;
  • a sparse trace reproduces the dense one (points, calls, methods);
  • a turn right after the start is not a fold;
  • settings validation;
  • two for the hvp hook.

Results: test_clamped_crack.py 38 passed, test_cubic_crystal_crack.py 6, test_rectilinear_anisotropic_crack.py 3, test_build_3D_crack.py 2. test_crack.py needs atomistica, which is not installed locally, so CI covers it.

Not included: ASE's Exp preconditioner was tried as the Jacobian and is too stiff on the soft in-plane modes (cos 0.08 with the exact tangent at 864 atoms). Faster sparse Hessian assembly in matscipy's calculators is a separate follow-up: it dominates the run time at ~10k atoms.

🤖 Generated with Claude Code

jameskermode and others added 3 commits October 23, 2026 09:46
… not a fold

The start is relaxed only to max|F| < ftol at fixed k, so on large clusters it can sit further
off the curve than the first step is long, and the first corrected point can then lie below it
in k. crackcell's driver (the same algorithm) counted that as a fold on 110_110 at 18816 atoms
(0.8 -> 0.790 -> 0.794, "fold" at 0.750). Port of crackcell 655901b: `_k_turns` ignores a sign
change of dk whose first point is the start.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TraceSettings(surrogate="sparse") (default "dense", unchanged) holds the corrector Jacobian as
B = S + U V^T (LowRankJacobian): S the surrogate Jacobian at the last refresh as a scipy.sparse
matrix, U V^T the limited-memory good-Broyden correction (one column pair per force call,
compressed by truncated SVD to max_rank = 120 at each refresh). The refresh rules are the dense
ones (projected adds -(Sn - S) Q Q^T to the low-rank part). Bordered solves use block
elimination on one splu of S's free-free block per refresh plus Woodbury for U V^T (S_u^-1 U
cached), one step of iterative refinement, and are reused by the tangent, every corrector
iteration and the Krylov preconditioner. Port of crackcell continuation/sparse.py.

HessianSurrogate.hessian_sparse / jac_sparse keep the calculator's sparse Hessian sparse (X-H
cap springs as sparse blocks); jac is unchanged, calibrate uses the sparse Jacobian. A
surrogate's jac may return dense or scipy.sparse in either mode. If the problem provides
hvp(x, v) (exact dF/dx v), KrylovCorrector uses it instead of finite differences.

SW, unpassivated 110_110, start k 0.8, ftol 2e-3, k_window (0.3, 3), max_calls_per_point 60,
start_max_iter 300, stopped just after K- (Apple M3 Pro):

  atoms  B       points  K+ / K-              calls/pt  rejected  wall   peak RSS
  864    dense   164     1.575250 / 0.601770  1.45      3         80 s   0.57 GB
  864    sparse  164     1.575250 / 0.601770  1.45      3         64 s   0.33 GB
  1536   dense   192     1.574234 / 0.590595  1.35      2         236 s  1.36 GB
  1536   sparse  192     1.574234 / 0.590595  1.35      2         134 s  0.48 GB
  3456   sparse  239     1.575392 / 0.580519  1.29      3         456 s  0.90 GB

Dense and sparse folds agree to 5e-8 with identical points and force calls.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d fracture docs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Base automatically changed from 26_continuation_tangent to master October 23, 2026 10:54
@jameskermode jameskermode reopened this Oct 23, 2026
@jameskermode
jameskermode merged commit 7a1eba6 into master Oct 23, 2026
47 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant