Skip to content

Add generic AutoDespecialize solver path - #4216

Merged
ChrisRackauckas merged 2 commits into
SciML:masterfrom
ChrisRackauckas-Claude:agent/despecialized-parameter-precompile
Aug 11, 2026
Merged

Add generic AutoDespecialize solver path#4216
ChrisRackauckas merged 2 commits into
SciML:masterfrom
ChrisRackauckas-Claude:agent/despecialized-parameter-precompile

Conversation

@ChrisRackauckas-Claude

@ChrisRackauckas-Claude ChrisRackauckas-Claude commented Aug 11, 2026

Copy link
Copy Markdown
Member

What changed

This adds DiffEqBase's generic AutoDespecialize solver path on top of SciMLBase 3.46. p is stored in SciMLBase.DespecializedParameters, solver-facing function calls use a stable barrier type, and the original concrete parameter is recovered only when the user function is invoked. The underlying SciMLBase implementation provides that call barrier for every built-in SciMLFunction family. DiffEqBase promotes the differential-equation families that reach its solve layer: ODE, DAE, DDE, SDE, SDDE, and split problems; ODE auxiliary functions and stochastic diffusion functions cross the same barrier.

AutoSpecialize is unchanged. AutoRespecialize remains the constrained, non-dynamic policy (with AutoDePSpecialize as its deprecated alias). OrdinaryDiffEq exposes all three policies as documented local aliases of the SciMLBase types. Common explicit and implicit solver workloads now precompile the AutoDespecialize path.

This is stacked on the independent docs prerequisite #4210 and requires the merged SciMLBase implementation and registration from SciML/SciMLBase.jl#1514 and JuliaRegistries/General#164118.

Please ignore this draft until it has been reviewed by @ChrisRackauckas.

Verification

Discriminating test

I ran the exact added lib/DiffEqBase/test/despecialized_p_test.jl against unmodified DiffEqBase 7.13.1 with registered SciMLBase 3.46.0:

AutoDespecialize uses the dynamic parameter barrier | 12 passed, 3 failed, 15 total

The failures showed that the two parameter layouts retained different concrete parameter/problem types. On this branch, the exact same test passes:

AutoDespecialize uses the dynamic parameter barrier | 15 passed, 15 total
Other specialization levels keep their parameter representation | 3 passed, 3 total
AutoDespecialize retains symbolic systems | 3 passed, 3 total
Other differential-equation functions use the parameter barrier | 18 passed, 18 total

The test checks same concrete problem/function types across parameter layouts, original-parameter recovery, RHS/Jacobian/time-gradient dispatch, and DAE/DDE/SDE/SDDE drift and diffusion dispatch.

Local test/build results

GROUP=Core julia +release --project=lib/DiffEqBase -e 'using Pkg; Pkg.test(coverage=false)'
Despecialized-p Hook | 39 passed, 39 total
Testing DiffEqBase tests passed

GROUP=QA julia +release --project=lib/DiffEqBase -e 'using Pkg; Pkg.test(coverage=false)'
Aqua | 9 passed, 9 total
Testing DiffEqBase tests passed

julia +release --project=docs docs/make.jl
Populate: populating indices
RenderDocument: rendering document
HTMLWriter: rendering HTML pages
exit 0
GROUP=InterfaceII julia +release --project=. -e 'using Pkg; Pkg.test(coverage=false)'
Despecialized Parameters | 7 passed, 7 total
AutoSparse Detection Tests | 4 passed, 4 total
Get du Tests | 34 passed, 34 total
Testing OrdinaryDiffEq tests passed

julia +release --project=. -e 'include("test/InterfaceII/despecialized_parameters.jl")'
exit 0 after adding the three SciMLBase policy-identity assertions

GROUP=QA julia +release --project=. -e 'using Pkg; Pkg.test(coverage=false)'
Quality Assurance Tests | 90 passed, 90 total
Testing OrdinaryDiffEq tests passed

Runic, typos over the additions/new tests, and git diff --check pass locally.

Compile-time/runtime benchmark

Fresh Julia processes, one thread pinned to one CPU, Tsit5, 1,000 fixed steps. The second problem changes the concrete parameter layout after compiling/solving the first.

policy                 same type  second-layout compile  warm median
AutoDespecialize       true       0.142 s                0.555 ms
AutoSpecialize         false      2.483 s                0.296 ms
FullSpecialize         false      2.273 s                0.184 ms

For this generic-function workload, AutoDespecialize reduced second-layout compilation by 94.3% versus AutoSpecialize, at a 0.259 ms absolute cost per 1,000-step warm solve (1.88x).

For generated MTK systems the function recompilation dominates, so the result is more modest. At 64 states, AutoDespecialize compiled the second layout in 0.572 s versus 0.550 s for AutoSpecialize and 1.457 s for FullSpecialize; warm medians were 1.203, 0.827, and 0.890 ms. At 512 states, the compile times were 0.575, 0.580, and 1.400 s; warm medians were 3.880, 5.273, and 3.331 ms. This shows the intended solver-code reuse versus FullSpecialize, while the comparison with existing AutoSpecialize is workload-dependent and the warm dynamic-dispatch cost is contained but measurable.

Not verified locally

GPU paths and external downstream packages were not run. This adds no dependency and does not change a dependency license boundary.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
@ChrisRackauckas-Claude

Copy link
Copy Markdown
Member Author

The first eight completed CI failures are package-server registry lag, before any feature test runs. Representative jobs:

Each fresh runner's package-server registry reports SciMLBase versions only through 3.45.0 and then exits during resolution because this PR correctly requires 3.46.0:

SciMLBase log:
├─possible versions are: 1.0.0-3.45.0 or uninstalled
└─restricted to versions 3.46.0-3 by OrdinaryDiffEq — no versions left

SciMLBase 3.46.0 is merged and registered through SciML/SciMLBase.jl#1514 and JuliaRegistries/General#164118. The exact registered source was used for the local failing-before/passing-after, QA, InterfaceII, and benchmark validations. I am leaving the 3.46 compatibility floor intact and will rerun the failed jobs after the package-server snapshot contains the registered release.

@ChrisRackauckas
ChrisRackauckas marked this pull request as ready for review August 11, 2026 14:29
@ChrisRackauckas
ChrisRackauckas merged commit 6f9ca9a into SciML:master Aug 11, 2026
20 of 279 checks passed
@ChrisRackauckas-Claude

Copy link
Copy Markdown
Member Author

Package-server propagation update (2026-08-11):

A fresh local probe using the package server eager registry preference now resolves and installs SciMLBase v3.46.0 at registered tree d7a1a327339865b422b89690d77bb135db810778.

The same probe with the default registry preference, after Pkg.Registry.update(), still fails:

SciMLBase log:
├─possible versions are: 1.0.0 - 3.44.0 or uninstalled
└─restricted to versions 3.46.0 by an explicit requirement — no versions left

The default registry tarball contains versions only through 3.44, whereas the eager tarball contains 3.46. I am therefore not retrying the failed standard CI jobs yet; they would still resolve against the stale default snapshot. Registration: JuliaRegistries/General#164118

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants