Skip to content

Isolate ModelingToolkit aggregate test environments#4749

Draft
ChrisRackauckas-Claude wants to merge 1 commit into
SciML:masterfrom
ChrisRackauckas-Claude:agent/opt-in-test-group-isolation
Draft

Isolate ModelingToolkit aggregate test environments#4749
ChrisRackauckas-Claude wants to merge 1 commit into
SciML:masterfrom
ChrisRackauckas-Claude:agent/opt-in-test-group-isolation

Conversation

@ChrisRackauckas-Claude

Copy link
Copy Markdown
Member

This PR should be ignored until reviewed by @ChrisRackauckas.

Summary

  • Opts ModelingToolkit aggregate test runs into fresh-process isolation for environment-backed groups.
  • Requires SciMLTesting 2.3 in the root test target and each standalone group environment.
  • Declares the packages and stdlibs imported directly by the Downstream, FMI, ModelingToolkitBase Extensions, and ModelingToolkitBase Optimization test bodies in their own Project.toml files.

Dependency

This is stacked on SciML/SciMLTesting.jl#23 and must remain draft until that change is merged and SciMLTesting 2.3 is available.

Motivation

The aggregate runner previously loaded LinearSolve 3.87 from one group, activated a later group environment resolving LinearSolve 4.3, and then combined the already-loaded module with the newer extension source. The extension referenced LinearSolve.ArnoldiMethodJL, which does not exist in 3.87. A fresh Julia process per environment-backed aggregate member prevents that mixed-module state. Direct GROUP selections intentionally retain their existing in-process behavior for fast iteration and CI lanes.

Local validation

Using Julia 1.10.11, the SciMLTesting #23 checkout, an isolated local depot, and a local wrapper that re-entered this repository test harness through the new fresh-process path:

  • Downstream: 37 passed, exit 0, 8m08.9s. The validation also asserted a new child PID, LinearSolve 4.3, the presence of ArnoldiMethodJL, and local ModelingToolkit/SciMLTesting sources.
  • FMI: 60 passed, exit 0, 32m12.5s.
  • ModelingToolkitBase Extensions: 96 passed, exit 0, 27m29.5s.
  • ModelingToolkitBase Optimization: 131 passed and 1 pre-existing broken assertion, exit 0, 20m49.6s.
  • Whole-repository Runic inplace and check/diff passes exited 0. A post-rebase whole-repository Runic check/diff also exited 0.

No QA pass is claimed. An isolated aggregate QA run reported 10 passes and 3 failures: the clean-base Aqua piracy set, existing JET top-level analysis errors, and the SciMLTesting 2.3 public-docstring audit. Direct GROUP=QA behavior is unchanged. The 31-method clean-base piracy reproduction and historical boundary are tracked in #4670. The exact 28-name SciMLTesting 2.3 documentation exposure is recorded on #4715. No skip, broken marker, allow-list, relaxed assertion, or JET-mode weakening is included here.

A full aggregate pass is also not claimed: a clean-base cross-group state failure is under a separate audit rather than being hidden by this environment fix.

Process notes

I first reproduced the mixed LinearSolve versions in one Julia process, then confirmed that a fresh process resolves and loads one coherent version. The SciMLTesting prerequisite adds synthetic same-UUID v1/v2 regression coverage on Julia 1.10 and 1.12; its complete local suites passed 299/299 and 314/314 respectively. After validating each real environment-backed ModelingToolkit group, I audited direct imports, added only the dependencies actually used by each standalone test body, ran repository-wide formatting, synchronized the fork master without overwriting divergent commits, rebased this branch onto current upstream master, and repeated the formatting/diff checks.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
@ChrisRackauckas-Claude

Copy link
Copy Markdown
Member Author

Scratchpad / commit d6200ad:

  • Rebased the single focused consumer commit onto upstream master db56a6a before publication.
  • Set isolate_group_environments=true only at the aggregate run_tests call. Direct named groups remain in process.
  • Updated SciMLTesting compat to 2.3 in the root and five group environments.
  • Added only imports that the four functional group bodies use directly; no internal dependency APIs were introduced.
  • Verified the four environment-backed functional groups locally, then ran whole-repository Runic and git diff checks after the rebase.
  • Kept QA findings visible and linked QA: track pre-existing Aqua/JET/ExplicitImports findings (root + ModelingToolkitBase) #4670 / Document public API coverage gaps #4715 instead of adding suppression.

Prerequisite implementation and its synthetic regression tests: SciML/SciMLTesting.jl#23.

@ChrisRackauckas-Claude

Copy link
Copy Markdown
Member Author

CI gate classification (initial run):

  • Inspected all 26 completed failure logs: 25 root test-matrix jobs plus the downgrade job.
  • 26/26 fail at dependency resolution because the registry offers SciMLTesting through 2.2.0 while this stacked PR correctly requires 2.3.
  • Unexpected failure classes: 0.
  • Runtime tests started: 0. The logs contain no test/runtests.jl execution and no Test Summary; each matrix job stops while Pkg constructs the test sandbox.

This is the expected prerelease gate until SciML/SciMLTesting.jl#23 is merged and SciMLTesting 2.3 is registered. No source change, retry, or test bypass is appropriate for this run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants