Branch exploration: moving the system tier out of matlab.unittest and into Simulink Test, and using that move to get requirements verification status rolling up in the Requirements Editor — headlessly, from a build script, with no interactive linking step.
Artifacts: ../tests/system/GalacticSoupSystemTests.mldatx (generated by ../tests/system/buildSystemTestFile.m), the remaining MATLAB tiers in ../tests/ and ../behavior/tests/ (now sltest.TestCase subclasses), ../tests/runAllTests.m. Decision: ADR-022 in 07_decision_log.md.
11_test_organization.md organized the system tier — tSystemNominal and tSystemFault — as two more matlab.unittest classes alongside the analysis and traceability tiers, on the same footing as everything else in the suite. That placement was reasonable as far as it went: the tier simulates the three physical architecture models and baselines throughput bands and fault-retention figures, and matlab.unittest is perfectly capable of calling sim and asserting on the result.
But §5 of that document also recorded a limitation that mattered more than it looked at the time: slreq.createLink rejects a matlab.unittest.Test element outright, and nothing in the matlabtest namespace fills the gap. A MATLAB test can baseline a number, but it cannot be linked to the requirement that number verifies — which meant SR-GS-002 (throughput floor) and SR-GS-026 (no single-fault production kill) had no path to a Requirements Editor verification status that traced back to an executed test, only to the static Requirements Table gate (ADR-010) checking the same numbers procedurally.
Simulink Test's test cases are a different kind of object — a sltest.testmanager.TestCase, not a matlab.unittest.Test — and slreq.createLink accepts them without complaint, producing a Verify link (domain linktype_rmi_testmgr) exactly where you'd expect one: a passing simulation test case is the verification of a requirement it targets. That capability, not any dissatisfaction with matlab.unittest's simulation support, is the reason the system tier moved. The requirement-linkage story was the driver; retooling how the model gets simulated was incidental.
tSystemNominal.m and tSystemFault.m are retired as redundant now that ../tests/system/GalacticSoupSystemTests.mldatx covers the same six simulations with the same baselined figures, plus the links matlab.unittest could never carry.
buildSystemTestFile.m generates the test file the same way buildComplianceGate.m generates the gate model (ADR-010): destructively and idempotently, never hand-edited. It purges stale Verify links to the artifact from the requirement set, deletes any existing .mldatx, and rebuilds from a literal table of cases in the script — the same golden-value-as-literal philosophy ADR-021 established for the MATLAB tiers.
Two suites, six cases, all simulation test cases against the three physical architecture models:
Nominal — steady-state throughput, run to 14,400 s simulated time:
| Case | Model | Regression band (bph) | Verify link |
|---|---|---|---|
| HyperCook nominal | PhysicalHyperCook |
308.4 ± 3 | SR-GS-002 |
| LeanBroth nominal — regression baseline | PhysicalLeanBroth |
196.8 ± 3 | (none) |
| EverSimmer nominal | PhysicalEverSimmer |
231.9 ± 3 | SR-GS-002 |
WorstFault — a Fault_T_* model-workspace variable overridden to 7,200 s via a parameter set, run out to 21,600 s:
| Case | Model | Fault variable | Retention | Verify link |
|---|---|---|---|---|
| HyperCook worst fault — regression baseline | PhysicalHyperCook |
Fault_T_QC |
0 | (none) |
| LeanBroth worst fault — regression baseline | PhysicalLeanBroth |
Fault_T_Prep |
0 | (none) |
| EverSimmer worst fault | PhysicalEverSimmer |
Fault_T_Cell1 |
0.672 | SR-GS-026 |
Every case leans on a custom-criteria callback rather than the Test Manager's built-in signal checks, because the assertions are computed quantities (a trapezoidal steady-rate integral, a pre/post-fault retention ratio), not raw signal comparisons. The callback pulls flow_bps and totalPower_kW/plantMode out of test.sltest_simout's logged yout, then asserts:
- Nominal cases with a floor check (HyperCook, EverSimmer):
verifyGreaterThanOrEqual(bph, 200, 'SR-GS-002 floor'), in addition to the regression band. This is the compliant-variant check — the floor assertion only appears on cases meant to carry theVerifylink, because a passing floor check is the thing being verified. - Every nominal case: the steady-rate regression band (
verifyEqualwithAbsTol3 bph) andplantModeending atNominal(mode code 1). - Every fault case: the worst-fault retention ratio (
verifyEqualwithAbsTol0.02), computed as post-fault mean rate over pre-fault mean rate. - EverSimmer's fault case only:
plantModeending atDegraded(mode code 2) — the one case where a whole-cell fault actually collapses health telemetry along with output, per ADR-020's noted limitation that the other two variants' single-string faults don't moveplantModeoffNominal.
Two generator gotchas worth carrying forward as pattern, not just incident: every new suite (and every createTestSuite call) ships a default placeholder test case that has to be found and removed, and the file is saved once before linking so the cases carry persistent IDs, then linked, then saved again — see §5.
Three Verify links exist in the built file: HyperCook nominal → SR-GS-002, EverSimmer nominal → SR-GS-002, EverSimmer worst fault → SR-GS-026. LeanBroth's two cases — nominal and worst-fault — are deliberately unlinked.
The rule adopted: a Verify link attaches only where a passing test result means the requirement is genuinely met. LeanBroth's nominal case baselines 196.8 bph, which is below the SR-GS-002 200 bph floor — the formal Requirements Table gate (ADR-010) already flags LeanBroth's Throughput row, and that failure is the correct, load-bearing finding, carried since 10_behavioral_trade_update.md §3. Linking that case to SR-GS-002 as Verify would have the Requirements Editor report the requirement as satisfied by a test whose own logged number contradicts it — a fabricated verification, worse than no link at all. LeanBroth's cases stay in the suite as regression baselines (the numbers are still worth protecting against silent drift) but carry no requirement traceability; that story belongs to the compliance gate, which is the artifact designed to say "this variant fails this requirement" without also being asked to claim the opposite.
The same reasoning is why HyperCook and LeanBroth's fault cases are unlinked to SR-GS-026 (no single-fault production kill): both collapse to zero output on their single-string failure mode, which is the expected, documented behavior for a non-EverSimmer variant (ADR-009, ADR-020) — not a requirement violation to verify against, and not a requirement satisfaction either. Only EverSimmer's case demonstrates the requirement being met, so only it carries the link.
The end-to-end recipe, confirmed both interactively (Requirements Editor shows both SRs green/Verified) and reproduced through pure API calls with no UI in the loop:
1. build the .mldatx (buildSystemTestFile — saves, then links, then saves again)
2. run it (sltest.testmanager.run)
3. reload the requirement set fresh
4. updateVerificationStatus(srSet)
5. read status (getVerificationStatus on each linked SR — both report verified-passed)
Two traps surfaced getting here, both state-ordering rather than API-capability problems:
The "unexecuted" trap. Reading verification status from a requirement set whose links had just been purged and recreated in the same MATLAB session reported the SRs as unexecuted, even though the test file had actually run and passed. The fix was not a different API call — it was slreq.clear() followed by a fresh slreq.load() of the requirement set before calling updateVerificationStatus, so the status computation sees persisted link state rather than an in-memory set still holding references from the link-churn that just happened. runAllTests.m follows exactly this ordering (lines calling slreq.clear() then slreq.load() immediately before updateVerificationStatus) rather than reusing whatever requirement-set handle was already in scope.
The total-count quirk. getVerificationStatus on an SR with two Verify links from the same test file (EverSimmer's cases don't double up, but the pattern was observed while iterating on link counts during development) reports a total field that aggregates to 1 rather than the number of links — an aggregation quirk in how the status struct counts sources, not a bug in the pass/fail determination. passed/failed/unexecuted are correct and are what runAllTests.m asserts on; total is not used for anything and should not be read as "number of links."
Collected here because most of them cost real debugging time and generalize past this one file:
- Save before linking.
slreq.createLinkagainst an unsaved test case captures whatever provisional ID the case has in memory; that ID never matches the ID recorded in an executed result. Save the file first, link second, save again after linking. - Placeholder cases. Every new test suite, and every
createTestSuitecall, ships a "New Test Case 1" default case. It has to be enumerated and removed explicitly, or it sits in the suite as a case that runs and (trivially) passes, inflating counts. This is the same gotcha as the Requirements Table placeholder row (08_formal_compliance_gate.md§5) — MathWorks generator APIs seem to consistently ship a stub row/case that generation scripts must know to delete. - Reserved characters in case names. Parentheses and
=in a test case name get mangled by the unittest adapter thatTestSuite.fromProjectuses to enumerate.mldatxcases alongside MATLAB tests. Case names in this file avoid parentheses entirely (e.g., "EverSimmer worst fault," not "EverSimmer worst fault (Fault_T_Cell1)"). OverrideStopTimeordering.setProperty(tc, 'OverrideStopTime', true)must be set before the numericStopTimeproperty; settingStopTimefirst has no effect because the override flag isn't yet enabled to honor it.- Results hierarchy. Extracting pass/fail requires walking
ResultSet→getTestFileResults→getTestSuiteResults→getTestCaseResults— there is no flat "give me every case result" accessor. - Wildcard-importing
sltest.testmanager.*. Importing the whole namespace into the base workspace shadowsclear,load, andrunfor the rest of the session — build and runner scripts call fully-qualifiedsltest.testmanager.clear/.load/.runinstead. fromProjectadapts.mldatxcases into the same suite as MATLAB tests. An earlier version of the runner filtered them out and ran the file separately throughsltest.testmanager.run, on the belief that only the Test Manager execution path registers results for requirement verification. That belief was wrong: running the adapted cases through the ordinarymatlab.unittestrunner registers verification results too (confirmed empirically — status readspassedafter an adapter-only run). §6 describes the unified runner this enabled.
runAllTests.m is still the single entry point (ADR-021), extended rather than replaced:
results = runAllTests() % all 37: MATLAB tiers + simulation cases, one suite
results = runAllTests("analysis") % one MATLAB tier by tag
results = runAllTests("system") % just the six simulation cases
The runner assembles ONE suite: TestSuite.fromProject(proj) collects the MATLAB test classes (via their Test classification labels) and adapts the .mldatx cases into the same suite, and a single matlab.unittest runner executes all 37 with the coverage plugin attached. Simulation cases and MATLAB tests appear in the same results table, count in the same pass/fail total, and — the discovery that unlocked this design — register their results for requirement verification identically through the adapter path. No second engine, no filtering, no double run.
When called with no argument, runAllTests runs the full suite and finishes with the headless verification-rollup recipe from §4, asserting that both SR-GS-002 and SR-GS-026 come back verified-passed. A tier-filtered call selects by TestTags; runAllTests("system") selects the adapted simulation cases by name, since .mldatx cases carry no tags. Tier-filtered runs skip the requirement-set refresh — the fast inner loop for iterating on one tier is unchanged from ADR-021.
Both tools stayed. Converting the remaining MATLAB tiers — 21 component tests in behavior/tests/, plus the analysis (tRollupInvariants, tGateAgreement, tTradeDeterminism) and traceability (tTraceability) tiers — to sltest.TestCase subclasses was a one-line superclass change per file; they still run via runtests/fromProject exactly as before, and gained Test-Manager compatibility for free. Nothing about them needed to become a .mldatx case.
MATLAB Test (sltest.TestCase / matlab.unittest) |
Simulink Test (.mldatx) |
|
|---|---|---|
| Pure-MATLAB analysis logic (roll-up totals, gate agreement, MCDA determinism) | Wins. No model in the loop; asserting on a MATLAB struct is direct and fast. | Would need a wrapper model with no purpose beyond hosting the assertion. |
| Requirement-link integrity (traceability tier) | Wins. Same reasoning — the thing under test is link-graph structure, not simulated behavior. | Not applicable. |
| Golden-value baselining ergonomics | Wins. A literal expected value and an AbsTol in a verifyEqual call, next to the code that produced the number. |
Same idiom is available in a custom-criteria callback, but it's a string of MATLAB inside a callback property rather than a method body — more ceremony to read and edit. |
| Coverage | Code coverage was tried via CodeCoveragePlugin and later removed as circular for this project (ADR-023). |
The coverage that matters here is requirements coverage — driven by the Verify links only Simulink Test cases can carry, summarized by runAllTests and reported by analysis/reporting/makeRequirementsReport. |
| Model simulation cases (steady-state throughput, fault injection) | Works — tSystemNominal/tSystemFault proved this in ADR-021 — but every case is bespoke sim/SimulationInput scripting. |
Wins. simulation test cases, parameter sets, and OverrideStopTime are purpose-built for exactly this; the six cases here are declarative table rows, not scripted sim calls. |
| Parameter-override fault injection | Possible via SimulationInput.setVariable, hand-rolled per test. |
Wins. addParameterOverride on a named parameter set is the intended mechanism and reads as data, not code. |
| Programmatic requirements verification | Cannot do this. slreq.createLink rejects matlab.unittest.Test elements; this was the R2026a limitation recorded in 11_test_organization.md §5 and never resolved for that object type. |
Wins, decisively. This is the capability that drove the whole branch: Verify links from test case to requirement, and updateVerificationStatus rolling up real, executed results into the Requirements Editor — headlessly. |
The honest read: this isn't a case where one tool should have replaced the other. MATLAB Test is still the better fit for the four tiers that never touch a running model, and Simulink Test is the only fit for the one tier that does and for the one capability — programmatic requirements verification — that the whole branch was chasing. They compose: MATLAB tiers for logic, one Simulink Test file for simulation and verification, one runner (runAllTests) that treats both as a single pass/fail stack. Full stack green today: 31 MATLAB tests plus 6 simulation cases, a coverage report, and SR-GS-002 / SR-GS-026 both verified-passed.
Every simulation case now runs its model inside an externally-saved Simulink Test harness (../tests/system/buildTestHarnesses.m; one per architecture model, stored beside the models). The harness boundary made the root interface honest: the architecture models have root inports (AmbientGravity, CustomerOrders, InboundSupplies) that bare simulation silently grounded and the harness turns into explicit Constant-0 sources, and the virtual Telemetry bus flattens at the component-under-test boundary into named scalar outports that the criteria harvest by name. The rework surfaced one hard rule: Test Manager parameter overrides reach dictionary entries through a harness but NOT model-workspace variables — which is why the fault and resupply-cutoff variables now live in the behavior dictionaries, with the direct-simulation setVariable path preserved via model-workspace shadowing.