External review: independent verification of the math, governance gates, and evidence chain

[alexroessner]

External review: independent verification of the math, governance gates, and evidence chain

Hey Sean — Alex here. You asked for a review; we went deep. Three independent passes over aeeaf72, then an adversarial audit of our own findings before posting any of this: every financial claim recomputed in Python (not eyeballed), a code-quality/architecture pass, a governance audit, and a fact-check pass that killed or corrected several of our own first-draft claims. Full test suite run live on macOS + Python 3.11.

The short version: the foundation is worth finishing. The closed-form math is uniformly correct (Black–Scholes matches Hull/McDonald to published values, all ten Greeks match finite differences to ≤1e-6, bond math matches textbook values, the VC election-equilibrium waterfall is more rigorous than most commercial cap-table tools). We spot-checked public cases against real-world figures — Microsoft–LinkedIn 2016, Tesla Feb-2020, BlackRock 2023 AUM, SPX/VIX 2024-01-02 — and every input number was right. The workbook-vs-LibreOffice-vs-oracle verification in verify_reference_calcs.py is real, binding validation. And the repo's restraint (0 M3 / 0 M4 self-declared, the Macy's 12,300-forecast-vs-24,460-realized miss recorded without flinching) is the strongest governance signal in it.

We're opening companion PRs for everything that's an objective, mechanically verifiable fix (listed at the bottom). This issue covers the findings that are your call, not ours — design decisions or numbers only you can source.

1. The maturity gate never runs the verification that exists — connect them

To be precise about what exists (our first draft overstated this, so here's the corrected map):

• Workbook-vs-recalc-vs-oracle (tools/verify_reference_calcs.py: LibreOffice recalculation compared to independent Python at 1e-6/1e-8) covers 6 checks across 5 of 24 domains (01, 03, 13, 14, 21), against the _template_* files.
• Oracle-vs-test-vector coverage is broader than we first credited: tools/legacy_engine_oracles.py and tools/domain_hardening_oracles.py map model ids to callables that run identity checks — including tokens like per_share_identity — in legacy-engine-hardening.yml, m1-domain-hardening.yml, and frontier-program.yml. That's ~15 of 24 models. But these oracles score JSON case inputs from the registries, not the workbook (their own docstring: "intentionally separate from workbook formulas"), and the token→callable maps are hardcoded dicts — adding a token to model_inventory.json binds nothing.
• tools/validate_model_inventory.py — the tool that actually gates declared_maturity — consults none of the above. Its M2 bar (lines 160–166) is: ≥1 string in reference_checks, ≥3 strings in required_engines, ≥20 formula cells, Cover/RefreshLog present. A workbook of =A1*2 cells with a wrong DCF passes it — with the fair qualification that build_all_models.py --require-parity means the wrong math would have to live in the builder, not be hand-dropped.

Net: for 19 of 24 domains, no CI step compares any workbook's computed value to an independent calculation — 10 of those have oracles that never touch the workbook, 9 have no oracle at all — and for all 24, the maturity claim is never linked to whatever verification does run. Suggestion: make the inventory's reference_checks tokens resolve through a real registry to the oracles you already have, run them in the maturity gate, and fail promotion on unregistered or failing checks. The M3 gate has the same shape (len(perspectives) >= 3 plus two sheet names), so it's worth fixing the validator before anything promotes.

Related inert check, flagged for honesty rather than fixed: the validator's "literal Excel errors" scan loads with data_only=False, so a formula cell always yields its formula string and the scan can only catch error tokens typed as text constants. Real cached-error coverage lives in recalc.py (which other workflows run against the instances). We prototyped a data_only=True second pass and dropped it after our own audit: the inventory only inspects builder-fresh templates, which carry no calculation cache — it would have been a check that structurally cannot fire. Either delete the inert branch or scope a cached-value scan to the instance workbooks.

2. All 48 forecasts are hindsight-dated — register forecasts now to start the M4 clock

Every case carries refresh.date: 2026-08-04 (we checked all 48; no exceptions); all forecasts and realizeds were authored in one pass with outcomes known. Several outcome rows are forecast == realized identities (Vogtle 10/10, Alleghany 11,600/11,600, transaction_completed 1.0/1.0), and the Tesla row compares gross vs. net proceeds where both figures come from the same prospectus (2032.55 and 2008.65, with the 23.9 discount between them, all in the snapshot's captured_values) — zero out-of-sample content. To your credit, M3_M4_OPERATING_STATUS.md says this explicitly.

The one fix no automation can do later: commit dated forecast files for outcomes that haven't resolved yet, before their windows close. That — and only that — starts the M4 evidence clock. An outcome_class field (out_of_sample_forecast vs retrospective_reconstruction vs same_document_identity) would keep the degenerate rows from counting as monitoring.

3. One unsourced number in the "0 synthetic" evidence chain

am-public-blackrock-2023's outcome forecast: 10500.0 (ending AUM, $bn) is the only number in the case that appears in no captured_values and no source-register row — it lives in the snapshot's outcome block and is hardcoded at tools/frontier_evidence_registry.py:302, with no filing that substantiates it. It isn't a rounding of anything in the evidence either: the snapshot's own captured values reconcile exactly to ending_aum_usd_bn: 10008.995. Meanwhile realized is still null although BlackRock's 2024 AUM has been public since January 2025 — the monitoring loop isn't running yet. Relatedly, tests/test_real_data_only.py enforces "0 synthetic" via a filename-substring check, a literal "synthetic": true key walk, and a self-declared boolean — nothing checks whether a number traces to a source, which is what let this through.

4. Provenance is a URL next to a hand-typed number

• Sources are landing pages: https://fred.stlouisfed.org/series/SP500 cannot substantiate 4742.83 (the options case's stated "frozen curated observation") — nothing extracts, stores, or re-fetches values, and no content digest exists.
• as_of dates are stamped with the case date, not the source-document date: the LinkedIn September-2016 10-Q carries "as_of": "2016-12-08" (the merger close) in every input block, the sources array, and the snapshot header.
• The implied share count 133.673469 = 26200/196 is honestly tagged derived in the case JSON, but the snapshot promotes it into captured_values under capture_method: "curated_public_observation" — a quotient of two captured values recorded as if captured itself.
• Tesla's underwritten primary is force-fit into the Rights Offering archetype: cum-rights price = subscription price = 767.0 (a zero subscription discount, which degenerates the TERP and rights-value outputs), participation forced to 1.0, and the 1.176% underwriting discount recorded in the "standby / backstop fee" slot. Sharper still: the case overrides only C7–C10, so the share-count inputs C5/C6 remain the template's Cap Table defaults — the sheet's own "Expected gross proceeds" cannot produce the recorded 2032.55 forecast; the outcome number doesn't come from the workbook at all.

tools/data_fabric/edgar_company_facts.py exists but isn't wired in — connecting it (store extracted values or a digest of the fetched document, stamp as_of from the source) would make "source-addressed" literal. This is your L2 anyway.

5. Three Black–Scholes implementations, two argument orders — pick a canonical engine

finance_segway/ is well-typed and well-tested, and no builder, validator, or tool imports it — only tests do (the only "finance_segway" strings under tools/ are a Postgres DSN name). It duplicates tools/reference_engines.py, and the duplication is already biting: tools.reference_engines.black_scholes(spot, strike, years, rate, **dividend_yield, volatility**) vs finance_segway.derivatives.black_scholes(spot, strike, maturity, rate, **volatility, dividend_yield=0.0**) — positions 5 and 6 swapped, and the default on dividend_yield means a 5-positional call ports silently with vol and dividend exchanged. There's a third implementation in tools/verify_reference_calcs.py:60 (which follows the tools ordering). All are internally correct; the hazard is the seam. Finish the cutover in whichever direction you intended and the swap disappears with the duplicate.

Smaller duplication, corrected from our first draft after fact-checking: only 2 builders (build_template.py, build_lbo_template.py) still inline the style-constant block; 16 import via from template_helpers import *; and 11 release builders each repeat an ~18-line try/except dual-import preamble. tools/template_helpers.py and tools/builders/template_helpers.py are byte-identical copies (verified with cmp) — a fix applied to one silently misses the other depending on sys.path.

6. Scope questions (no action needed, just flagging)

• finance_segway/consulting/ is a coherent second product with zero connection to workbooks or governance — integrate deliberately or split it out before it grows.
• 17 workflows / six registry systems / stratified vocabulary ("hardening", "frontier", "legacy engine", "final six", "flagship", "reconciled") — sediment from the 48-hour campaign. Your own roadmap's "don't dilute L1" logic applies to the process layer too.
• One convention worth unifying eventually: the VC engine pays higher seniority numbers first while the debt and structured engines pay lower priority numbers first — opposite directions in sibling modules (documented, for now, in one of the PRs below).

Companion PRs

Each PR states its verification evidence per CONTRIBUTING, changes one thing, and was itself adversarially reviewed before posting (one earlier fix didn't survive that review and was dropped rather than submitted).

PR	What	Evidence
receipt-check-binding	the dedicated integrity workflow runs --check without --refresh	refresh-then-check validated a hash against the value it just wrote — the mismatch branch was unreachable, and the refresh also blinded the final_public_evidence step behind it. (m2-committed-state.yml:66-67 already runs a binding bare --check on broader triggers, so this removes a false green rather than adding new coverage.)
debt-sweep-shortfall-tracking	debt_sweep pays from actual cash and reports funding_shortfall instead of conjuring payments	verified: 18.0 of debt service "paid" from 2.0 of cash with the balance still amortizing to 90; now 2.0 of interest paid, debt unchanged, 16.0 shortfall reported. Tracking (not raising) chosen deliberately — your distressed cases model exactly this state
debt-schedule-funding	unfunded interest+amortization raises; the maturity-retires-without-cash convention becomes an explicit documented refinance_at_maturity=True default with a strict mode	verified: 8+10 due against 1.0 of cash silently amortized to 90; bullet maturities retired with zero cash. Funded paths unchanged
waterfall-recovery-routing	recoveries pay principal, not coupons	recoveries here are recovery of principal by construction; under the old routing every recovery dollar landed in a bucket whose excess is discarded — measured ~34 of recoveries doing literally nothing on the existing test pool
macos-path-resolution	fixes the 2 tests that fail on every Mac (/var → /private/var)	suite goes 2-failed → 141-passed on macOS; Linux provably unchanged (the platform check short-circuits)
document-engine-conventions	seniority direction + honest multi-root IRR semantics	docs only; verified: seniority=2 paid before seniority=1; [-100, 230, -132] has IRRs at exactly 10% and 20% but the message claimed no root existed

Happy to go deeper on any of these, and to help wire the reference_checks registry from #1 if you want a hand.