You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Today the deterministic S-1 parsers preempt the AI extractors from inside each modelExtractChain closure. That placement is invisible at the call site, not configurable, and is the mechanism behind the data loss #311 fixes.
Proposal: expose the deterministic path as an ordinary model id that an operator can place anywhere in a section's model list — first to try it before paying for a model, last as a fallback, or omit it entirely.
Most of this already exists
sectionRunner's RunSectionArgs already carries an ordered fallback chain, in production:
readonly extract: (text: string)=>Promise<TRow[]>;/** Tried in order when extract (and any earlier fallback) returns [] or throws … */
readonly emptyExtracts?: readonly((text: string)=>Promise<TRow[]>)[];
readonly fallbackOnEmpty?: boolean;/** Ids tried for this section; named in the MODEL_EMPTY detail when length > 1. */
readonly modelIds?: readonlystring[];
plus SectionPersistMeta.modelIndex (0 = primary, 1+ = fallback index), and the docstring already guarantees "Fallbacks do not consume VERIFICATION_ATTEMPTS".
DETERMINISTIC_MODEL_ID = "deterministic" also already exists (s1/parseOfferingTables.ts:12), and source: "deterministic" is already a field on the offering-terms and sponsor-promote row schemas.
So this is mostly wiring an existing constant into an existing chain, not new machinery.
The gap that makes it more than wiring
The chain advances on [] or a throw. It does not advance on "returned rows but cannot fill the columns the caller is about to clear."
That is exactly the shape of the data loss. parseRelatedPartyTables returns non-empty rows (the party observations) while hardcoding transactions: [], and processFormS1 has already cleared related_party_transaction for the accession. To a naive chain that is a success: it stops, the AI never runs, and the transactions are gone with no dead letter.
Consequences if position alone is the control:
deterministic first reproduces the current bug exactly;
deterministic last is safe but nearly useless — it runs only when the model already returned nothing, which is rare and forfeits the cost saving that motivates it.
Suggested design
Combine the ordering with the coverage contract #311 introduces, rather than choosing between them:
Keep DeterministicPass.covers and RunSectionArgs.clears as the safety contract.
Use preempts(pass, clears) to make the deterministic entry decline — yield [] — when it cannot cover everything the section clears, instead of returning a partial result.
Register it as a chain entry with a real id, so modelIds reports it and modelIndex attributes it.
Net effect: a declining deterministic pass looks like an empty one, which the chain already knows how to handle, and an operator gets --models-style control in production — which today only the eval harness (sec eval extract --models) has.
Design details worth settling first
Span verification. The chain currently keeps span-verification re-asks on the model that produced the rows. For a pure function that is three identical re-asks followed by an UNVERIFIED_SOURCE_SPAN dead letter blaming a model that was never called. Verification failure on the deterministic entry should count as a decline (fall through) rather than a re-ask.
fallbackOnEmpty: false sections. The 8-K redemption / LOI detectors set this deliberately, because empty is the expected negative and falling through would re-pay a model on every non-event 8-K. Decide whether a deterministic decline is distinguishable from a genuine empty there, or whether those sections simply never list it.
Model resolution.secModelRecord dispatches on id shape and EnsureModelDownloadedTask expects a provider; "deterministic" needs an explicit bypass in both rather than falling through to the unknown-id error.
Per-section availability. Only some sections have a deterministic parser. Listing "deterministic" for a section that has none should be a startup-time error naming the section, not a silent no-op.
Provenance.model_id = "deterministic" is currently written as a post-hoc sentinel. Making it a real chain entry should let modelIndex drive that attribution instead, so stored provenance and the configured list cannot disagree.
#311 is the safety half and should land first — it stops active data loss and the coverage contract has to exist either way. This issue is the control half, and can be built on top of it.
Today the deterministic S-1 parsers preempt the AI extractors from inside each
modelExtractChainclosure. That placement is invisible at the call site, not configurable, and is the mechanism behind the data loss #311 fixes.Proposal: expose the deterministic path as an ordinary model id that an operator can place anywhere in a section's model list — first to try it before paying for a model, last as a fallback, or omit it entirely.
Most of this already exists
sectionRunner'sRunSectionArgsalready carries an ordered fallback chain, in production:plus
SectionPersistMeta.modelIndex(0= primary,1+= fallback index), and the docstring already guarantees "Fallbacks do not consumeVERIFICATION_ATTEMPTS".DETERMINISTIC_MODEL_ID = "deterministic"also already exists (s1/parseOfferingTables.ts:12), andsource: "deterministic"is already a field on the offering-terms and sponsor-promote row schemas.So this is mostly wiring an existing constant into an existing chain, not new machinery.
The gap that makes it more than wiring
The chain advances on
[]or a throw. It does not advance on "returned rows but cannot fill the columns the caller is about to clear."That is exactly the shape of the data loss.
parseRelatedPartyTablesreturns non-empty rows (the party observations) while hardcodingtransactions: [], andprocessFormS1has already clearedrelated_party_transactionfor the accession. To a naive chain that is a success: it stops, the AI never runs, and the transactions are gone with no dead letter.Consequences if position alone is the control:
Suggested design
Combine the ordering with the coverage contract #311 introduces, rather than choosing between them:
DeterministicPass.coversandRunSectionArgs.clearsas the safety contract.preempts(pass, clears)to make the deterministic entry decline — yield[]— when it cannot cover everything the section clears, instead of returning a partial result.modelIdsreports it andmodelIndexattributes it.emptyExtractsfallback carry the rest. This also retires the bespoke pre-loop code path Deterministic passes may not preempt what they cannot supply #311 adds.Net effect: a declining deterministic pass looks like an empty one, which the chain already knows how to handle, and an operator gets
--models-style control in production — which today only the eval harness (sec eval extract --models) has.Design details worth settling first
UNVERIFIED_SOURCE_SPANdead letter blaming a model that was never called. Verification failure on the deterministic entry should count as a decline (fall through) rather than a re-ask.fallbackOnEmpty: falsesections. The 8-K redemption / LOI detectors set this deliberately, because empty is the expected negative and falling through would re-pay a model on every non-event 8-K. Decide whether a deterministic decline is distinguishable from a genuine empty there, or whether those sections simply never list it.secModelRecorddispatches on id shape andEnsureModelDownloadedTaskexpects a provider;"deterministic"needs an explicit bypass in both rather than falling through to the unknown-id error."deterministic"for a section that has none should be a startup-time error naming the section, not a silent no-op.model_id = "deterministic"is currently written as a post-hoc sentinel. Making it a real chain entry should letmodelIndexdrive that attribution instead, so stored provenance and the configured list cannot disagree.Relationship to #311
#311 is the safety half and should land first — it stops active data loss and the coverage contract has to exist either way. This issue is the control half, and can be built on top of it.