Skip to content

Make the deterministic parsers a listable model id, ordered alongside the AI models #312

Description

@sroussey

Today the deterministic S-1 parsers preempt the AI extractors from inside each modelExtractChain closure. That placement is invisible at the call site, not configurable, and is the mechanism behind the data loss #311 fixes.

Proposal: expose the deterministic path as an ordinary model id that an operator can place anywhere in a section's model list — first to try it before paying for a model, last as a fallback, or omit it entirely.

Most of this already exists

sectionRunner's RunSectionArgs already carries an ordered fallback chain, in production:

readonly extract: (text: string) => Promise<TRow[]>;
/** Tried in order when extract (and any earlier fallback) returns [] or throws … */
readonly emptyExtracts?: readonly ((text: string) => Promise<TRow[]>)[];
readonly fallbackOnEmpty?: boolean;
/** Ids tried for this section; named in the MODEL_EMPTY detail when length > 1. */
readonly modelIds?: readonly string[];

plus SectionPersistMeta.modelIndex (0 = primary, 1+ = fallback index), and the docstring already guarantees "Fallbacks do not consume VERIFICATION_ATTEMPTS".

DETERMINISTIC_MODEL_ID = "deterministic" also already exists (s1/parseOfferingTables.ts:12), and source: "deterministic" is already a field on the offering-terms and sponsor-promote row schemas.

So this is mostly wiring an existing constant into an existing chain, not new machinery.

The gap that makes it more than wiring

The chain advances on [] or a throw. It does not advance on "returned rows but cannot fill the columns the caller is about to clear."

That is exactly the shape of the data loss. parseRelatedPartyTables returns non-empty rows (the party observations) while hardcoding transactions: [], and processFormS1 has already cleared related_party_transaction for the accession. To a naive chain that is a success: it stops, the AI never runs, and the transactions are gone with no dead letter.

Consequences if position alone is the control:

  • deterministic first reproduces the current bug exactly;
  • deterministic last is safe but nearly useless — it runs only when the model already returned nothing, which is rare and forfeits the cost saving that motivates it.

Suggested design

Combine the ordering with the coverage contract #311 introduces, rather than choosing between them:

  1. Keep DeterministicPass.covers and RunSectionArgs.clears as the safety contract.
  2. Use preempts(pass, clears) to make the deterministic entry decline — yield [] — when it cannot cover everything the section clears, instead of returning a partial result.
  3. Register it as a chain entry with a real id, so modelIds reports it and modelIndex attributes it.
  4. Let the existing emptyExtracts fallback carry the rest. This also retires the bespoke pre-loop code path Deterministic passes may not preempt what they cannot supply #311 adds.

Net effect: a declining deterministic pass looks like an empty one, which the chain already knows how to handle, and an operator gets --models-style control in production — which today only the eval harness (sec eval extract --models) has.

Design details worth settling first

  • Span verification. The chain currently keeps span-verification re-asks on the model that produced the rows. For a pure function that is three identical re-asks followed by an UNVERIFIED_SOURCE_SPAN dead letter blaming a model that was never called. Verification failure on the deterministic entry should count as a decline (fall through) rather than a re-ask.
  • fallbackOnEmpty: false sections. The 8-K redemption / LOI detectors set this deliberately, because empty is the expected negative and falling through would re-pay a model on every non-event 8-K. Decide whether a deterministic decline is distinguishable from a genuine empty there, or whether those sections simply never list it.
  • Model resolution. secModelRecord dispatches on id shape and EnsureModelDownloadedTask expects a provider; "deterministic" needs an explicit bypass in both rather than falling through to the unknown-id error.
  • Per-section availability. Only some sections have a deterministic parser. Listing "deterministic" for a section that has none should be a startup-time error naming the section, not a silent no-op.
  • Provenance. model_id = "deterministic" is currently written as a post-hoc sentinel. Making it a real chain entry should let modelIndex drive that attribution instead, so stored provenance and the configured list cannot disagree.

Relationship to #311

#311 is the safety half and should land first — it stops active data loss and the coverage contract has to exist either way. This issue is the control half, and can be built on top of it.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions