eval s1 risk-factors: pin the category convention, and stop double-counting long-caption drift

[sroussey]

Findings from the first full golden sweep of risk-factors after #260 merged:

sec eval s1 --extractors risk-factors --reference golden

Result — claude-haiku-4-5 over 42 real S-1 sections, 3,029 golden rows:

agree	recall	prec	rows	dist	ok
91%	89%	83%	3301	3300	41/42

#260 itself came out clean and needs nothing here: zero MIXED_CAPTION_SHAPE failures across all 42 filings (the risk its U.S. initialism fix was flagged for), and the echo path fired once, taking the drop branch. This issue is about three separate things the sweep exposed. None is a regression from #260; the sweep is simply the first time these numbers have been looked at.

---

1. Category is half the score and is systematically empty (highest leverage)

EVAL_EXTRACTORS["risk-factors"] sets compareFields: ["headline", "category"], so category is one of two scored fields — roughly half of agree. The disagreement list is full of category-only mismatches, and they cluster in a way that looks like an unstated convention rather than a capability gap:

· category: "Summary of Risk Factors"   → ""
· category: "Risks Relating to our Securities" → ""
· category: "Risk Factor Summary" → "Risks Related to Our Business"

The third line is the tell: the model returned a more specific category than the label. So on a filing whose section is an Item 105(b) summary list, the label says the category is the summary heading ("Summary of Risk Factors", "Risk Factor Summary") while the model reports either nothing or the underlying risk category. Neither answer is obviously wrong — the question was never specified.

This is the same shape as the two worked examples already in CLAUDE.md (the ownership group subtotal, the zero-holding rows): a convention the prompt never stated, where the labels and the model each picked a different reasonable reading, and the score reads as a model deficiency.

Suggested fix. Decide the convention, then pin it in both places:

• state it in riskFactorsInstructions() — for a caption taken from a summary list, is category the summary heading, or the risk category the caption belongs to?
• re-label the affected golden entries in goldenS1Labels.ts to match.

My suggestion is the summary heading as printed, on the grounds the table already uses elsewhere: the row should be faithful to the filing, and the underlying category is a derivation on top. But this needs someone to read one of the affected filings and decide — s1_1822912, s1_1853138 and s1_1880613 are the clearest instances.

Until it is pinned, agree for this extractor is not measuring extraction quality.

---

2. Most "missing" rows are also "extra" rows

s1_1507957 is the clean case — missing 1, extra 1, same displayed prefix. The same shape appears in s1_1083743 (3 missing, the first 3 of 5 extra identical), s1_1918102, s1_2091349 and others.

This is not punctuation drift: scoreExtraction's normalize() already folds typographic quotes and dashes, lowercases, strips commas and non-decimal periods, and collapses whitespace. The captions genuinely differ past the 38-char display truncation. Given SPAC risk captions run 200+ chars and verifyRow requires the headline to be verbatim section text, the model is reproducing a slightly different span of one long caption.

The consequence is that a single near-miss is counted twice — once against recall (the golden row went unmatched) and once against precision (the model row went unmatched). That makes 83% precision read as hallucination when over-production is essentially nil: rows 3301 vs dist 3300 is one duplicate in the entire sweep.

Suggested fixes, in preference order:

1. Report a third bucket in the disagreement output — near-misses aligned by prefix or edit distance — so a reader can see how much of missing/extra is one drifting caption versus a genuine miss. Diagnostic only; leaves scoring untouched.
2. Print the full caption (or a much longer prefix) for risk-factors disagreements. At 38 chars, formulaic SPAC captions are indistinguishable, which is what makes the current output hard to act on.
3. Consider whether keyField: "headline" should align on a prefix or normalized-token basis for this extractor specifically. Riskier — it could mask real misses — so only after (1) shows how big the effect is.

---

3. One filing loses all 42 of its rows to an array-as-string response

s1_2087989 is the sole ok failure:

#/risks: Expected `[ {...}, {...} ]` (string) in `#/risks` to be of type `array`

The model returned a valid JSON array serialized as a string into the risks field. CLAUDE.md already documents this for the reference model ("strong models intermittently emit a nested array as a JSON string the strict schema rejects"); here a candidate hit it and runStructured's maxRetries: 1 burned both attempts on the same malformation.

Cost: 42 golden rows, about 13% of the total recall gap, from one filing.

Suggested fix. A narrow repair in the structured-generation path: when validation fails solely because an array-typed property arrived as a string, attempt one JSON.parse of that string and re-validate. It is unambiguous (the string parses to an array of the right shape or it does not), it is cheap (no extra model call), and it recovers a whole filing. Failing that, a targeted extra retry for this specific validation error is a cruder alternative.

---

Not in scope here

risk-factors is currently disabled: true in EVAL_EXTRACTORS, so a bare sec eval s1 skips it and explicit --extractors risk-factors is its only coverage. Worth revisiting once (1) is pinned and the numbers mean something — but that decision belongs with #275, which is already touching how the s1 sweep honors disabled.