Skip to content

Commit 81b2d1b

Browse files
committed
report: strip speculation, dedupe per-engine section, drop failure-mode samples
Tighten the report on three axes: remove speculative claims that aren't backed by data or research docs, cut sections that duplicate content covered elsewhere, and shorten verbose plot commentaries. Speculation removed: - "gpt-5.2 was tuned for Content Understanding's structured-extraction workload" (3 places) — not supported by the CU research doc; CU lets customers plug in any compatible Foundry model, no documented tuning for gpt-5.2 specifically. Reframed as "we don't have a confirmed mechanism for the ordering — it's what the dataset shows." - "HR0081 (10) is the densest sample" — not true; synth-full (1) has more handwritten content (HR0081 mostly has 0s in income cells). Reframed as "the sample where E03 sent the most tokens" with a note that this is not the same as the densest-by-content sample. - "CU drops small handwritten zeros at high rate by design" — not verifiable; we don't know they're small or that drop-by-design is the mechanism. Reframed as observation: CU's stage-1 sometimes omits the 0 from OCR or mis-classifies it via the selectionMark detector. Note that the OCR-vs-image-recovery split inside E08 is not separately broken out in the benchmark data. - "Applicant-column income lines are harder because of handwriting variation" — the more likely explanation is that applicant cells are filled on more samples than spouse cells, so more opportunities for errors. Form description below the image dropped (we can't see the form). "None of them are trained on this specific form" corrected — E00 and E01 are trained on the form; the other engines aren't. Sections removed: - "What this report compares" (duplicated Overview). - "Failure-mode samples" (anecdotal, covered by per-sample heatmap). - E08 "Cost trade-off — measured" subsection (all content duplicates the Cost per page section). - E08 "What this means architecturally" and "Net interpretation" subsections (duplicate the Headline finding and Reflection). - The deep "Why E08 beats E03" 4-point mechanism breakdown (the same three architectural levers are stated in Headline finding, Headline aggregate metrics observation #1, the blank-vs-zero section, and Reflection point #3). - "Previous version of this table reported lower numbers under an earlier evaluator schema" — readers don't care about prior versions. Per-engine section renamed "Per-engine deep dive" → "Per-engine notes", with intro re-scoped to "what each engine is and how we ran it". Strengths/weaknesses bullets and errors-by-category one-liners removed from every engine sub-section — those reproduce information already in the per-category table, per-field heatmap, and Reflection. Per-field accuracy heatmap and Per-sample F1 heatmap commentaries shortened to bullet summaries; verbose paragraph explanations removed. Per-category-bars clarification added: ensemble (E06) ties each category's leader by construction (it *is* that engine for that category); the aggregate lift comes from no single engine winning all categories. Industry-context link to hybrid-model-research.md dropped per request. Net: 992 → 825 lines (~17% shorter), with denser per-engine notes, fewer speculative attributions, and clearer separation between observation and inference.
1 parent 0382217 commit 81b2d1b

1 file changed

Lines changed: 45 additions & 232 deletions

File tree

0 commit comments

Comments
 (0)