Commit 81b2d1b
committed
report: strip speculation, dedupe per-engine section, drop failure-mode samples
Tighten the report on three axes: remove speculative claims that aren't
backed by data or research docs, cut sections that duplicate content
covered elsewhere, and shorten verbose plot commentaries.
Speculation removed:
- "gpt-5.2 was tuned for Content Understanding's structured-extraction
workload" (3 places) — not supported by the CU research doc; CU lets
customers plug in any compatible Foundry model, no documented tuning
for gpt-5.2 specifically. Reframed as "we don't have a confirmed
mechanism for the ordering — it's what the dataset shows."
- "HR0081 (10) is the densest sample" — not true; synth-full (1) has
more handwritten content (HR0081 mostly has 0s in income cells).
Reframed as "the sample where E03 sent the most tokens" with a note
that this is not the same as the densest-by-content sample.
- "CU drops small handwritten zeros at high rate by design" — not
verifiable; we don't know they're small or that drop-by-design is
the mechanism. Reframed as observation: CU's stage-1 sometimes
omits the 0 from OCR or mis-classifies it via the selectionMark
detector. Note that the OCR-vs-image-recovery split inside E08 is
not separately broken out in the benchmark data.
- "Applicant-column income lines are harder because of handwriting
variation" — the more likely explanation is that applicant cells
are filled on more samples than spouse cells, so more opportunities
for errors.
Form description below the image dropped (we can't see the form).
"None of them are trained on this specific form" corrected — E00 and
E01 are trained on the form; the other engines aren't.
Sections removed:
- "What this report compares" (duplicated Overview).
- "Failure-mode samples" (anecdotal, covered by per-sample heatmap).
- E08 "Cost trade-off — measured" subsection (all content duplicates
the Cost per page section).
- E08 "What this means architecturally" and "Net interpretation"
subsections (duplicate the Headline finding and Reflection).
- The deep "Why E08 beats E03" 4-point mechanism breakdown (the same
three architectural levers are stated in Headline finding, Headline
aggregate metrics observation #1, the blank-vs-zero section, and
Reflection point #3).
- "Previous version of this table reported lower numbers under an
earlier evaluator schema" — readers don't care about prior versions.
Per-engine section renamed "Per-engine deep dive" → "Per-engine notes",
with intro re-scoped to "what each engine is and how we ran it".
Strengths/weaknesses bullets and errors-by-category one-liners removed
from every engine sub-section — those reproduce information already in
the per-category table, per-field heatmap, and Reflection.
Per-field accuracy heatmap and Per-sample F1 heatmap commentaries
shortened to bullet summaries; verbose paragraph explanations removed.
Per-category-bars clarification added: ensemble (E06) ties each
category's leader by construction (it *is* that engine for that
category); the aggregate lift comes from no single engine winning all
categories.
Industry-context link to hybrid-model-research.md dropped per request.
Net: 992 → 825 lines (~17% shorter), with denser per-engine notes,
fewer speculative attributions, and clearer separation between
observation and inference.1 parent 0382217 commit 81b2d1b
1 file changed
Lines changed: 45 additions & 232 deletions
0 commit comments