Skip to content

Commit 73c0bf8

Browse files
strukalexclaude
andcommitted
report: strip internal source-code links; fix tilde strikethrough and label crowding
- REPORT.md: drop "live source" links and per-engine path references so the report stands alone - md-to-pdf.js: treat lone ~ as literal so "~$0.003" / "~20×" no longer render as strikethrough - compare-engines.py: rotate error-class x-axis labels when any engine label exceeds 16 chars - regenerate REPORT.pdf and sample image Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 81b2d1b commit 73c0bf8

5 files changed

Lines changed: 45 additions & 12 deletions

File tree

experiments/results/report/REPORT.md

Lines changed: 7 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -459,7 +459,7 @@ For production workloads where literal zeros matter (income reporting, financial
459459

460460
# Appendix A — E06 ensemble combiner
461461

462-
This appendix documents the ensemble combiner that produces the E06 row in the comparison tables above. The full source data and benchmark export live in [`../06-engine-ensemble/`](../06-engine-ensemble/).
462+
This appendix documents the ensemble combiner that produces the E06 row in the comparison tables above.
463463

464464
## Production note (read this first)
465465

@@ -469,7 +469,7 @@ If the architecture observations in the body of this report led to E08 alone bei
469469

470470
## Strategies explored
471471

472-
Six deployable strategies plus one oracle baseline ([`../06-engine-ensemble/scripts/build-ensemble.py`](../06-engine-ensemble/scripts/build-ensemble.py)):
472+
Six deployable strategies plus one oracle baseline:
473473

474474
| code | how it picks |
475475
|---|---|
@@ -505,8 +505,6 @@ S1 and S2 tie at F1.mean 0.984; the null-fallback in S2 doesn't fire often enoug
505505

506506
**Per-category specialist routing wins outright.** With eight engines and clear category leaders, the weighted-majority strategies (S3/S4/S5/S6) trail S1 by 1.1–1.4 pp F1.mean: their votes drag the result toward the average of the engine pool rather than the specialist's answer. The per-category leaders are clearly differentiated — E01 dominates sin, E07 dominates signature, E08 dominates income, E03 dominates checkboxes — so trusting the specialist unconditionally outperforms diluting its vote with non-specialists.
507507

508-
Full strategy CSV: [`../06-engine-ensemble/data/strategy-comparison.csv`](../06-engine-ensemble/data/strategy-comparison.csv).
509-
510508
## Per-category accuracy — E06 vs the leader of each category
511509

512510
| category | E06 (S1) | best single | engine | delta |
@@ -556,7 +554,7 @@ E06 closes roughly **half** of the F1.mean gap between the best single engine (0
556554

557555
## E06 residual errors
558556

559-
The per-sample mismatch table for E06 is at [`../06-engine-ensemble/iteration/errors-for-gt-cleanup.md`](../06-engine-ensemble/iteration/errors-for-gt-cleanup.md) **63 mismatches across 23 samples** (17 samples have zero mismatches under strict eval). This is the lowest mismatch count of any engine on this dataset:
557+
E06 has **63 mismatches across 23 samples** (17 samples have zero mismatches under strict eval). This is the lowest mismatch count of any engine on this dataset:
560558

561559
| engine | mismatches | samples with ≥1 mismatch |
562560
|---|---|---|
@@ -575,14 +573,14 @@ E06's structural strength shows up not just in fewer total mismatches but also i
575573
The residual error categories on E06 break down into:
576574
- **Single-character handwriting** (X-marks, isolated `0`s). When the category specialist misreads the character, S1 has no fallback because it doesn't consult the other engines.
577575
- **Numeric blank-vs-zero ambiguity** on a small handful of income-amount fields where the form has a stray pen mark visible. E08 (the income specialist) extracts `0`; the GT is `""`.
578-
- **One-of-array GT not yet covering an engine's format variant.** Caught by `promote-gt-format-variants.ts` and absorbed in subsequent GT cleanup passes.
576+
- **One-of-array GT not yet covering an engine's format variant.** Caught by the GT format-variant promotion step and absorbed in subsequent GT cleanup passes.
579577
- **Genuine OCR misreads**`5` vs `8`, `1` vs `7` confusions on dense handwriting. The irreducible per-engine errors the category specialist makes that no router can fix without consulting another engine.
580578

581579
---
582580

583581
# Appendix B — Extraction prompts (verbatim)
584582

585-
The prompts below are the *global instruction text* sent to each generative engine, captured verbatim from `experiments/results/<engine>/iteration/prompt.md` as of this report. They are paired with a 74-field JSON schema (per-field descriptions in `iteration/field-descriptions.json` for each engine — not included here for length; available alongside each engine's iteration kit) which carries the structural typing for output values.
583+
The prompts below are the *global instruction text* sent to each generative engine, captured verbatim as of this report. They are paired with a 74-field JSON schema carrying the per-field type declarations and descriptions (not included here for length).
586584

587585
E00 has no prompt (it's a supervised trained template — extraction behaviour is encoded in the labelled training set, not in instructions).
588586

@@ -613,8 +611,6 @@ Conventions:
613611
Be conservative: if a number is illegible, return null (treat as blank). Do NOT guess values that aren't visibly written on the form.
614612
```
615613

616-
Live source: [experiments/results/05-vlm-ocr-hybrid/iteration/prompt.md](../05-vlm-ocr-hybrid/iteration/prompt.md) (mirrored byte-for-byte under each of `04-vlm-direct/`, `07-vlm-ocr-hybrid-gpt-4o/`, and `08-vlm-ocr-hybrid-gpt-5.2/`).
617-
618614
## E03 — Azure Content Understanding prompt
619615

620616
CU receives this prompt as part of the analyzer configuration's `description` field. CU's managed "Contextualization" layer wraps it server-side with additional schema metadata, in-context examples, and grounding instructions before sending the final prompt to gpt-5.2 — we do not see and cannot edit that wrapping. The text below is the *only* prompt-side lever we control on E03.
@@ -639,7 +635,7 @@ Conventions:
639635
Be conservative: if a number is illegible, return null (treat as blank). Do NOT guess values that aren't visibly written on the form.
640636
```
641637

642-
Live source: [experiments/results/03-content-understanding/iteration/prompt.md](../03-content-understanding/iteration/prompt.md). The CU prompt is a near-twin of the shared VLM/hybrid prompt — only minor wording differs (no thousands-separator clause, no parenthesis/dot guidance on phone formatting, no `→ 03 / SEP → 09` example on date abbreviation). Worth noting: both prompts are short (~18 lines, ~2.5 KB). The accuracy gap between E03 and E08 is therefore *not* a prompt-quality story — both engines get roughly the same instructions; the gap traces to whether the image reaches the LLM and how the OCR layer is tuned (see the [E08 notes](#e08--gpt-52-vlm--azure-di-layout-hybrid)).
638+
The CU prompt is a near-twin of the shared VLM/hybrid prompt — only minor wording differs (no thousands-separator clause, no parenthesis/dot guidance on phone formatting, no `→ 03 / SEP → 09` example on date abbreviation). Worth noting: both prompts are short (~18 lines, ~2.5 KB). The accuracy gap between E03 and E08 is therefore *not* a prompt-quality story — both engines get roughly the same instructions; the gap traces to whether the image reaches the LLM and how the OCR layer is tuned (see the [E08 notes](#e08--gpt-52-vlm--azure-di-layout-hybrid)).
643639

644640
## E02 — Mistral on Azure AI Foundry prompt
645641

@@ -821,5 +817,5 @@ truly empty. If you can read SOMETHING inside the box (even partially),
821817
return what you see.
822818
```
823819

824-
Live source: [experiments/results/02-mistral-doc-ai-azure/iteration/prompt.md](../02-mistral-doc-ai-azure/iteration/prompt.md). Note the prompt opens with *"Read the form image carefully"* even though Mistral's annotation pass does not actually receive the image — that wording is aspirational, preserved from earlier prompt iterations. In practice the annotation model sees only the OCR text output of Mistral's first stage, which is why the prompt has to encode the form layout (income-row order, checkbox grid) explicitly.
820+
Note the prompt opens with *"Read the form image carefully"* even though Mistral's annotation pass does not actually receive the image — that wording is aspirational, preserved from earlier prompt iterations. In practice the annotation model sees only the OCR text output of Mistral's first stage, which is why the prompt has to encode the form layout (income-row order, checkbox grid) explicitly.
825821

1.2 MB
Binary file not shown.
-42.7 KB
Loading

scripts/benchmark analysis/compare-engines.py

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -671,7 +671,14 @@ def plot_error_class_breakdown(engines: list[Engine], out_dir: Path) -> None:
671671
ax.text(x[i], bottoms[i] + max(bottoms) * 0.01, f"total {e.total_errors}",
672672
ha="center", va="bottom", fontsize=9, weight="bold")
673673
ax.set_xticks(x)
674-
ax.set_xticklabels([e.label for e in engines])
674+
# Rotate engine labels when any exceeds ~16 chars (e.g. expanded model
675+
# names like "Neural Custom Model (V2 current)") so adjacent bar labels
676+
# don't visually collide. Same rotation as the per-field heatmap uses.
677+
max_label_len = max((len(e.label) for e in engines), default=0)
678+
if max_label_len > 16:
679+
ax.set_xticklabels([e.label for e in engines], rotation=15, ha="right", fontsize=9)
680+
else:
681+
ax.set_xticklabels([e.label for e in engines])
675682
ax.set_ylabel("Errors")
676683
ax.set_title("Error-class breakdown (missing / extra / wrong)")
677684
ax.legend(loc="upper right")

scripts/benchmark analysis/md-to-pdf.js

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,36 @@ try {
4343
process.exit(1);
4444
}
4545
marked.setOptions({ gfm: true, breaks: false });
46+
// Disable GFM's single-tilde strikethrough. Marked's default treats ~text~ as
47+
// <del>...</del>, which collides with our use of ~ as the "approximately"
48+
// sign (~$0.046, ~37%, ~20×) and produces accidental strikethrough whenever
49+
// two ~'s happen to land on either side of a non-whitespace boundary
50+
// (e.g. "~$0.003/page (~20×" → "<del>$0.003/page (</del>20×").
51+
//
52+
// A `tokenizer.del` override that returned false didn't help — when it
53+
// declines, marked falls back to the default del tokenizer. Instead we
54+
// register an inline extension that *consumes* a lone ~ as a literal text
55+
// token before the strikethrough tokenizer ever gets to see it. The
56+
// extension intentionally skips ~~...~~, so real GFM strikethrough still
57+
// works.
58+
marked.use({
59+
extensions: [
60+
{
61+
name: "literalTilde",
62+
level: "inline",
63+
start(src) {
64+
const i = src.indexOf("~");
65+
return i === -1 ? undefined : i;
66+
},
67+
tokenizer(src) {
68+
const match = /^(?<!~)~(?!~)/.exec(src);
69+
if (match) {
70+
return { type: "text", raw: "~", text: "~" };
71+
}
72+
},
73+
},
74+
],
75+
});
4676

4777
// ---------------------------------------------------------------------------
4878
// Argument parsing.

0 commit comments

Comments
 (0)