Commit d075ca1
feat(experiment-05): VLM + OCR hybrid provider (gpt-5.4) + canonical 40-sample run
Two-leg hybrid extraction: Azure DI prebuilt-layout produces markdown +
bbox layout, then Azure OpenAI gpt-5.4 reads the document image alongside
the OCR markdown with a strict-mode JSON Schema response_format and a
trust-hierarchy system prompt ("prefer the image when image and OCR text
disagree"). Mapper folds the upstream layout's word/line polygons into
OCRResult, closing the bbox gap E04 documented.
Per the user's session-start scope reduction, only variant 1 + gpt-5.4
is implemented; variant 2/3 + gpt-4o/gpt-5 deferred (the plumbing is in
place — variant 3 is `params.includeBboxAnnotations: true`, deployment
swaps via `params.azureOpenAiDeployment`).
Canonical run (cb677a90, 40 samples): pass_rate 0.975 (39/40),
f1.median 0.965 (tied with CU), f1.mean 0.941 (best across all 5
experiments), recall.mean 0.917 (best). Hybrid is the production-stack
winner on this dataset.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>1 parent 3cc81f5 commit d075ca1
30 files changed
Lines changed: 120339 additions & 3 deletions
File tree
- apps
- backend-services/src/workflow
- temporal/src
- __fixtures__/experiment-05
- activities
- ocr-providers/vlm-ocr-hybrid
- scripts
- docs-md
- graph-workflows
- templates
- experiments/results/05-vlm-ocr-hybrid
- iteration
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
47 | 47 | | |
48 | 48 | | |
49 | 49 | | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
50 | 58 | | |
51 | 59 | | |
52 | 60 | | |
| |||
0 commit comments