Skip to content

Commit d075ca1

Browse files
strukalexclaude
andcommitted
feat(experiment-05): VLM + OCR hybrid provider (gpt-5.4) + canonical 40-sample run
Two-leg hybrid extraction: Azure DI prebuilt-layout produces markdown + bbox layout, then Azure OpenAI gpt-5.4 reads the document image alongside the OCR markdown with a strict-mode JSON Schema response_format and a trust-hierarchy system prompt ("prefer the image when image and OCR text disagree"). Mapper folds the upstream layout's word/line polygons into OCRResult, closing the bbox gap E04 documented. Per the user's session-start scope reduction, only variant 1 + gpt-5.4 is implemented; variant 2/3 + gpt-4o/gpt-5 deferred (the plumbing is in place — variant 3 is `params.includeBboxAnnotations: true`, deployment swaps via `params.azureOpenAiDeployment`). Canonical run (cb677a90, 40 samples): pass_rate 0.975 (39/40), f1.median 0.965 (tied with CU), f1.mean 0.941 (best across all 5 experiments), recall.mean 0.917 (best). Hybrid is the production-stack winner on this dataset. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 3cc81f5 commit d075ca1

30 files changed

Lines changed: 120339 additions & 3 deletions

apps/backend-services/src/workflow/activity-registry.ts

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,14 @@ export const REGISTERED_ACTIVITY_TYPES: Record<string, RegisteredActivityType> =
4747
description:
4848
"VLM-direct extraction (Azure OpenAI chat completions with vision input + strict JSON schema response_format)",
4949
},
50+
"vlmOcrHybrid.extract": {
51+
description:
52+
"VLM + OCR hybrid extraction (Azure DI prebuilt-layout markdown + Azure OpenAI chat completions with vision + strict JSON schema response_format)",
53+
},
54+
"azureOcr.readPlain": {
55+
description:
56+
"Azure DI prebuilt-layout (markdown content + per-line/per-word polygons; no field extraction). Sync submit+poll wrapper for the VLM hybrid pre-pass.",
57+
},
5058
"ocr.cleanup": { description: "Post-OCR text normalization" },
5159
"ocr.enrich": {
5260
description: "Enrich OCR results using field schema and optional LLM",

0 commit comments

Comments
 (0)