PR 6: E05 — VLM + OCR hybrid (gpt-5.4) by alex-struk · Pull Request #160 · bcgov/ai-adoption-document-intelligence

alex-struk · 2026-05-12T22:17:22Z

Summary

Fifth and final "engine" experiment: a VLM + OCR hybrid that feeds both the page image and OCR-extracted text/layout to gpt-5.4 in the same prompt, betting that the model performs better with both modalities than with either alone. Benchmarked against the same canonical 40-sample dataset.

This PR adds the hybrid provider, a preflight script for the hybrid path, an iteration harness, and the full results.

Merge order: PR 6 of the experiments series. Depends on PRs 1 → 2 → 3 → 4 → 5.

What's added

Provider + workflow

VLM + OCR hybrid provider — combines OCR layout output with page images in a single VLM prompt
apps/temporal/src/scripts/templates/experiment-05-vlm-ocr-hybrid-workflow.json — workflow template
docs-md/graph-workflows/05-vlm-ocr-hybrid-OCR.md — workflow doc

Tooling

apps/temporal/src/scripts/preflight-hybrid.ts — validates Azure OpenAI deployment + PDF rendering + OCR layout extraction before benchmarking
apps/temporal/src/scripts/iterate-hybrid-extraction.ts — single-sample iteration harness for prompt development on the hybrid path

Results

experiments/results/05-vlm-ocr-hybrid/SUMMARY.md
experiments/results/05-vlm-ocr-hybrid/benchmark-run.json — canonical 40-sample run
experiments/results/05-vlm-ocr-hybrid/iteration/ — request/response/diff/layout captures, field descriptions, converged prompt

Notes for reviewers

Workflow test placement (tracked in AI-1641). This PR adds apps/temporal/src/experiment-05-vlm-ocr-hybrid.test.ts, which sits at the temporal/src root and bundles a real Temporal-cluster end-to-end layer (via the shared apps/temporal/src/__testlib__/integration-harness.ts) that is currently CI-gated / dev-only (process.env.CI ? describe.skip, run against localhost:7233). This placement is intentional for now and consistent across all five experiments. Relocating these into a general, feature-agnostic comprehensive-workflow integration suite — and enabling it in CI — is tracked in AI-1641, not this PR.
Stacked-PR note — azureOcr.readPlain is later removed (R1). This PR introduces a dedicated azureOcr.readPlain activity (azure-di-read-plain.ts) for the DI plain/markdown pre-pass. It is later removed (review item R1) in favour of the standard azureOcr.submit + azureOcr.poll activities with outputFormat=markdown — commit 46d59316, in the 08-part-2 slice (PR PR 12: E08 part 2 — extraction-engine review fixes + mock-only-paid integration harness #220). In the final merged state, azureOcr.readPlain is not a registered activity type and azure-di-read-plain.ts is deleted; the E05 workflow uses the standard submit/poll chain. Reviewing this slice in isolation shows the pre-consolidation state.
Stacked-PR note — dev scripts relocated later. The dev tooling this PR adds under apps/temporal/src/scripts/ (iterate-hybrid-extraction.ts, preflight-hybrid.ts) is moved out of the compiled worker src/ into apps/temporal/scripts/ in the 08-part-2 patch PR (PR 12: E08 part 2 — extraction-engine review fixes + mock-only-paid integration harness #220), commit e4036113 (excluded from the worker build; type-checked via tsconfig.scripts.json). In the final merged state they live at apps/temporal/scripts/; reviewing this slice in isolation shows them at the old path.
Hybrid is the most expensive variant in compute (image and OCR), so the preflight is more thorough — it catches deployment, image-rendering, and OCR-layout issues before a real run.
Benchmark JSON (~85k lines) + iteration layout JSON (~9.6k lines) dominate the diff.

Testing

preflight-hybrid.ts exercised before the canonical run.
iterate-hybrid-extraction.ts used during prompt convergence.
Full 40-sample canonical benchmark captured in benchmark-run.json.

Checklist

By submitting this pull request, I acknowledge that I have attempted to meet the following:

a self-review of my code

commented code particularly in hard-to-understand areas

corresponding changes to the documentation where required

changes tested to the best of my ability

no new errors or non-functional code

+
+  await fs.promises.writeFile(
+    path.join(ITERATION_DIR, "last-response.json"),
+    JSON.stringify(resultBody, null, 2),


+    elapsedMs,
+    rawResponseSummary,
+  });
+  await fs.promises.writeFile(path.join(ITERATION_DIR, "last-diff.md"), md);


+    elapsedMs,
+    rawResponseSummary,
+  });
+  await fs.promises.writeFile(path.join(ITERATION_DIR, "last-diff.md"), md);


+  };
+  await fs.promises.writeFile(
+    path.join(ITERATION_DIR, "last-response.json"),
+    JSON.stringify(responseForDisk, null, 2),


+      );
+      const outFile = path.join(outDir, "benchmark-run.json");
+      fs.mkdirSync(outDir, { recursive: true });
+      fs.writeFileSync(outFile, JSON.stringify(downloadResp.data, null, 2));


+    {
+      model: "mistral-document-ai-2512",
+      document: { type: "document_url", document_url: dataUrl },
+      document_annotation_format: annotationFormat,
+    },


+    "experiment-02",
+    `mistral-azure-ocr-response-${sampleId}-smoketest.json`,
+  );
+  await fs.promises.writeFile(outPath, JSON.stringify(data, null, 2));


…40-sample run Two-leg hybrid extraction: Azure DI prebuilt-layout produces markdown + bbox layout, then Azure OpenAI gpt-5.4 reads the document image alongside the OCR markdown with a strict-mode JSON Schema response_format and a trust-hierarchy system prompt ("prefer the image when image and OCR text disagree"). Mapper folds the upstream layout's word/line polygons into OCRResult, closing the bbox gap E04 documented. Per the user's session-start scope reduction, only variant 1 + gpt-5.4 is implemented; variant 2/3 + gpt-4o/gpt-5 deferred (the plumbing is in place — variant 3 is `params.includeBboxAnnotations: true`, deployment swaps via `params.azureOpenAiDeployment`). Canonical run (cb677a90, 40 samples): pass_rate 0.975 (39/40), f1.median 0.965 (tied with CU), f1.mean 0.941 (best across all 5 experiments), recall.mean 0.917 (best). Hybrid is the production-stack winner on this dataset. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

kmandryk · 2026-07-09T20:31:17Z

fails typescript compilation

Fixed in #220: experiment-05-vlm-ocr-hybrid.test.ts (with the other four experiment tests) was ported to develop's current GraphWorkflow API — graphWorkflow/getStatus from ./graph-workflow, GraphWorkflowConfig, ctx read via the getStatus query rather than off the result. Verified 2026-07-10 on the cumulative branch: apps/temporal npm run type-check → exit 0, and experiment-01..05 → 120/120 pass against the dev-stack Temporal (paid APIs mocked). Merges to develop via #221.

alex-struk · 2026-07-20T17:47:28Z

Superseded by #221 (cumulative extraction-experiments merge), now merged into develop. This branch's commits are all contained in develop; closing the review-only PR. Branch retained.

github-advanced-security AI found potential problems May 12, 2026

View reviewed changes

alex-struk force-pushed the experiment/05-vlm-ocr-hybrid branch 2 times, most recently from a7d099d to 224f9ad Compare June 30, 2026 22:48

alex-struk changed the base branch from develop to experiment/04-vlm-direct June 30, 2026 22:48

alex-struk mentioned this pull request Jun 30, 2026

Extraction experiments — cumulative merge (E01–E08 + improvements + review fixes + harness) #221

Merged

alex-struk force-pushed the experiment/04-vlm-direct branch from b17f39f to 5819a0c Compare July 2, 2026 16:19

alex-struk force-pushed the experiment/05-vlm-ocr-hybrid branch from 224f9ad to 288dce4 Compare July 2, 2026 16:19

alex-struk force-pushed the experiment/04-vlm-direct branch from 5819a0c to 13beb03 Compare July 2, 2026 16:39

alex-struk force-pushed the experiment/05-vlm-ocr-hybrid branch from 288dce4 to 7454edb Compare July 2, 2026 16:39

alex-struk marked this pull request as ready for review July 2, 2026 20:42

alex-struk requested review from NoorChasib, antsand, dbarkowsky and kmandryk as code owners July 2, 2026 20:42

kmandryk requested changes Jul 9, 2026

View reviewed changes

alex-struk closed this Jul 20, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

PR 6: E05 — VLM + OCR hybrid (gpt-5.4) - #160

PR 6: E05 — VLM + OCR hybrid (gpt-5.4)#160
alex-struk wants to merge 1 commit into
experiment/04-vlm-directfrom
experiment/05-vlm-ocr-hybrid

alex-struk commented May 12, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kmandryk Jul 9, 2026

Uh oh!

alex-struk Jul 10, 2026

Uh oh!

alex-struk commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

alex-struk commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What's added

Provider + workflow

Tooling

Results

Notes for reviewers

Testing

Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kmandryk Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

alex-struk Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

alex-struk commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

alex-struk commented May 12, 2026 •

edited

Loading