PR 5: E04 — VLM-direct (gpt-5.4) by alex-struk · Pull Request #159 · bcgov/ai-adoption-document-intelligence

alex-struk · 2026-05-12T22:17:20Z

Summary

Fourth experiment: VLM-direct — feed page images straight to a vision-language model (gpt-5.4) and let it extract structured fields in a single pass, with no OCR layer in between. Benchmarked against the same canonical 40-sample dataset.

This PR adds the VLM-direct provider, a preflight script that validates the deployment + image rendering pipeline before a real run, an iteration harness for prompt development, and the full results. It also lands POST_BENCHMARK_FOLLOWUPS.md — a cross-experiment work-tracking doc for follow-ups that don't belong to a single engine.

Merge order: PR 5 of the experiments series. Depends on PRs 1 → 2 → 3 → 4.

What's added

Provider + workflow

VLM-direct provider (single-pass image → structured output on gpt-5.4)
apps/temporal/src/scripts/templates/experiment-04-vlm-direct-workflow.json — workflow template
docs-md/graph-workflows/04-vlm-direct-OCR.md — workflow doc

Tooling

apps/temporal/src/scripts/preflight-vlm.ts — verifies the Azure OpenAI deployment + PDF → image rendering before a benchmark run (saves expensive failed runs)
apps/temporal/src/scripts/iterate-vlm-extraction.ts — single-sample iteration harness for prompt development

Results

experiments/results/04-vlm-direct/SUMMARY.md — with corrected iteration-vs-benchmark framing
experiments/results/04-vlm-direct/benchmark-run.json — canonical 40-sample run
experiments/results/04-vlm-direct/iteration/ — request/response/diff captures, field descriptions, converged prompt

Cross-experiment

experiments/POST_BENCHMARK_FOLLOWUPS.md — running list of follow-ups discovered during benchmarking that span multiple engines (e.g., GT format normalization, evaluator strictness). Drives the "improve" branches in PRs 7–9.

Notes for reviewers

Workflow test placement (tracked in AI-1641). This PR adds apps/temporal/src/experiment-04-vlm-direct.test.ts, which sits at the temporal/src root and bundles a real Temporal-cluster end-to-end layer (via the shared apps/temporal/src/__testlib__/integration-harness.ts) that is currently CI-gated / dev-only (process.env.CI ? describe.skip, run against localhost:7233). This placement is intentional for now and consistent across all five experiments. Relocating these into a general, feature-agnostic comprehensive-workflow integration suite — and enabling it in CI — is tracked in AI-1641, not this PR.
Stacked-PR note — dev scripts relocated later. The dev tooling this PR adds under apps/temporal/src/scripts/ (iterate-vlm-extraction.ts, preflight-vlm.ts) is moved out of the compiled worker src/ into apps/temporal/scripts/ in the 08-part-2 patch PR (PR 12: E08 part 2 — extraction-engine review fixes + mock-only-paid integration harness #220), commit e4036113 (excluded from the worker build; type-checked via tsconfig.scripts.json). In the final merged state they live at apps/temporal/scripts/; reviewing this slice in isolation shows them at the old path.
This is the first VLM-only experiment (no OCR-text step), so the workflow is simpler than E02/E03 but the prompt is more complex.
The benchmark JSON (~88k lines) again dominates the diff. Code changes are confined to scripts + workflow template.

Testing

preflight-vlm.ts run successfully before the canonical benchmark.
iterate-vlm-extraction.ts used during prompt convergence.
Full 40-sample canonical benchmark captured in benchmark-run.json.

Checklist

By submitting this pull request, I acknowledge that I have attempted to meet the following:

a self-review of my code

commented code particularly in hard-to-understand areas

corresponding changes to the documentation where required

changes tested to the best of my ability

no new errors or non-functional code

+
+  await fs.promises.writeFile(
+    path.join(ITERATION_DIR, "last-response.json"),
+    JSON.stringify(resultBody, null, 2),


+    elapsedMs,
+    rawResponseSummary,
+  });
+  await fs.promises.writeFile(path.join(ITERATION_DIR, "last-diff.md"), md);


+    elapsedMs,
+    rawResponseSummary,
+  });
+  await fs.promises.writeFile(path.join(ITERATION_DIR, "last-diff.md"), md);


+  };
+  await fs.promises.writeFile(
+    path.join(ITERATION_DIR, "last-response.json"),
+    JSON.stringify(responseForDisk, null, 2),


+      );
+      const outFile = path.join(outDir, "benchmark-run.json");
+      fs.mkdirSync(outDir, { recursive: true });
+      fs.writeFileSync(outFile, JSON.stringify(downloadResp.data, null, 2));


+    {
+      model: "mistral-document-ai-2512",
+      document: { type: "document_url", document_url: dataUrl },
+      document_annotation_format: annotationFormat,
+    },


+    "experiment-02",
+    `mistral-azure-ocr-response-${sampleId}-smoketest.json`,
+  );
+  await fs.promises.writeFile(outPath, JSON.stringify(data, null, 2));


…ple run Implements E04 single-pass VLM-direct extraction. Per the user's session-start scope reduction, only variant 1 + gpt-5.4 is in scope; chain-of-thought, self-consistency, and the gpt-4o/gpt-5/gpt-5.5 axes are deferred. Provider: apps/temporal/src/ocr-providers/vlm-direct/ - vlm-types.ts parsed { fields, source_quotes } payload types - vlm-prompt-builder.ts FieldDefinition[] -> strict-mode JSON Schema - vlm-direct-extract.ts the chat-completions activity (PDF guard, base64 image, structured-output strict mode, parse, map) - vlm-to-ocr-result.ts payload -> canonical OCRResult with evidence- based confidence synthesis (0.95 with quote / 0.50 without) Activity: vlmDirect.extract registered in all three registries (runtime function, workflow-safe constant, backend allow-list); 30 attempts x 15s x 1.5x x 60s cap retry policy mirroring CU + Mistral on Foundry. Workflow: docs-md/graph-workflows/templates/experiment-04-vlm-direct- workflow.json -- sync chain (prepareFileData -> vlmDirectExtract -> cleanup -> checkConfidence -> reviewSwitch -> humanReview/storeResults). Auto-discovered by seedExperimentWorkflows(). gpt-5.4 default. Iteration kit: experiments/results/04-vlm-direct/iteration/ -- copied from E03 (SDPR-form quirks port verbatim) + iterate-vlm-extraction.ts direct-call smoke-test script. Pre-flight: scripts/preflight-vlm.ts asserts env vars + deployment reachability + vision capability + strict-mode round-trip + DB state before any paid call. Runs in ~5s. Tests: 16 static + 4 fixture-aware + 2 runtime (CI-gated) workflow- level tests + 16 unit tests on prompt-builder + 16 on mapper. All 55 non-runtime tests pass. Benchmark run d5db8a69-c802-49c1-9b71-492586d459fd: pass_rate 0.925 (37/40) f1.median 0.943 f1.mean 0.911 precision.mean 0.972 recall.mean 0.864 matchedFields.median 66 (of 74) wallclock ~3:52 (5.8s/sample) -- ~3.8x faster than E03 (CU) evaluator schema-aware (fuzzy@0.85, pass threshold 0.8) Tied with E02 on f1.median + matchedFields.median, ~2pp behind E03 on f1.median, ~3 fields behind on matchedFields.median, but ~3.8x faster per sample because VLM-direct skips CU's content-extraction layer. Source_quotes hallucination guard turned out to be a structural signal (useful for verifying strict-mode is on) rather than a per-field accuracy signal -- gpt-5.4 emits a quote even for wrong values. Documented in SUMMARY.md and 04-vlm-direct-OCR.md. PDF support deferred (canonical dataset is 100% JPEG; activity throws on PDF inputs with a clear error message). Documented in SUMMARY.md "Gaps". Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

The first SUMMARY framed the gap between iteration's 70.3% per-field accuracy and the benchmark's 0.943 f1.median as "iteration was overly strict, benchmark's fuzzy-0.85 was more forgiving." Looking at the per-sample numbers, that framing is misleading: - synth-full (1) iteration scorer (strict-with-normalisation): 52/74 = 70.3% match - synth-full (1) benchmark scorer (fuzzy@0.85): 54/74 matched, recall 0.730, F1 0.844 Fuzzy gave 2 extra credits on close-but-not-exact OCR misreads. The fuzzy-vs-strict gap is small. The actual driver is that synth-full (1) is genuinely a hard sample for gpt-5.4 (F1 0.844 << run median 0.943); the headline aggregate was pulled up by easier samples scoring 0.95+, not by the scorer being lenient. Also affirm explicitly that the schema-aware + fuzzy@0.85 + threshold- 0.8 evaluator config is identical across E01-E04 (wired in seed.ts: 2044-2062), so the cross-experiment comparison is apples-to-apples. Updates: - Iteration-section framing: scorer-difference is small; sample selection is the dominant factor. - Comparison-vs-E01-E03 section: explicit "identical evaluator" note + per-sample variance caveat. - Retrospective lesson #1: rewritten — single-sample iteration is a poor proxy for benchmark performance because per-sample variance dominates. - Retrospective lesson #12: rewritten — pick 2-3 samples spanning the dataset rather than relying on fuzzy-vs-strict scoring difference to bridge the gap. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…work Track the strict-equality re-evaluation, cost normalisation, latency percentile aggregation, and iteration-scorer rebaseline as items to do after E05 lands. None blocks the experiment chain; all benefit from having every run's benchmark-run.json committed first. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

kmandryk · 2026-07-09T20:23:47Z

npm run type-check fails

Fixed in #220: experiment-04-vlm-direct.test.ts (with the other four experiment tests) was ported to develop's current GraphWorkflow API — graphWorkflow/getStatus from ./graph-workflow, GraphWorkflowConfig, ctx read via the getStatus query rather than off the result. Verified 2026-07-10 on the cumulative branch: apps/temporal npm run type-check → exit 0, and experiment-01..05 → 120/120 pass against the dev-stack Temporal (paid APIs mocked). Merges to develop via #221.

alex-struk · 2026-07-20T17:47:26Z

Superseded by #221 (cumulative extraction-experiments merge), now merged into develop. This branch's commits are all contained in develop; closing the review-only PR. Branch retained.

github-advanced-security AI found potential problems May 12, 2026

View reviewed changes

alex-struk force-pushed the experiment/04-vlm-direct branch 2 times, most recently from 3451273 to b17f39f Compare June 30, 2026 22:48

alex-struk changed the base branch from develop to experiment/03-content-understanding June 30, 2026 22:48

alex-struk force-pushed the experiment/03-content-understanding branch from f592545 to 0853b36 Compare July 2, 2026 16:19

alex-struk force-pushed the experiment/04-vlm-direct branch from b17f39f to 5819a0c Compare July 2, 2026 16:19

strukalex and others added 3 commits July 2, 2026 09:37

alex-struk force-pushed the experiment/03-content-understanding branch from 0853b36 to f96096d Compare July 2, 2026 16:39

alex-struk force-pushed the experiment/04-vlm-direct branch from 5819a0c to 13beb03 Compare July 2, 2026 16:39

alex-struk marked this pull request as ready for review July 2, 2026 17:28

alex-struk requested review from NoorChasib, antsand, dbarkowsky and kmandryk as code owners July 2, 2026 17:28

kmandryk requested changes Jul 9, 2026

View reviewed changes

alex-struk closed this Jul 20, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

PR 5: E04 — VLM-direct (gpt-5.4) - #159

PR 5: E04 — VLM-direct (gpt-5.4)#159
alex-struk wants to merge 3 commits into
experiment/03-content-understandingfrom
experiment/04-vlm-direct

alex-struk commented May 12, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kmandryk Jul 9, 2026

Uh oh!

alex-struk Jul 10, 2026

Uh oh!

alex-struk commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

alex-struk commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What's added

Provider + workflow

Tooling

Results

Cross-experiment

Notes for reviewers

Testing

Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kmandryk Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

alex-struk Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

alex-struk commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

alex-struk commented May 12, 2026 •

edited

Loading