PR 4: E03 — Azure Content Understanding by alex-struk · Pull Request #158 · bcgov/ai-adoption-document-intelligence

alex-struk · 2026-05-12T22:17:18Z

Summary

Third experiment: Azure Content Understanding as the extraction engine, benchmarked against the same canonical 40-sample dataset. Adds the provider, a one-off trigger script to run a single experiment benchmark on demand, a setup script for the Content Understanding analyzer defaults, and the full results.

This PR also lands a polished "next experiment" startup prompt that captures everything learned in E01–E03 so E04 can begin from a strong baseline.

Merge order: PR 4 of the experiments series. Depends on PRs 1 → 2 → 3.

What's added

Provider + workflow

Azure Content Understanding provider integration
apps/temporal/src/scripts/templates/experiment-03-content-understanding-workflow.json — workflow template
docs-md/graph-workflows/03-content-understanding-OCR.md — workflow doc

Tooling

apps/temporal/src/scripts/trigger-experiment-benchmark.ts — generic trigger for a single experiment's benchmark run
experiments/results/03-content-understanding/setup-cu-defaults.sh — provisions the Content Understanding analyzer with the schema/defaults this experiment needs

Results

experiments/results/03-content-understanding/SUMMARY.md
experiments/results/03-content-understanding/benchmark-run.json — canonical 40-sample run
experiments/results/03-content-understanding/iteration/ — request/response/diff captures, field descriptions, converged prompt

Meta-process / E04 startup

experiments/results/03-content-understanding/NEXT-EXPERIMENT-PROMPT.md — meta-process retrospective + the exact prompt that bootstraps E04 (VLM-direct)
E04 prompt iterated several times: swap gpt-5.5 → gpt-5.4 (no quota ticket needed), scope to single-pass + gpt-5.4 only, drop pdf.renderToImages and add a runtime guard

Notes for reviewers

Workflow test placement (tracked in AI-1641). This PR adds apps/temporal/src/experiment-03-content-understanding.test.ts, which sits at the temporal/src root and bundles a real Temporal-cluster end-to-end layer (via the shared apps/temporal/src/__testlib__/integration-harness.ts) that is currently CI-gated / dev-only (process.env.CI ? describe.skip, run against localhost:7233). This placement is intentional for now and consistent across all five experiments. Relocating these into a general, feature-agnostic comprehensive-workflow integration suite — and enabling it in CI — is tracked in AI-1641, not this PR.
Stacked-PR note — dev scripts relocated later. The dev tooling this PR adds under apps/temporal/src/scripts/ (iterate-cu-extraction.ts, poll-experiment-run.ts, setup-cu-defaults.ts, trigger-experiment-benchmark.ts) is moved out of the compiled worker src/ into apps/temporal/scripts/ in the 08-part-2 patch PR (PR 12: E08 part 2 — extraction-engine review fixes + mock-only-paid integration harness #220), commit e4036113 (excluded from the worker build; type-checked via tsconfig.scripts.json). In the final merged state they live at apps/temporal/scripts/; reviewing this slice in isolation shows them at the old path.
The benchmark JSON is huge (~85k lines) — that's expected, it captures every per-sample response. Skim by checking SUMMARY.md instead.
The E04 prompt iterations are in this PR (not PR 5) because they were written during E03 to seed the next experiment.

Testing

Canonical 40-sample benchmark captured in benchmark-run.json.
setup-cu-defaults.sh validated against the dev Content Understanding resource.

Checklist

By submitting this pull request, I acknowledge that I have attempted to meet the following:

a self-review of my code

commented code particularly in hard-to-understand areas

corresponding changes to the documentation where required

changes tested to the best of my ability

no new errors or non-functional code

+
+  await fs.promises.writeFile(
+    path.join(ITERATION_DIR, "last-response.json"),
+    JSON.stringify(resultBody, null, 2),


+    elapsedMs,
+    rawResponseSummary,
+  });
+  await fs.promises.writeFile(path.join(ITERATION_DIR, "last-diff.md"), md);


+
+  await fs.promises.writeFile(
+    path.join(ITERATION_DIR, "last-response.json"),
+    JSON.stringify(data, null, 2),


+    elapsedMs,
+    rawResponseSummary,
+  });
+  await fs.promises.writeFile(path.join(ITERATION_DIR, "last-diff.md"), md);


+      );
+      const outFile = path.join(outDir, "benchmark-run.json");
+      fs.mkdirSync(outDir, { recursive: true });
+      fs.writeFileSync(outFile, JSON.stringify(downloadResp.data, null, 2));


+    {
+      model: "mistral-document-ai-2512",
+      document: { type: "document_url", document_url: dataUrl },
+      document_annotation_format: annotationFormat,
+    },


+    "experiment-02",
+    `mistral-azure-ocr-response-${sampleId}-smoketest.json`,
+  );
+  await fs.promises.writeFile(outPath, JSON.stringify(data, null, 2));


… 40-sample run E03 lands a new provider folder at apps/temporal/src/ocr-providers/azure- content-understanding/. Builds an analyzer JSON from the SDPR template's field_schema (vocabulary mapping: number->number/extract, selectionMark-> string/classify+enum, etc.), idempotently deploys it via PUT (with DELETE- before-PUT when the body changed), submits documents inline as data+mimeType, polls until terminal, and maps the response to the canonical OCRResult. Canonical run 1f8240fa-4abe-426d-877e-bb96ce78c509 (40 samples, 6:44 wallclock, gpt-5.2 capacity 100): pass_rate 0.95, f1.median 0.965, f1.mean 0.927, precision.mean 0.975, recall.mean 0.903, matchedFields. median 69/74 — beats E02 on every primary metric. Iteration kit got synth-full (1) to 97.3% per-field accuracy on the first prompt-tuning pass; benchmark_ocr_cache populated with the 1 81 fixture. Documents the operational lessons in SUMMARY.md retrospective: CU analyzerId charset (no -), required PATCH /defaults pre-flight, gpt-5.2 capacity bump (10->100) to clear the 10K TPM ceiling, runtimeSettings override for the 3600 s per-sample timeout, and the cuResponse -> ocrResponse rename for sync-provider cache emission. Tests: 16 static + 4 fixture-aware + 2 runtime + 7 schema-builder unit + 9 mapper unit. CI-gated runtime suite skips on GitHub Actions (no local Temporal cluster). Backend + temporal activity registry tests updated for the new activity types. Also: biome auto-fixed pre-existing 4-space indentation in apps/temporal/src/__fixtures__/experiment-02/mistral-azure-ocr-response- 1-81.json so the repo-wide lint passes; pure formatting, no content change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Add 11 engine-independent process lessons to E03 SUMMARY.md retrospective (things that wasted time on E03 because of how the experiment was sequenced or how the harness/repo behaved, not CU itself): preflight-as-script, cloud-resource setup in brief, output-port-contract test, iteration-kit copy as starter, draft SUMMARY skeleton on day 1, trust latest doc not external AI, reusable trigger/poll TS scripts, TEST_API_KEY env file location, silent post-success failures, pre-existing graph-workflow.test.ts breakage, and capacity-bumps-are-cheap. Add NEXT-EXPERIMENT-PROMPT.md — a self-contained prompt for starting E04 in a fresh session. Includes the reading order, branch instructions, prereq checklist (gpt-5/gpt-4o deployment status verified at handoff), pre-emptive lessons applied to VLM-direct specifically, and a typical hour-by-hour flow. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…ticket needed) Verified gpt-5.5 is the only gpt-5.x model in the subscription gated behind a quota uplift ticket (0K TPM in every region for both GlobalStandard and DataZoneStandard). Every sibling has 1M+ TPM available immediately — gpt-5.4 (released 2026-03-05, GA) is the closest substitute. Updated NEXT-EXPERIMENT-PROMPT.md to default to gpt-5.4 and moved the gpt-5.5 quota request to an optional follow-up the user can pursue in parallel with E04. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

User opted to scope E04 down from the brief's 3 variants x 2 models matrix to a single deliverable: variant 1 (single-pass) on gpt-5.4 only. Updated NEXT-EXPERIMENT-PROMPT.md with an explicit "SCOPE REDUCTION" header at the top so the next agent doesn't build the cot/self-consistency variants. Also: the standard https://aka.ms/oai/quotaincrease form does not list gpt-5.5 (only goes up to gpt-5.4 under "Global Standard Model"). Flagged this in the optional follow-up section — gpt-5.5 access is likely gated through a Microsoft preview/registration program, not the generic OpenAI quota form. Deferred to a future experiment. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…time guard All 40 samples in seed-local-samples-mix-private-v1 are JPEGs — nothing in the canonical benchmark exercises a PDF render. Replaced the standalone pdf.renderToImages activity with a runtime guard at the top of the VLM-direct activity that throws on fileType === "pdf" with a clear deferral message. Saves ~30-45 minutes + a new dependency (pdf2pic / pdfjs-dist + sharp) that no sample triggers. Also updated typical-flow estimate: 1.5-2 hours wallclock for the scoped-down version (single-pass + gpt-5.4 only, single workflow JSON, guarded PDF path). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

kmandryk · 2026-07-09T18:59:21Z

fails typescript compilation:
'graph' does not exist in type 'GraphWorkflowInput'
Property 'ctx' does not exist on type 'GraphWorkflowResult'

Fixed in #220: experiment-03-content-understanding.test.ts (with the other four experiment tests) was ported to develop's current GraphWorkflow API — graphWorkflow/getStatus from ./graph-workflow, GraphWorkflowConfig, ctx read via the getStatus query rather than off the result. Verified 2026-07-10 on the cumulative branch: apps/temporal npm run type-check → exit 0, and experiment-01..05 → 120/120 pass against the dev-stack Temporal (paid APIs mocked). Merges to develop via #221.

alex-struk · 2026-07-20T17:47:24Z

Superseded by #221 (cumulative extraction-experiments merge), now merged into develop. This branch's commits are all contained in develop; closing the review-only PR. Branch retained.

github-advanced-security AI found potential problems May 12, 2026

View reviewed changes

alex-struk force-pushed the experiment/03-content-understanding branch 2 times, most recently from 3612b40 to f592545 Compare June 30, 2026 22:47

alex-struk changed the base branch from develop to experiment/02-mistral-doc-ai-azure June 30, 2026 22:48

alex-struk force-pushed the experiment/02-mistral-doc-ai-azure branch from 25745f3 to e77f455 Compare July 2, 2026 16:19

alex-struk force-pushed the experiment/03-content-understanding branch from f592545 to 0853b36 Compare July 2, 2026 16:19

strukalex and others added 5 commits July 2, 2026 09:37

alex-struk force-pushed the experiment/02-mistral-doc-ai-azure branch from e77f455 to e37ea13 Compare July 2, 2026 16:39

alex-struk force-pushed the experiment/03-content-understanding branch from 0853b36 to f96096d Compare July 2, 2026 16:39

alex-struk marked this pull request as ready for review July 2, 2026 16:55

alex-struk requested review from NoorChasib, antsand, dbarkowsky and kmandryk as code owners July 2, 2026 16:55

kmandryk requested changes Jul 9, 2026

View reviewed changes

alex-struk closed this Jul 20, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

PR 4: E03 — Azure Content Understanding - #158

PR 4: E03 — Azure Content Understanding#158
alex-struk wants to merge 5 commits into
experiment/02-mistral-doc-ai-azurefrom
experiment/03-content-understanding

alex-struk commented May 12, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kmandryk Jul 9, 2026

Uh oh!

alex-struk Jul 10, 2026

Uh oh!

alex-struk commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

alex-struk commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What's added

Provider + workflow

Tooling

Results

Meta-process / E04 startup

Notes for reviewers

Testing

Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kmandryk Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

alex-struk Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

alex-struk commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

alex-struk commented May 12, 2026 •

edited

Loading