PR 3: E02 — Mistral Document AI on Azure AI Foundry by alex-struk · Pull Request #157 · bcgov/ai-adoption-document-intelligence

alex-struk · 2026-05-12T22:17:15Z

Summary

Second experiment: a Mistral Document AI provider running on Azure AI Foundry, benchmarked against the same canonical 40-sample dataset as E01. After prompt tuning, E02 converges to roughly E01-level extraction quality, validating that the suite framework is engine-agnostic.

This PR adds the provider + workflow, a single-sample test harness for fast iteration, a force-resync mode for re-running benchmarks on an aligned dataset, and the full results writeup.

Merge order: PR 3 of the experiments series. Depends on PRs 1 → 2. After upstream PRs merge, click "Update branch" here before merging.

What's added

Provider + workflow

New Mistral Document AI on Azure AI Foundry provider
apps/temporal/src/scripts/templates/experiment-02-mistral-doc-ai-azure-workflow.json — workflow template (auto-discovered by the seed pipeline)
docs-md/graph-workflows/02-mistral-doc-ai-azure-OCR.md — workflow doc

Iteration harness

apps/temporal/src/scripts/test-mistral-foundry-single.ts — fire one sample through the Mistral Foundry path for fast prompt iteration
Fix: make Mistral Foundry annotation actually run (initial version was silently no-op'ing)

Benchmark + force-resync

Force-resync mode in the harness — re-runs samples regardless of cache, used to produce comparable benchmarks across engines
Canonical 40-sample benchmark on the aligned dataset

Results

experiments/results/02-mistral-doc-ai-azure/SUMMARY.md — analysis showing E02 ≈ E01 after prompt tuning
experiments/results/02-mistral-doc-ai-azure/benchmark-run.json — full benchmark run
experiments/results/02-mistral-doc-ai-azure/iteration/ — request/response/diff captures from the iteration harness, plus field-descriptions.json and the converged prompt.md

Brief polish

E02 lessons baked into _shared-rules.md (so E03+ start from a stronger baseline)
E03 brief refined with what we learned in E02

Notes for reviewers

Workflow test placement (tracked in AI-1641). This PR adds apps/temporal/src/experiment-02-mistral-doc-ai-azure.test.ts, which sits at the temporal/src root and bundles a real Temporal-cluster end-to-end layer (via the shared apps/temporal/src/__testlib__/integration-harness.ts) that is currently CI-gated / dev-only (process.env.CI ? describe.skip, run against localhost:7233). This placement is intentional for now and consistent across all five experiments. Relocating these into a general, feature-agnostic comprehensive-workflow integration suite — and enabling it in CI — is tracked in AI-1641, not this PR.
Stacked-PR note — dev scripts relocated later. The dev tooling this PR adds under apps/temporal/src/scripts/ (iterate-mistral-extraction.ts, test-mistral-foundry-single.ts) is moved out of the compiled worker src/ into apps/temporal/scripts/ in the 08-part-2 patch PR (PR 12: E08 part 2 — extraction-engine review fixes + mock-only-paid integration harness #220), commit e4036113 (excluded from the worker build; type-checked via tsconfig.scripts.json). In the final merged state they live at apps/temporal/scripts/; reviewing this slice in isolation shows them at the old path.
Stacked-PR note — this activity is later consolidated. This PR introduces the Mistral Foundry path as a separate mistralAzureOcr.process activity. It is later merged (review item R2) into the single mistralOcr.process activity keyed by variant: "native" | "azure" — commit ab330374, in the 08-part-2 slice (PR PR 12: E08 part 2 — extraction-engine review fixes + mock-only-paid integration harness #220). In the final merged state, mistralAzureOcr.process is not a registered activity type; the E02 workflow node uses activityType: "mistralOcr.process" with "variant": "azure" (the node id stays labelled mistralAzureOcr). Reviewing this slice in isolation therefore shows the pre-consolidation state. (Note: docs-md/EXTRACTION_EXPERIMENTS_PR_REVIEW.md:52 still lists the old mistralAzureOcr.process name — stale, intentionally left.)
E02's converged extraction prompt is in experiments/results/02-mistral-doc-ai-azure/iteration/prompt.md — went through several rounds of tuning before matching E01.
The big diff is again dominated by the benchmark JSON. Code/config changes are modest.

Testing

test-mistral-foundry-single.ts exercised against the dev Azure AI Foundry deployment during prompt iteration.
Full 40-sample canonical benchmark captured in benchmark-run.json after the converged prompt.

Checklist

By submitting this pull request, I acknowledge that I have attempted to meet the following:

a self-review of my code

commented code particularly in hard-to-understand areas

corresponding changes to the documentation where required

changes tested to the best of my ability

no new errors or non-functional code

+
+  await fs.promises.writeFile(
+    path.join(ITERATION_DIR, "last-response.json"),
+    JSON.stringify(data, null, 2),


+    elapsedMs,
+    rawResponseSummary,
+  });
+  await fs.promises.writeFile(path.join(ITERATION_DIR, "last-diff.md"), md);


+    {
+      model: "mistral-document-ai-2512",
+      document: { type: "document_url", document_url: dataUrl },
+      document_annotation_format: annotationFormat,
+    },


+    "experiment-02",
+    `mistral-azure-ocr-response-${sampleId}-smoketest.json`,
+  );
+  await fs.promises.writeFile(outPath, JSON.stringify(data, null, 2));


Forks the Mistral provider for the Foundry serverless deployment (mistral-document-ai-2512 on strukalex-8338-resource), running parallel to the public-API path. Single sync HTTP call, same canonical post-processing chain, with the Foundry-specific endpoint, Bearer auth, and stricter request body (no confidence_scores_granularity). Includes the cross-engine audit follow-throughs from E02's brief: the shared Mistral mapper now populates word/line polygons from optional bbox corners, and the provider doc captures Mistral's internal preprocessing handling. Activity emits ocrResponse alongside ocrResult so benchmark_ocr_cache populates for sync providers. Live benchmark: 33/33 OCR'd via Foundry (run id 0fd7eef6, 156 s wallclock through the deployment's 10 RPM quota); annotation step is skipped on the deployment despite a valid document_annotation_format in the request (pages_processed_annotation: 0) — captured as an open follow-up in SUMMARY.md. Tests: 21/21 in experiment-02-mistral-doc-ai-azure.test.ts (static + runtime against local Temporal at localhost:7233, CI gate via process.env.CI), 9/9 unit tests for the new activity, 4/4 mapper tests covering the bbox-population path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Two missing pieces caused E02's first benchmark to look "completed but no extraction": 1. The Foundry route silently skips the annotation step unless the JSON-schema wrapper has `strict: true` (Microsoft Q&A 5767943). Without it, Foundry returns OCR markdown and `document_annotation: null` — so OCRResult.documents/keyValuePairs were always empty. Fix: emit `strict: true` from the shared field-definitions-to-mistral-annotation-format converter. Public- API path is unaffected (the public Mistral OCR API ignores the flag). 2. The workflow JSON's templateModelId default was a stale id copied from the public-Mistral template; the activity logged "template_not_found_or_empty_schema" and skipped sending document_annotation_format at all. Fix: point it at the seeded `seed-sdpr-monthly-report-template`. Live benchmark with both fixes (run id 3b4baeaf, 149 s): - pass_rate 0.273 (9/33), f1.median 0.563, f1.max 0.965 - precision.mean 0.840, recall.mean 0.454 - 33/33 cache rows now have ~2915 chars of populated annotation (was null on every sample previously). Adds scripts/test-mistral-foundry-single.ts — a one-shot Foundry smoke-test helper that hits the deployment for one sample (~14s) so schema fixes can be verified without burning a full 33-sample run. Tests: 22/22 in experiment-02-mistral-doc-ai-azure.test.ts (test for populated annotation replaces the prior "is null" assertion); 4/4 mapper tests; converter test asserts strict:true is emitted. Note: bbox population in the mapper still works as designed for any Mistral response that carries per-word bbox corners, but Mistral OCR itself doesn't return per-word bboxes (only image bboxes for embedded charts/figures), so polygons stay empty in production traffic against this engine. Documented as engine-shape, not a bug. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Adds an editable prompt-iteration kit and bakes the tuned prompts into the E02 workflow JSON's mistralAzureOcr node parameters. Changes: - Converter now accepts a `descriptions` overlay (per-field `description` strings attached to each JSON Schema property) and a `numericFieldsNullable` flag (every numeric field becomes `["number", "null"]`, letting Mistral distinguish blank cells from explicit zeros — the form's ground truth treats `""` as blank and `"0"` as the literal value, and the unconstrained schema lost that distinction). - Activity (mistralAzureOcrProcess) forwards `fieldDescriptions` and `numericFieldsNullable` from its params through to the converter. - E02 workflow JSON's mistralAzureOcr node now sets: - documentAnnotationPrompt (2.1 KB SDPR-aware instructions) - fieldDescriptions (74 fields) - numericFieldsNullable: true - experiments/results/02-mistral-doc-ai-azure/iteration/ — editable source-of-truth (prompt.md + field-descriptions.json + README) plus last-{request,response,diff} for the most recent iteration. - apps/temporal/src/scripts/iterate-mistral-extraction.ts — single- document smoke-test against Foundry (~14s per call) for fast iteration on the prompt without burning a 33-sample benchmark. Live benchmark with the tuned prompts (run id 21ce5b11, 145s for 33 samples): pass_rate 0.485 (16/33), f1.median 0.770, f1.mean 0.705, precision.mean 0.900, recall.mean 0.621, falsePositives.mean 2.30 — essentially even with E01 (pass_rate 0.515, f1.median 0.806). E02 now beats E01 on f1.mean and recall.mean; trails on pass_rate by one sample and on falsePositives.mean (Mistral is a general-purpose engine, occasionally hallucinates a value). Tests: 138 in the touched suites (converter + activity + experiment-02 + activity-registry); full temporal sweep 966 pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…sample dataset LocalDatasetSyncService now supports a force mode (env var FORCE_RESYNC_LOCAL_DATASETS=true) that wipes the dataset's blob prefix before re-uploading. The default idempotent sync skips files that already exist on cloud, so local renames or content edits never propagate. Canonical E02 benchmark run on the aligned dataset (40 samples; run id 1b97de43, 4m50s wallclock): pass_rate 0.875 (35/40), f1.median 0.943, f1.mean 0.907, precision.mean 0.975, recall.mean 0.864, matchedFields.median 66/74, falsePositives.mean 1.25. Synth-full (1-3) hit f1 0.986-0.993 confirming the alignment fix. HR0081 cluster is the floor of Mistral's handwriting capability; manual samples land at f1 0.86-0.98. SUMMARY.md retrospective covers schema-level Foundry quirks, operational gotchas, process improvements, _shared-rules.md updates and E03 implications. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Updates _shared-rules.md with the concrete patterns and gotchas E02 surfaced — production-grade prompts (per-field descriptions + global prompt + strict-mode flag) as a deliverable not a follow-up; iteration kit pattern (prompt.md + field-descriptions.json + iterate-*.ts smoke script) as canonical scaffolding; force-resync runbook entry; sync- provider raw-response output port for benchmark_ocr_cache; Foundry RPM quota retry tuning (30 × 15 s × 1.5x × 60 s cap); blank-vs-zero nullable-numerics; bbox-availability caveat; stale templateModelId warning; capture-a-fixture-before-trusting-the-preamble rule. Rewrites the E03 brief to bake those lessons in pre-emptively: capture a real CU response before writing the mapper, set up the iteration kit before any benchmark, verify the strict-mode equivalent on the first sample, use the canonical Foundry retry policy from the start, emit ocrResponse for cache persistence. No behavioral code changes — docs only. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

alex-struk · 2026-07-20T17:47:22Z

Superseded by #221 (cumulative extraction-experiments merge), now merged into develop. This branch's commits are all contained in develop; closing the review-only PR. Branch retained.

github-advanced-security AI found potential problems May 12, 2026

View reviewed changes

alex-struk force-pushed the experiment/02-mistral-doc-ai-azure branch 2 times, most recently from a62f619 to 25745f3 Compare June 30, 2026 22:47

alex-struk changed the base branch from develop to experiment/01-neural-doc-intelligence June 30, 2026 22:48

alex-struk force-pushed the experiment/01-neural-doc-intelligence branch from 7e50fcb to 6938e5b Compare July 2, 2026 16:19

alex-struk force-pushed the experiment/02-mistral-doc-ai-azure branch from 25745f3 to e77f455 Compare July 2, 2026 16:19

strukalex and others added 5 commits July 2, 2026 09:37

alex-struk force-pushed the experiment/01-neural-doc-intelligence branch from 6938e5b to 56c4671 Compare July 2, 2026 16:39

alex-struk force-pushed the experiment/02-mistral-doc-ai-azure branch from e77f455 to e37ea13 Compare July 2, 2026 16:39

alex-struk marked this pull request as ready for review July 2, 2026 16:52

alex-struk requested review from NoorChasib, antsand, dbarkowsky and kmandryk as code owners July 2, 2026 16:52

kmandryk approved these changes Jul 9, 2026

View reviewed changes

alex-struk closed this Jul 20, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

PR 3: E02 — Mistral Document AI on Azure AI Foundry - #157

PR 3: E02 — Mistral Document AI on Azure AI Foundry#157
alex-struk wants to merge 5 commits into
experiment/01-neural-doc-intelligencefrom
experiment/02-mistral-doc-ai-azure

alex-struk commented May 12, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alex-struk commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

alex-struk commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What's added

Provider + workflow

Iteration harness

Benchmark + force-resync

Results

Brief polish

Notes for reviewers

Testing

Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alex-struk commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

alex-struk commented May 12, 2026 •

edited

Loading