PR 2: E01 — Neural Document Intelligence workflow + canonical benchmark run by alex-struk · Pull Request #156 · bcgov/ai-adoption-document-intelligence

alex-struk · 2026-05-12T22:17:13Z

Summary

First concrete experiment on top of the suite scaffolding from PR 1: a neural Document Intelligence workflow (E01) wired end-to-end into the extraction pipeline, plus its canonical 40-sample benchmark run captured as the baseline that every later experiment compares against.

This PR wires the neural-DI provider into a Temporal workflow with a post-processing chain, fixes a long-standing dev-server reload issue, and lands the first "results" folder under experiments/results/.

Merge order: PR 2 of the experiments series. Depends on PR 1. After PR 1 merges, click "Update branch" here, then merge → repeat for downstream PRs.

What's added

Workflow + worker

apps/temporal/src/scripts/templates/experiment-01-neural-doc-intelligence-workflow.json — the workflow template (auto-discovered by the seed pipeline introduced in PR 1)
Worker registration updates in apps/temporal/src/worker.ts so the new workflow runs

Post-processing chain + tests

apps/temporal/src/scripts/experiment-01-neural-doc-intelligence.test.ts — runtime workflow tests covering the neural DI happy path
Shared-rules updates in experiments/briefs/_shared-rules.md codifying test conventions discovered while writing these tests
Tests are skipped in CI (require Azure credentials) but run locally

Dev experience

apps/temporal/package.json — npm run dev now reloads reliably (was occasionally hanging on file changes)

Results

experiments/results/01-neural-doc-intelligence/SUMMARY.md — analysis writeup
experiments/results/01-neural-doc-intelligence/benchmark-run.json — full canonical 40-sample benchmark
apps/temporal/src/scripts/experiment-01/neural-ocr-response-1-81.json — archived reference response (used by the test suite)

Brief polish

E01 brief: dropped the APIM-vs-direct DI TODO (resolved)
E02 brief: closed handoff gaps so the next experiment can pick up cleanly

Notes for reviewers

Workflow test placement (tracked in AI-1641). This PR adds apps/temporal/src/experiment-01-neural-doc-intelligence.test.ts, which sits at the temporal/src root and bundles a real Temporal-cluster end-to-end layer (via the shared apps/temporal/src/__testlib__/integration-harness.ts) that is currently CI-gated / dev-only (process.env.CI ? describe.skip, run against localhost:7233). This placement is intentional for now and consistent across all five experiments. Relocating these into a general, feature-agnostic comprehensive-workflow integration suite — and enabling it in CI — is tracked in AI-1641, not this PR.
The big diff is dominated by the benchmark JSON (recorded run output). The actual code change is small: workflow JSON + worker wiring + tests + a handful of seed/script tweaks.
Tests are skipped in CI by design — they need live Azure DI access.

Testing

experiment-01-neural-doc-intelligence.test.ts runs locally against the dev environment.
Full 40-sample canonical benchmark captured in benchmark-run.json.

Checklist

By submitting this pull request, I acknowledge that I have attempted to meet the following:

a self-review of my code

commented code particularly in hard-to-understand areas

corresponding changes to the documentation where required

changes tested to the best of my ability

no new errors or non-functional code

Drops in `docs-md/graph-workflows/templates/experiment-01-neural-doc-intelligence-workflow.json`, auto-discovered by `seedExperimentWorkflows()` and benched against the user's `seed-local-samples-mix-private-v1` dataset (33 samples) using the trained neural model `sdpr_synth_test`. Chain: file.prepare → azureOcr.submit/poll/extract → ocr.cleanup → ocr.normalizeFields → ocr.characterConfusion → ocr.checkConfidence → reviewSwitch → (humanReview)? → ocr.storeResults. Spellcheck dropped per the brief; LLM enrichment and cross-field validation out of scope. Real-API benchmark run id 2295feed-1c99-493e-ae20-546499b5d685: pass_rate 0.515, F1 median 0.806, precision 0.90, recall 0.59 over 33 samples in ~83s. Workflow chain executed end-to-end on every sample. Tests: 17 static + structural assertions on the template and recorded neural OCR fixture (apps/temporal/src/experiment-01-neural-doc-intelligence.test.ts). TestWorkflowEnvironment runtime tests skipped — same temporal.download TLS blocker as the existing graph-workflow.test.ts; the live benchmark is the runtime integration coverage. Parent-shared infra fixes (user-authorized to keep seed-driven and manual-create dataset paths consistent): * local-dataset-sync.service.ts: store relative `dataset-manifest.json` instead of full blob key, matching dataset.service.ts createVersion. * seed.ts seedLocalDatasets: same relative manifestPath. * seed.ts seedExperimentWorkflows: switch evaluatorType from the unregistered `field-accuracy` to `schema-aware`, with a fuzzy default rule and 0.8 pass threshold. Outstanding parent gap (NOT fixed here, flagged in SUMMARY): scripts/run-experiment-benchmarks.sh sends `tags` as an array but CreateRunDto requires an object — one-line fix in the script. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…mark export Two unrelated fixes bundled because both surfaced while finishing E01: 1. ts-node-dev hot reload was stopping the worker without respawning. Root causes: a. The Rust-backed `NativeConnection` was never explicitly closed, so its tokio runtime kept the Node event loop alive after `worker.run()` resolved. ts-node-dev's `--respawn` waits for the child to exit before starting a new one, so the new instance never came up. Fixed by `await connection.close()` and an explicit `process.exit(0)` after the workers stop. b. `--watch src` told chokidar to watch every file under src/ including JSON fixtures (e.g. the 290 KB neural-OCR poll fixture). Touching a fixture forced a worker drain. ts-node-dev's `--ignore-watch` only filters require-loaded files, not explicitly watched paths, so the only practical fix is to narrow the watch glob — switched to `'src/**/*.ts'` so JSON fixture edits no longer trigger a reload. Verified live: - touch src/__fixtures__/.../neural-ocr-response-1-81.json → no restart - touch src/logger.ts → drain → "Worker stopped" → ~600ms later "Worker initializing" → RUNNING 2. Archive the full benchmark export for run `2295feed-1c99-493e-ae20-546499b5d685` (the canonical E01 result) at `experiments/results/01-neural-doc-intelligence/benchmark-run.json`. Includes per-sample ground truth + prediction + evaluation details + per-field results + error-detection analysis. Pulled from `GET /api/benchmark/projects/.../runs/{runId}/download`. Set up so each experiment in the stack can drop its `benchmark-run.json` alongside `SUMMARY.md` and the cross-experiment comparison can read them locally without re-querying the backend. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Three things, all driven by E01 retrospective: 1. Workflow-level runtime tests against the local dev-stack Temporal (`localhost:7233`). Each test loads the actual JSON template, runs the real `graphWorkflow` workflow, and replays the recorded neural-OCR fixture through mocked activities. Covers both `reviewSwitch` branches — high-confidence skips humanReview; low-confidence routes through humanReview, gets a `humanApproval` signal, then completes via `storeResults`. Both tests patch `pollOcrResults` interval/initialDelay to `1ms` so they finish in ~2s each. 19/19 pass in ~7s. Sidesteps `TestWorkflowEnvironment.createTimeSkipping()` which TLS-fails on the binary download; documented as the canonical pattern. 2. `scripts/run-experiment-benchmarks.sh`: send `tags` as an object (not an array) per the `CreateRunDto` `@IsObject()` validation, and pass `persistOcrCache: true` by default so each run captures engine responses for the test fixture. 3. `experiments/briefs/_shared-rules.md` overhaul, incorporating E01 retrospective: * **No upstream backporting** — fix shared infra bugs in your experiment branch and let the chained stack carry them upstream. Replaces the old "stop and raise it back" guidance. * **Authentication** subsection — how to get and use TEST_API_KEY, the relevant API endpoints, that the seed-time key is dev-only. * **Common bugs runbook** — the five footguns E01 hit (script tags shape, manifestPath value, evaluator name, confidence-threshold recalibration, hot-reload watcher). * **Dev loop** rewritten — benchmark-with-persistOcrCache IS the real-API run, fixture-capture is now an explicit step, full benchmark export is a deliverable at `experiments/results/<slug>/benchmark-run.json`. * **Test layout** — pin per-experiment tests at `apps/temporal/src/experiment-<slug>.test.ts` with fixtures at `apps/temporal/src/__fixtures__/experiment-<slug>/`. * **`metadata.targetLocalDataset` is now required**, not just an override, so benchmark-target choice is deterministic. * **Done criteria** — `benchmark-run.json` saved as a deliverable; runtime tests required, not just static. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

CI's temporal-qa.yml runs `npm test` without a Temporal sidecar, so the runtime layer fails to connect to localhost:7233. Gate the runtime `describe` block on `process.env.CI` (set automatically by GitHub Actions). Static suite continues to run on every CI build. Verified: - local: 19 passed - CI=true: 17 passed, 2 skipped Pattern documented in `_shared-rules.md` so subsequent experiments use the same gate. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

The TODO asked E01 to verify Azure DI activities don't have APIM-specific path manipulation that would break under direct access. E01 ran the chain end-to-end against direct Azure DI without issue, so the concern is empirically resolved — no path-prefix surgery was needed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Updates from the E01 retrospective + open audit items: - Task 2: clarify it's a single HTTP call (no two-call orchestration in our code). Mistral's API handles the OCR-then-annotation chain server-side via `document_annotation_format` + `document_annotation_prompt` in the body, mirroring how `mistral-ocr-process.ts` already works. (Was an ambiguity that surfaced in the handoff review.) - Task 3: explicit list of all THREE activity registries + activities.ts export, so the next session doesn't miss one and ship a worker that fails save-time validation. - Task 5: require `metadata.targetLocalDataset = "samples-mix-private"` per the updated _shared-rules.md (was "override if needed", now required for deterministic dataset binding). - New Task 6: capture the cross-engine audit Item 5 ⚠️ (Mistral mapper sets `polygon: []`). Populate per-word polygons from Mistral's bbox data so E05's VLM+OCR hybrid has spatial info. Update existing mapper test. Benefits both public-API and Foundry paths since they share the mapper. - New Task 7: capture the cross-engine audit Item 9 ⚠️ (Mistral internal preprocessing not documented). Resolve from Mistral docs and capture in the per-engine OCR doc; same doc covers Foundry-vs-public diffs. - Task 8: align with the post-E01 trigger (`./scripts/run-experiment-benchmarks.sh 02`, which now sends `tags` as object and `persistOcrCache: true` by default). Tag is now `{"experiment":"02-mistral-doc-ai-azure"}` (full slug) — supersedes the old `experiment-02-mistral-azure` short form. - New Tasks 9, 10: explicit deliverables — `benchmark-run.json` saved from the download endpoint; one OCR fixture captured to `__fixtures__/experiment-02/` for the test replay. - Task 11: replaces the old "Mock-based tests" line with the canonical two-layer pattern (static + runtime) and the `process.env.CI` gate for the runtime suite, pointing at `experiment-01-neural-doc-intelligence.test.ts` as the reference implementation. - Task 12: SUMMARY.md fields explicit — deployment id, run id, headline metrics, comparison row vs public-API, confidence-distribution observations, infra-fix log, bbox-fix verification. - Cross-engine audit follow-through section: list both ⚠️ items (5 and 9) explicitly with the tasks that address them. - Header pointer to `_shared-rules.md` so the canonical patterns and "fix forward" rule are surfaced up front. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

dbarkowsky

apps/temporal/src/fixtures/experiment-01/neural-ocr-response-1-81.json
apps/temporal/src/experiment-01-neural-doc-intelligence.test.ts

Do we want this type of content saved in the /src folder? It's not actually source code for the application.

alex-struk · 2026-07-07T16:35:43Z

Workflow test placement (tracked in AI-1641). This PR adds apps/temporal/src/experiment-01-neural-doc-intelligence.test.ts, which sits at the temporal/src root and bundles a real Temporal-cluster end-to-end layer (via the shared apps/temporal/src/__testlib__/integration-harness.ts) that is currently CI-gated / dev-only (process.env.CI ? describe.skip, run against localhost:7233). This placement is intentional for now and consistent across all five experiments. Relocating these into a general, feature-agnostic comprehensive-workflow integration suite — and enabling it in CI — is tracked in AI-1641, not this PR.

Thanks @dbarkowsky, for now this is intentional. I have a separate ticket to get these new types of tests to run in CI, so these files will be moved into appropriate location then:

Workflow test placement (tracked in AI-1641). This PR adds apps/temporal/src/experiment-01-neural-doc-intelligence.test.ts, which sits at the temporal/src root and bundles a real Temporal-cluster end-to-end layer (via the shared apps/temporal/src/testlib/integration-harness.ts) that is currently CI-gated / dev-only (process.env.CI ? describe.skip, run against localhost:7233). This placement is intentional for now and consistent across all five experiments. Relocating these into a general, feature-agnostic comprehensive-workflow integration suite — and enabling it in CI — is tracked in AI-1641, not this PR.

kmandryk · 2026-07-09T18:49:36Z

This test fails- it uses an outdated workflow input pattern.

Fixed in #220: experiment-01-neural-doc-intelligence.test.ts (with the other four experiment tests) was ported to develop's current GraphWorkflow API — graphWorkflow/getStatus from ./graph-workflow, GraphWorkflowConfig, ctx read via the getStatus query rather than off the result. Verified 2026-07-10 on the cumulative branch: apps/temporal npm run type-check → exit 0, and experiment-01..05 → 120/120 pass against the dev-stack Temporal (paid APIs mocked). Merges to develop via #221.

alex-struk · 2026-07-20T17:47:20Z

Superseded by #221 (cumulative extraction-experiments merge), now merged into develop. This branch's commits are all contained in develop; closing the review-only PR. Branch retained.

github-advanced-security AI found potential problems May 12, 2026

View reviewed changes

Comment thread apps/backend-services/src/seed/local-datasets.ts Dismissed

Comment thread apps/backend-services/src/seed/local-datasets.ts Dismissed

alex-struk force-pushed the experiment/01-neural-doc-intelligence branch 2 times, most recently from 6e1d78b to 7e50fcb Compare June 30, 2026 22:47

alex-struk changed the base branch from develop to feature/extraction-experiments June 30, 2026 22:48

alex-struk mentioned this pull request Jun 30, 2026

Extraction experiments — cumulative merge (E01–E08 + improvements + review fixes + harness) #221

Merged

alex-struk force-pushed the experiment/01-neural-doc-intelligence branch from 7e50fcb to 6938e5b Compare July 2, 2026 16:19

alex-struk force-pushed the feature/extraction-experiments branch from 13f7187 to 1054cf5 Compare July 2, 2026 16:19

strukalex and others added 6 commits July 2, 2026 09:37

alex-struk force-pushed the experiment/01-neural-doc-intelligence branch from 6938e5b to 56c4671 Compare July 2, 2026 16:39

alex-struk force-pushed the feature/extraction-experiments branch from 1054cf5 to 0fa6d97 Compare July 2, 2026 16:39

alex-struk marked this pull request as ready for review July 2, 2026 16:46

alex-struk requested review from NoorChasib, antsand, dbarkowsky and kmandryk as code owners July 2, 2026 16:46

dbarkowsky reviewed Jul 7, 2026

View reviewed changes

kmandryk requested changes Jul 9, 2026

View reviewed changes

alex-struk closed this Jul 20, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

PR 2: E01 — Neural Document Intelligence workflow + canonical benchmark run - #156

PR 2: E01 — Neural Document Intelligence workflow + canonical benchmark run#156
alex-struk wants to merge 6 commits into
feature/extraction-experimentsfrom
experiment/01-neural-doc-intelligence

alex-struk commented May 12, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

dbarkowsky left a comment

Uh oh!

alex-struk commented Jul 7, 2026 •

edited

Loading

Uh oh!

kmandryk Jul 9, 2026

Uh oh!

alex-struk Jul 10, 2026

Uh oh!

alex-struk commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

alex-struk commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What's added

Workflow + worker

Post-processing chain + tests

Dev experience

Results

Brief polish

Notes for reviewers

Testing

Checklist

Uh oh!

Uh oh!

Uh oh!

dbarkowsky left a comment

Choose a reason for hiding this comment

Uh oh!

alex-struk commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kmandryk Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

alex-struk Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

alex-struk commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

alex-struk commented May 12, 2026 •

edited

Loading

alex-struk commented Jul 7, 2026 •

edited

Loading