E09: SDPR HITL inline editor + timing-experiment harness (stacked on E08 / #165) - #184
Draft
alex-struk wants to merge 251 commits into
Draft
E09: SDPR HITL inline editor + timing-experiment harness (stacked on E08 / #165)#184alex-struk wants to merge 251 commits into
alex-struk wants to merge 251 commits into
Conversation
alex-struk
force-pushed
the
experiment/08-vlm-ocr-hybrid-gpt-5.2
branch
from
June 27, 2026 07:35
fe953cf to
cd87c3c
Compare
alex-struk
force-pushed
the
experiment/09-sdpr-hitl-committed
branch
from
June 27, 2026 07:35
d4ed00c to
985a849
Compare
Sets up the parent branch (feature/extraction-experiments) for a 5-engine extraction comparison: neural DI + post-processing, Mistral Doc AI on Azure, Azure Content Understanding, VLM-direct, and VLM+OCR hybrid. Defines parent-branch scaffolding (brief library, deployment-selection plumbing, local-folder dataset seed convention) and per-experiment branch deliverables. Stacks on PR #134's neural training capability. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ting-engine audit, dev-loop, future candidates Updates the extraction-experiments design spec per user feedback: - Replaces the AI-feedback 12-item checklist with one derived from this codebase (file paths, types, conventions actually present here). - Documents the dev loop as "implement → run on real document → run benchmark via API → write mock tests when stable", not an accuracy-threshold gate. - Adds existing-engine audit (Azure DI + Mistral) against the new checklist to the provider-architecture doc on the parent branch. - Adds vendor-doc citations confirming Mistral and CU use OCR-first → LLM annotation, which E05 recreates explicitly. - Adds "Candidates for future rounds" section listing trends not in the core 5 experiments (confidence-based routing, per-field calibration, multi-engine voting, agentic correction, open-source VLMs, etc.) so they can be re-pitched after E01–E05 land. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… scaffolding Removes: - Duplicate "Cross-engine audit" section (already in deliverable #3) - "Expected gaps to surface" speculation - "Open questions / assumptions" section (was for pre-approval review) - Goals section (overlaps Context + deliverables) - Brief-list file-content descriptions - Experiment-branch deliverables list (overlaps the integration checklist) - "Why deferred" column from future-candidates table Compresses: - Three Azure resource tables into one - Per-experiment sections to elevator-pitch length - Sequence of work (drops "commit spec" step which is done) - TODO-callout section moved into the E01 brief description Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
gpt-5.5 GlobalStandard quota is 0 in this subscription. gpt-5 (vanilla, 2025-08-07) has 1000 TPM quota in westus, on the same account as gpt-4o, so deployment succeeded immediately and the plumbing stays simple (single endpoint, deployment-name as the only switch). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ure audit Parent-branch deliverables for the extraction-experiments suite: - experiments/briefs/_shared-rules.md — codebase-derived 12-item engine-integration checklist and dev loop discipline (no accuracy threshold gating; "implement → run on real document → benchmark via API → mock-replay tests once stable") - experiments/briefs/01..05-*.md — per-experiment briefs for neural DI, Mistral on Azure, Content Understanding, VLM-direct, VLM+OCR hybrid - docs-md/EXTRACTION_EXPERIMENTS.md — hub doc with per-experiment status table and engine-integration checklist scaffolds - docs-md/EXTRACTION_PROVIDER_ARCHITECTURE.md — current state + decision to defer formal OcrProvider interface + audit of existing Azure DI and Mistral providers against the 12-item checklist (gaps captured as scoped TODOs in the affected experiment briefs) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…convention
- Append new env vars to apps/{backend-services,temporal}/.env.sample for the
experiments not yet wired (E02 Mistral on Foundry, E03 Content Understanding,
E04/E05 selectable Azure OpenAI deployments). User redirects values to their
personal Azure via the existing override file at
~/.config/bcgov-di/{backend-services,temporal}.env (Claude does not read it).
- AZURE_OPENAI_DEPLOYMENT remains the default fallback for backward compatibility;
AZURE_OPENAI_DEPLOYMENTS is the new comma-separated allow-list for per-node
selection.
- Add data/datasets/ as a tracked path with .gitkeep; gitignore
data/datasets/*/private/** so users can drop personal/test docs without
committing them. Convention documented in the design spec.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…-folder dataset seed
Two pieces of parent-branch scaffolding that all 5 experiments depend on:
1. Per-workflow-node Azure OpenAI deployment selection
- apps/temporal/src/activities/enrich-results.ts: read params.azureOpenAiDeployment
with fall-through to AZURE_OPENAI_DEPLOYMENT env var.
- apps/backend-services/src/azure/azure-openai.controller.ts: GET /api/azure-openai/deployments
parses AZURE_OPENAI_DEPLOYMENTS (comma-separated). Mirrors GET /api/models.
Falls back to AZURE_OPENAI_DEPLOYMENT singleton, then empty array.
- DTO + Swagger decorators, registered in AzureModule.
- Tests: 7 controller cases + 2 enrich-results override cases.
2. Local-folder dataset seed extension
- apps/backend-services/src/seed/local-datasets.ts: pure parser scanning
data/datasets/<name>/{public,private}/manifest.json. Validates JSON +
samples array; warns and skips on errors.
- apps/shared/prisma/seed.ts: new seedLocalDatasets() called after the
existing benchmark-data seed. Idempotent upsert of Dataset + DatasetVersion
rows. Skips quietly when data/datasets/ is empty.
- Test: 10 cases covering parser edge cases + id-helper coverage.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extends the local-folder dataset seed to handle the natural drop pattern of <name>.<image-ext> + <name>.json pairs without a manifest.json. When no manifest exists, scan for pairs and auto-derive one, writing it to disk with metadata.autoGenerated = true so subsequent re-seeds refresh it. User-curated manifests (no auto-gen marker) are never touched. Validated end-to-end on a 33-sample drop (samples-mix/private). - New deriveManifestFromFlatPairs(folderPath, folder, log) — pure helper. - Accepts: jpg/jpeg/png/pdf/tif/tiff/webp/bmp/gif as document extensions. - Warns about unpaired documents (missing JSON) and unpaired ground-truth (missing document). - 7 new tests covering flat-pair, regen-on-rerun, user-curated takes precedence, unpaired warnings, empty folders, mixed-case extensions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Captures the quota-request flow + the deploy command + a fallback (run E04 with gpt-4o and gpt-5 first if approval is delayed). Also notes that gpt-5.5 will live in eastus2 (different endpoint than the existing westus account), so the workflow node may need azureOpenAiEndpoint/Key params threaded — current parent-branch plumbing only threads deployment. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ev only)
The seed creates Dataset/DatasetVersion DB records, but the existing benchmark
infrastructure (Sample Preview, BenchmarkRun execution) pulls files from blob
storage. Without uploading the actual files, dropped local datasets show up
in the DB but not in the UI and aren't usable for benchmarks.
Adds LocalDatasetSyncService that runs at backend startup (NestJS
onApplicationBootstrap) and uploads each local dataset's files to blob storage
following the existing convention used by dataset.service.ts uploadFiles:
groups/{groupId}/benchmark/datasets/{datasetId}/{versionId}/inputs/{file}
groups/{groupId}/benchmark/datasets/{datasetId}/{versionId}/ground-truth/{file}
groups/{groupId}/benchmark/datasets/{datasetId}/{versionId}/dataset-manifest.json
The blob dataset-manifest.json follows the schemaVersion=1.0 shape the existing
runtime expects (with `inputs/` / `ground-truth/` path prefixes baked in).
- Idempotent: skips files already in blob storage; always rewrites the manifest
(cheap, lets it pick up local manifest edits between runs).
- Updates DatasetVersion.storagePrefix and manifestPath to the blob paths so
the runtime finds the synced files.
- Gated to NODE_ENV !== production; can be turned off via
SYNC_LOCAL_DATASETS_ON_START=false.
- Pure helper syncLocalDatasetToBlobStorage exported for unit testing — 4 cases
covering first-sync, idempotent re-sync, partial-resync after files added,
and blob-key convention check.
Also updates seedLocalDatasets to write the eventual blob-convention paths
into the DB records (storagePath, storagePrefix, manifestPath) so the records
are correct from the moment the seed completes, even before the sync service
runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…tself PrismaService is a wrapper that exposes the actual PrismaClient via `prismaService.prisma` (see apps/backend-services/src/database/prisma.service.ts). The sync service was casting the wrapper directly to DatasetVersionUpdater, so this.prisma.datasetVersion was undefined and the storagePrefix update threw at runtime. The unit tests passed because they bypass NestJS DI and call the pure syncLocalDatasetToBlobStorage helper with a hand-rolled stub already shaped like a PrismaClient. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… land
Closes two readiness gaps so the user has zero hand-holding once experiment
implementation completes:
1. Seed scaffolding for the benchmark suite, on the parent branch:
- Adds BenchmarkProject id 'seed-experiments-project' ("Extraction
Experiments") that all 5 experiment branches add their BenchmarkDefinition
rows to.
- Adds a Split per local dataset version with id pattern
'seed-local-{folder}-{visibility}-split', type=test, populated with all
sample IDs from the (auto-generated or curated) manifest. Verified: 33
sample IDs land on the split for the user's samples-mix/private drop.
2. Codifies the per-experiment seeding requirement in _shared-rules.md (item 11
Benchmark integration) — each experiment must seed: a WorkflowLineage +
WorkflowVersion, plus a BenchmarkDefinition with the deterministic id
'seed-experiment-{slug}-definition' referencing the parent-seeded project
and split. Once landed, the definition is invokable via the standard API
endpoint POST /api/benchmark/projects/.../definitions/.../runs.
3. Adds scripts/run-experiment-benchmarks.sh — triggers all 5 experiment runs
(or a subset by leading number) using TEST_API_KEY from the override file.
Tags runs with experiment-{slug} for cross-experiment comparison.
4. Updates docs-md/EXTRACTION_EXPERIMENTS.md with a "Run all benchmarks"
section and the on-demand curl pattern.
After E01-E05 land and seed runs:
./scripts/run-experiment-benchmarks.sh
…fires all 5 runs against the user's seeded dataset.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ll dataset
Two refinements after auditing the existing workflow conventions:
1. Drop the Split seeding for local datasets. BenchmarkDefinition.splitId is
optional in the Prisma schema, and benchmarks should run on the full
user-dropped dataset (33 samples), not a subset. Each experiment's
BenchmarkDefinition will reference the full DatasetVersion directly via
datasetVersionId.
2. Align _shared-rules.md with the existing OCR-provider conventions in the
codebase, which the prior version partially missed:
- Add "Required reading" pointing at ADDING_OCR_PROVIDERS.md,
ADDING_GRAPH_NODES_AND_ACTIVITIES.md, DAG_WORKFLOW_ENGINE.md,
GRAPH_TYPES.md, and the templates folder. These are authoritative.
- Update the "files you may edit" list to call out all THREE activity
registries (apps/temporal/src/activity-registry.ts,
apps/temporal/src/activity-types.ts, AND
apps/backend-services/src/workflow/activity-registry.ts) — missing any
one breaks workflow validation or worker resolution.
- Update item 2 (activity registration): three registries, naming
conventions <provider>Ocr.process or <provider>Ocr.{submit,poll,extract}.
- Update item 8 (workflow graph): templates live at
docs-md/graph-workflows/templates/<slug>-workflow.json, validated by
graph-schema-validator.test.ts. Standard node sequence per the docs.
Provider-specific options via initialCtx, not new global env vars.
- Update item 11 (benchmark integration): no Split — reference full
DatasetVersion directly. Pointer to seedLineageVersion pattern.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Seed.ts now scans docs-md/graph-workflows/templates/ for files matching
'experiment-*-workflow.json'. For each file found, idempotently creates:
- WorkflowLineage seed-experiment-{slug}-workflow
- WorkflowVersion wv_seed-experiment-{slug}-workflow (full JSON config)
- BenchmarkDefinition seed-experiment-{slug}-definition in
seed-experiments-project, targeting the (single) local DatasetVersion
Slug derived from filename: experiment-04-vlm-direct-workflow.json → "04-vlm-direct".
If multiple local datasets exist, workflows can target a specific one via
metadata.targetLocalDataset = "{folder}-{visibility}". Defaults to first.
Verified end-to-end with a smoke-test template (experiment-99-smoke-workflow.json):
seed picked it up, created lineage/version/definition targeting
seed-local-samples-mix-private-v1. Removed file after — clean state.
Implication for experiment AIs: drop a workflow JSON, get the full benchmark
plumbing for free. No edits to seed.ts per experiment.
Updates the briefs accordingly:
- _shared-rules.md item 11: explains the auto-discovery contract.
- E01: copy standard-ocr-workflow-with-corrections.json (already wires
cleanup/normalize/character-confusion/spellcheck), change modelId default
to neural model id, optionally add ocr.enrich.
- E02: copy mistral-standard-ocr-workflow.json, swap mistralOcr.process →
mistralAzureOcr.process.
- E03: base on standard-ocr-workflow.json (async pattern: submit/pollUntil/
extract), replace Azure DI nodes with azureContentUnderstanding.deployAnalyzer
+ azureContentUnderstanding.analyze.
- E04: base on mistral-standard-ocr-workflow.json (sync), insert
pdf.renderToImages + vlmDirect.extract.
- E05: base on standard-ocr-workflow.json, replace extraction with
pdf.renderToImages + azureOcr.readPlain + vlmOcrHybrid.extract.
Each variant in E04/E05 can have its own JSON file (one per variant) — each
gets its own BenchmarkDefinition automatically.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… report
Three changes so the user has a populated final-tip branch ready to run all
benchmarks and produce a cross-experiment report:
1. Chained branches (was hub-and-spoke).
- E01 branches from feature/extraction-experiments
- E02 branches from E01
- E03 branches from E02
- E04 branches from E03
- E05 branches from E04
By the time E05 lands, its tip contains every workflow JSON, every
provider folder, and every seed change from E01-E05. Updates the spec
diagram, hub doc, _shared-rules.md, and each per-experiment brief's
"Branch" line to reflect this. Also enumerates the exact `git checkout -b`
commands per experiment in the hub doc.
2. E01 brief simplification (per user).
- modelId default = "test" (the existing trained neural model — no
training step needed in this experiment).
- Drops spellcheck node (`ocr.spellcheck`) from the workflow.
- Drops LLM enrichment (`ocr.enrich`) — out of scope for E01.
- Drops cross-field validation (`document-validate-fields`) — out of scope.
- Final node sequence: file.prepare → azureOcr.submit/poll/extract →
ocr.cleanup → ocr.normalizeFields → ocr.characterConfusion →
ocr.checkConfidence → reviewSwitch → humanReview/storeResults.
- Adds an explicit "Out of scope" section so future readers don't add
these activities back unintentionally.
3. scripts/compare-experiment-benchmarks.sh.
- Lists runs in seed-experiments-project, filters by tag pattern
experiment-*, picks the most recent COMPLETED run per tag.
- Downloads each via the existing GET /runs/:runId/download endpoint to
/tmp/extraction-experiments-YYYYMMDD-HHMMSS/<slug>/run.json.
- Writes COMPARISON.md — a markdown table with status, field/character/
word accuracy, duration, cost, run id per experiment.
- Intentionally simple ("start with something simple" per user); easy to
extend with per-field-class metrics, P95 latency, etc.
- Hub doc updated with the full two-command flow:
./scripts/run-experiment-benchmarks.sh
(wait for runs)
./scripts/compare-experiment-benchmarks.sh
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three references in the brief updated. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ult) The existing standard-ocr-workflow-with-corrections.json template already sets ctx.modelId.defaultValue to sdpr_synth_test, so the experiment AI just copies the template and leaves modelId untouched. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drops in `docs-md/graph-workflows/templates/experiment-01-neural-doc-intelligence-workflow.json`,
auto-discovered by `seedExperimentWorkflows()` and benched against the user's
`seed-local-samples-mix-private-v1` dataset (33 samples) using the trained
neural model `sdpr_synth_test`.
Chain: file.prepare → azureOcr.submit/poll/extract → ocr.cleanup →
ocr.normalizeFields → ocr.characterConfusion → ocr.checkConfidence →
reviewSwitch → (humanReview)? → ocr.storeResults. Spellcheck dropped per the
brief; LLM enrichment and cross-field validation out of scope.
Real-API benchmark run id 2295feed-1c99-493e-ae20-546499b5d685: pass_rate
0.515, F1 median 0.806, precision 0.90, recall 0.59 over 33 samples in ~83s.
Workflow chain executed end-to-end on every sample.
Tests: 17 static + structural assertions on the template and recorded neural
OCR fixture (apps/temporal/src/experiment-01-neural-doc-intelligence.test.ts).
TestWorkflowEnvironment runtime tests skipped — same temporal.download TLS
blocker as the existing graph-workflow.test.ts; the live benchmark is the
runtime integration coverage.
Parent-shared infra fixes (user-authorized to keep seed-driven and
manual-create dataset paths consistent):
* local-dataset-sync.service.ts: store relative `dataset-manifest.json`
instead of full blob key, matching dataset.service.ts createVersion.
* seed.ts seedLocalDatasets: same relative manifestPath.
* seed.ts seedExperimentWorkflows: switch evaluatorType from the
unregistered `field-accuracy` to `schema-aware`, with a fuzzy default
rule and 0.8 pass threshold.
Outstanding parent gap (NOT fixed here, flagged in SUMMARY):
scripts/run-experiment-benchmarks.sh sends `tags` as an array but
CreateRunDto requires an object — one-line fix in the script.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…mark export
Two unrelated fixes bundled because both surfaced while finishing E01:
1. ts-node-dev hot reload was stopping the worker without respawning.
Root causes:
a. The Rust-backed `NativeConnection` was never explicitly closed,
so its tokio runtime kept the Node event loop alive after
`worker.run()` resolved. ts-node-dev's `--respawn` waits for the
child to exit before starting a new one, so the new instance
never came up. Fixed by `await connection.close()` and an
explicit `process.exit(0)` after the workers stop.
b. `--watch src` told chokidar to watch every file under src/
including JSON fixtures (e.g. the 290 KB neural-OCR poll fixture).
Touching a fixture forced a worker drain. ts-node-dev's
`--ignore-watch` only filters require-loaded files, not
explicitly watched paths, so the only practical fix is to
narrow the watch glob — switched to `'src/**/*.ts'` so JSON
fixture edits no longer trigger a reload.
Verified live:
- touch src/__fixtures__/.../neural-ocr-response-1-81.json → no restart
- touch src/logger.ts → drain → "Worker stopped" → ~600ms later
"Worker initializing" → RUNNING
2. Archive the full benchmark export for run
`2295feed-1c99-493e-ae20-546499b5d685` (the canonical E01 result) at
`experiments/results/01-neural-doc-intelligence/benchmark-run.json`.
Includes per-sample ground truth + prediction + evaluation details
+ per-field results + error-detection analysis. Pulled from
`GET /api/benchmark/projects/.../runs/{runId}/download`. Set up so
each experiment in the stack can drop its `benchmark-run.json`
alongside `SUMMARY.md` and the cross-experiment comparison can read
them locally without re-querying the backend.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three things, all driven by E01 retrospective:
1. Workflow-level runtime tests against the local dev-stack Temporal
(`localhost:7233`). Each test loads the actual JSON template, runs the
real `graphWorkflow` workflow, and replays the recorded neural-OCR
fixture through mocked activities. Covers both `reviewSwitch` branches
— high-confidence skips humanReview; low-confidence routes through
humanReview, gets a `humanApproval` signal, then completes via
`storeResults`. Both tests patch `pollOcrResults` interval/initialDelay
to `1ms` so they finish in ~2s each. 19/19 pass in ~7s. Sidesteps
`TestWorkflowEnvironment.createTimeSkipping()` which TLS-fails on the
binary download; documented as the canonical pattern.
2. `scripts/run-experiment-benchmarks.sh`: send `tags` as an object (not
an array) per the `CreateRunDto` `@IsObject()` validation, and pass
`persistOcrCache: true` by default so each run captures engine
responses for the test fixture.
3. `experiments/briefs/_shared-rules.md` overhaul, incorporating E01
retrospective:
* **No upstream backporting** — fix shared infra bugs in your
experiment branch and let the chained stack carry them upstream.
Replaces the old "stop and raise it back" guidance.
* **Authentication** subsection — how to get and use TEST_API_KEY,
the relevant API endpoints, that the seed-time key is dev-only.
* **Common bugs runbook** — the five footguns E01 hit (script tags
shape, manifestPath value, evaluator name, confidence-threshold
recalibration, hot-reload watcher).
* **Dev loop** rewritten — benchmark-with-persistOcrCache IS the
real-API run, fixture-capture is now an explicit step, full
benchmark export is a deliverable at
`experiments/results/<slug>/benchmark-run.json`.
* **Test layout** — pin per-experiment tests at
`apps/temporal/src/experiment-<slug>.test.ts` with fixtures at
`apps/temporal/src/__fixtures__/experiment-<slug>/`.
* **`metadata.targetLocalDataset` is now required**, not just an
override, so benchmark-target choice is deterministic.
* **Done criteria** — `benchmark-run.json` saved as a deliverable;
runtime tests required, not just static.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CI's temporal-qa.yml runs `npm test` without a Temporal sidecar, so the runtime layer fails to connect to localhost:7233. Gate the runtime `describe` block on `process.env.CI` (set automatically by GitHub Actions). Static suite continues to run on every CI build. Verified: - local: 19 passed - CI=true: 17 passed, 2 skipped Pattern documented in `_shared-rules.md` so subsequent experiments use the same gate. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The TODO asked E01 to verify Azure DI activities don't have APIM-specific path manipulation that would break under direct access. E01 ran the chain end-to-end against direct Azure DI without issue, so the concern is empirically resolved — no path-prefix surgery was needed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Updates from the E01 retrospective + open audit items: - Task 2: clarify it's a single HTTP call (no two-call orchestration in our code). Mistral's API handles the OCR-then-annotation chain server-side via `document_annotation_format` + `document_annotation_prompt` in the body, mirroring how `mistral-ocr-process.ts` already works. (Was an ambiguity that surfaced in the handoff review.) - Task 3: explicit list of all THREE activity registries + activities.ts export, so the next session doesn't miss one and ship a worker that fails save-time validation. - Task 5: require `metadata.targetLocalDataset = "samples-mix-private"` per the updated _shared-rules.md (was "override if needed", now required for deterministic dataset binding). - New Task 6: capture the cross-engine audit Item 5⚠️ (Mistral mapper sets `polygon: []`). Populate per-word polygons from Mistral's bbox data so E05's VLM+OCR hybrid has spatial info. Update existing mapper test. Benefits both public-API and Foundry paths since they share the mapper. - New Task 7: capture the cross-engine audit Item 9⚠️ (Mistral internal preprocessing not documented). Resolve from Mistral docs and capture in the per-engine OCR doc; same doc covers Foundry-vs-public diffs. - Task 8: align with the post-E01 trigger (`./scripts/run-experiment-benchmarks.sh 02`, which now sends `tags` as object and `persistOcrCache: true` by default). Tag is now `{"experiment":"02-mistral-doc-ai-azure"}` (full slug) — supersedes the old `experiment-02-mistral-azure` short form. - New Tasks 9, 10: explicit deliverables — `benchmark-run.json` saved from the download endpoint; one OCR fixture captured to `__fixtures__/experiment-02/` for the test replay. - Task 11: replaces the old "Mock-based tests" line with the canonical two-layer pattern (static + runtime) and the `process.env.CI` gate for the runtime suite, pointing at `experiment-01-neural-doc-intelligence.test.ts` as the reference implementation. - Task 12: SUMMARY.md fields explicit — deployment id, run id, headline metrics, comparison row vs public-API, confidence-distribution observations, infra-fix log, bbox-fix verification. - Cross-engine audit follow-through section: list both⚠️ items (5 and 9) explicitly with the tasks that address them. - Header pointer to `_shared-rules.md` so the canonical patterns and "fix forward" rule are surfaced up front. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Forks the Mistral provider for the Foundry serverless deployment (mistral-document-ai-2512 on strukalex-8338-resource), running parallel to the public-API path. Single sync HTTP call, same canonical post-processing chain, with the Foundry-specific endpoint, Bearer auth, and stricter request body (no confidence_scores_granularity). Includes the cross-engine audit follow-throughs from E02's brief: the shared Mistral mapper now populates word/line polygons from optional bbox corners, and the provider doc captures Mistral's internal preprocessing handling. Activity emits ocrResponse alongside ocrResult so benchmark_ocr_cache populates for sync providers. Live benchmark: 33/33 OCR'd via Foundry (run id 0fd7eef6, 156 s wallclock through the deployment's 10 RPM quota); annotation step is skipped on the deployment despite a valid document_annotation_format in the request (pages_processed_annotation: 0) — captured as an open follow-up in SUMMARY.md. Tests: 21/21 in experiment-02-mistral-doc-ai-azure.test.ts (static + runtime against local Temporal at localhost:7233, CI gate via process.env.CI), 9/9 unit tests for the new activity, 4/4 mapper tests covering the bbox-population path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two missing pieces caused E02's first benchmark to look "completed but no extraction": 1. The Foundry route silently skips the annotation step unless the JSON-schema wrapper has `strict: true` (Microsoft Q&A 5767943). Without it, Foundry returns OCR markdown and `document_annotation: null` — so OCRResult.documents/keyValuePairs were always empty. Fix: emit `strict: true` from the shared field-definitions-to-mistral-annotation-format converter. Public- API path is unaffected (the public Mistral OCR API ignores the flag). 2. The workflow JSON's templateModelId default was a stale id copied from the public-Mistral template; the activity logged "template_not_found_or_empty_schema" and skipped sending document_annotation_format at all. Fix: point it at the seeded `seed-sdpr-monthly-report-template`. Live benchmark with both fixes (run id 3b4baeaf, 149 s): - pass_rate 0.273 (9/33), f1.median 0.563, f1.max 0.965 - precision.mean 0.840, recall.mean 0.454 - 33/33 cache rows now have ~2915 chars of populated annotation (was null on every sample previously). Adds scripts/test-mistral-foundry-single.ts — a one-shot Foundry smoke-test helper that hits the deployment for one sample (~14s) so schema fixes can be verified without burning a full 33-sample run. Tests: 22/22 in experiment-02-mistral-doc-ai-azure.test.ts (test for populated annotation replaces the prior "is null" assertion); 4/4 mapper tests; converter test asserts strict:true is emitted. Note: bbox population in the mapper still works as designed for any Mistral response that carries per-word bbox corners, but Mistral OCR itself doesn't return per-word bboxes (only image bboxes for embedded charts/figures), so polygons stay empty in production traffic against this engine. Documented as engine-shape, not a bug. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds an editable prompt-iteration kit and bakes the tuned prompts into
the E02 workflow JSON's mistralAzureOcr node parameters.
Changes:
- Converter now accepts a `descriptions` overlay (per-field
`description` strings attached to each JSON Schema property) and a
`numericFieldsNullable` flag (every numeric field becomes
`["number", "null"]`, letting Mistral distinguish blank cells from
explicit zeros — the form's ground truth treats `""` as blank and
`"0"` as the literal value, and the unconstrained schema lost that
distinction).
- Activity (mistralAzureOcrProcess) forwards `fieldDescriptions` and
`numericFieldsNullable` from its params through to the converter.
- E02 workflow JSON's mistralAzureOcr node now sets:
- documentAnnotationPrompt (2.1 KB SDPR-aware instructions)
- fieldDescriptions (74 fields)
- numericFieldsNullable: true
- experiments/results/02-mistral-doc-ai-azure/iteration/ — editable
source-of-truth (prompt.md + field-descriptions.json + README) plus
last-{request,response,diff} for the most recent iteration.
- apps/temporal/src/scripts/iterate-mistral-extraction.ts — single-
document smoke-test against Foundry (~14s per call) for fast
iteration on the prompt without burning a 33-sample benchmark.
Live benchmark with the tuned prompts (run id 21ce5b11, 145s for 33
samples): pass_rate 0.485 (16/33), f1.median 0.770, f1.mean 0.705,
precision.mean 0.900, recall.mean 0.621, falsePositives.mean 2.30 —
essentially even with E01 (pass_rate 0.515, f1.median 0.806). E02 now
beats E01 on f1.mean and recall.mean; trails on pass_rate by one
sample and on falsePositives.mean (Mistral is a general-purpose
engine, occasionally hallucinates a value).
Tests: 138 in the touched suites (converter + activity + experiment-02
+ activity-registry); full temporal sweep 966 pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ld breakdown Tooling additions used to prepare steering-committee briefing materials. - hitl-planner: track per-document sample index on each prediction; compute per-document pass-through (zero-flag rate) at each target recall and emit hitl-passthrough.csv alongside the existing per-category / combined CSVs. - report-errors: add --include-all-predictions to emit all-predictions.csv (every cell, matched + non-matched), same schema as wrong-by-category.csv. Share wrapper passes the flag through. - hitl-per-category-plots: new diagnostic plotting tool that emits one two-panel PNG per category (recall-vs-workload + threshold-vs-workload) to make threshold cliffs in small-error categories (e.g. name, date) visually obvious. Share wrapper included. - verify-all-predictions: aggregate-only verification over all-predictions.csv (counts, accuracy per category, no values). Also emits per-field-breakdown with reviewable counts so the "fields with content per doc" figure is traceable to specific fields.
The extraction-experiment per-workflow tests (experiment-01..05) were written
against the old inline-graph GraphWorkflow contract. develop since changed it:
- GraphWorkflowInput.graph (inline) -> workflowVersionId (graph loaded via
the getWorkflowGraphConfig activity by id)
- GraphWorkflowResult no longer returns `ctx`; terminal ctx is read via the
getStatus query
Port each suite to the current contract (mirrors graph-workflow.test.ts):
- makeWorkflowInput returns workflowVersionId + a test-only __testGraph,
stripped before execution
- withGraphConfigLoader mocks getWorkflowGraphConfig to serve the test graph,
and mocks document.getStatus for develop's new post-execution hook
- read terminal ctx via the getStatus query inside worker.runUntil
Also guard redactCtxForQuery against ctx values that are `undefined`:
JSON.stringify(undefined) returns undefined, so `.length` crashed the getStatus
query whenever any ctx key held undefined (surfaced by these tests on a real
Temporal cluster). Defensive one-liner; strictly more permissive.
All 5 suites pass (119 tests) against the dev-stack Temporal; tsc --noEmit clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Single living tracker for the OCR-engine bake-off review findings (B1-B7, E-1/E-2, R1-R5, T1-T6, W1, H1-H5, U1). Fixes land on this branch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… (B7) evidenceConfidence scored any field with an empty source_quote at 0.50, including fields the form legitimately left blank — dragging the page-mean confidence below the 0.95 HITL gate and routing fine documents to review. Now a missing quote only lowers confidence for a populated value; a blank value with no quote is a correct empty extraction and stays at 0.95. Does not affect benchmark accuracy metrics (confidence is not an input to f1/precision/recall) — only live HITL routing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…idence (B1) The hybrid mapper replaced the result pages with the real Azure DI pages, whose hundreds of ~0.99 OCR word confidences swamped the evidence-based field confidences in ocr.checkConfidence's single average — so empty-quote 0.5 signals never pulled the mean under the 0.95 gate and review never fired (the docstring's 'gate behaviour unchanged' claim was false). clonePages now omits per-word confidence (Word.confidence made optional; both readers already tolerate undefined), so the gate is driven by the structured-field evidence confidence. Word content + polygons kept for layout. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… operation-location (B2/B3)
B2: a synchronous 200 was always cast to the long-running-operation envelope
({status,result}); a bare CuAnalyzeResult ({contents}) was silently dropped
and the code fell through to polling. Now both shapes are handled.
B3: the poll fallback GET a URL built from a locally-generated requestId that
was never sent to CU (guaranteed 404). Now we fail fast with a clear error
when a 202 carries no operation-location header and no inline result.
Analyze-path coverage (inline-200 / fail-fast) tracked under T5.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…+ tests (B4/T1) The fence regex was anchored to end-of-string, so any text after the closing ``` made the whole fenced string (backticks included) fall through to JSON.parse and throw. The two byte-identical copies (E04 + E05) are now one shared parseVlmStructuredJson that matches the first fenced block (no end anchor), so a trailing model note no longer crashes parsing. Adds vlm-response-parser.test.ts covering clean/fenced/trailing-prose/ prose-before/missing-fields/missing-source_quotes/malformed cases (T1 — previously zero direct tests for this code). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…lure (B5)
A terminal status:"failed" (and a succeeded-but-missing-analyzeResult) threw
a plain retryable Error, so with maximumAttempts:5 a genuinely-failed document
was re-submitted up to 5x. Now throws ApplicationFailure.create({nonRetryable:
true}), mirroring develop's poll-ocr-results.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A blank/absent CU date produced valueDate:"", which downstream extractAzureFieldDisplayValue prefers over content — so a blank date read as a populated value. Now valueDate is only set when non-empty, mirroring the numeric blank handling. Adds a blank-date assertion to the existing missing-fields test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… (E-1/E-2/T3) The schema-aware evaluator compared every value via String(), so nested objects collapsed to "[object Object]" (any two objects falsely matched — E-1) and arrays of rows could never match (E-2). compareField now detects structured GT by value shape and compares it structurally: nested objects field-by-field, arrays of rows positionally with an explicit length check — leaves still flow through the per-field matching rules. Scalar one-of- alternate arrays (e.g. ["","0"]) are unchanged. Adds T3 tests: nested match/mismatch, table match / length-mismatch / row-mismatch, scalar-array one-of, and empty-GT. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…n (R4) parseNumeric now reuses field-format-engine's "number" canonicalizer, which strips currency symbols (£$€¥) as well as commas/spaces — so "$6,191.12" parses instead of silently falling back to an exact-string mismatch. Extracts the byte-identical levenshteinDistance (duplicated in the schema-aware and ocr-correction evaluators) into evaluators/levenshtein.ts. Adds a currency numeric-match test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…t found' (R5) loadTemplate's catch returned null on any failure, and the caller maps null to 'template field_schema not found' — so a transient Prisma/connection error surfaced as a missing-template error (and Temporal retried the wrong failure mode). Now genuine DB errors rethrow; a genuinely-absent/empty schema still returns null. Applied to both VLM-direct (E04) and VLM-hybrid (E05). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Both azure-cu-analyze and azure-cu-deploy-analyzer carried identical local readEnv/sleep copies; export them from the shared azure-cu-client instead. R1/R2 (DI submit+poll consolidation, base-Mistral-loader parameterization) deferred: they require modifying develop's production OCR activities, which is safer as its own focused PR with the production integration tests. Noted in the tracker. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…(W1) EXPECTED_ACTIVITY_TYPES was missing azureOcr.readPlain, vlmOcrHybrid.extract (E05) and benchmark.flattenPredictionFromRefs, and the subset/>= assertions let the gap pass — those activities had no registration-test coverage. Adds the three types and replaces the >= size check with an exact set comparison so the registry can't drift ahead of the fixture again. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…/T5) Extracts the B2 synchronous-200 interpretation into a pure extractInlineResult helper and unit-tests both shapes (envelope + bare result) and the no-result cases (azure-cu-analyze.test.ts). T2 is covered by the B1 hybrid-mapper test (replicates the gate average over a result with 50 high-confidence DI words + empty quotes, asserts it stays at 0.5). T4/T6 deferred — they edit dev-cluster e2e tests (describe.skip under CI) that can't be validated here. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…his work) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ep 1) Enables R1 (drop azure-di-read-plain in favour of the regular DI submit/poll path): vlmOcrHybrid.extract now accepts layoutResponse as OCRResponse | OcrPayloadRef and resolves the ref internally via loadOcrResponseFromPort — the same convention azureOcr.extract uses, so only the small ref sits in Temporal history, never the full response. Backward-compatible (still accepts an inline OCRResponse). Next: rewire the E05 graph to submit->poll(markdown), remove azure-di-read-plain + its registry entries, then rebuild the integration tests to mock only paid APIs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Captures branch state, environment, resolved design decisions (mock-only-paid, R1 option A, R2 variant naming), gotchas (3-list+2-fixture activity registry, graph node format, DI port contract), and the step-by-step remaining plan so the work survives a chat compaction. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ll (R1) Remove the bespoke `azureOcr.readPlain` activity and rewire the VLM+OCR hybrid templates (E05/E07/E08) to use the production `azureOcr.submit` → `azureOcr.poll` pair with `outputFormat=markdown` for the DI pre-pass. The hybrid consumes the poll's `ocrResponseRef` (the lightweight OcrPayloadRef it already resolves internally), keeping the full payload out of Temporal history. Adds a `diModelId` (prebuilt-layout) ctx distinct from the VLM `modelId`, plumbed through `file.prepare`. - Delete `activities/azure-di-read-plain.ts`; drop `azureOcr.readPlain` from the 3 registration lists + both test fixtures. Caught a pre-existing gap: the backend spec was missing `vlmOcrHybrid.extract` (now exact bijection). - Add a shared "mock only paid services" integration harness (`__testlib__/integration-harness.ts`): real worker + real activities + real Postgres/blob, stubbing only the paid APIs (Azure DI via MOCK_AZURE_OCR, Azure OpenAI via axios-mock-adapter). Migrate the E05 runtime tests onto it — confidence is now driven by the real gate over the mapper's evidence-synthesised field confidences, not a hard-coded value. - Update E05 static assertions for the submit/poll chain. Removes the deferral note's R1 blocker. E05: 32/32 pass (incl. 2 real end-to-end runs against local Temporal). Also removes the now-stale continuation/handoff doc. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-keyed activity (R2) Collapse the duplicated `mistralAzureOcr.process` into `mistralOcr.process` with a `variant: "native" | "azure"` param. `native` = public Mistral Document AI API (confidence_scores_granularity, MISTRAL_API_KEY); `azure` = Azure Foundry deployment (Bearer MISTRAL_DOC_AI_AZURE_*, omits the granularity field, supports the ocr3Features knobs + per-field description / nullable overlay). The previously byte-identical helpers (readBlobData, buildDataUrl, prismaFieldToAnnotationInput, loadTemplateForAnnotation) now exist once. Unify the return on the lightweight `OcrPayloadRef` (the history-small pattern the native production path + E05 already use): both variants persist the result blob and emit the raw `ocrResponse` for benchmark cache capture. The azure experiment (E02) switches from an inline OCRResult to the ref; cleanup already resolves refs transparently. - Delete `ocr-providers/mistral-azure/` (activity + test); fold its coverage into `activities/mistral-ocr-process.test.ts` (both variants). Remove `mistralAzureOcr.process` from the 3 registration lists + 2 spec fixtures. - Rewire the E02 graph to `mistralOcr.process` with `parameters.variant: "azure"` + a `documentId` input. - Convert the E02 runtime tests to the mock-only-paid harness (extend it to stub the Mistral axios call), driving the real confidence gate. Closes T6: the raw ocrResponse is asserted to reach ctx for persistOcrCache. E02: 23/23 (both runtime branches live). Mistral unit: 117/117. Backend spec green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the over-mocked runtime (whole activities stubbed) with the shared harness: real worker + real activities + real DB/blob, stubbing only the Azure OpenAI call (axios). Confidence is driven by the real gate over the mapper's evidence-synthesised field confidences. T4: the mapper fixture test now loads the REAL seeded template field types (DB-gated) instead of guessing from `applicant_`/`spouse_` key-name prefixes, and asserts every template-typed numeric field keeps its parsed `valueNumber` (the silent drop the heuristic could cause). The near-vacuous evidence assertion (`> populated/2`) is tightened to require quotes on ≥ 90% of a substantial populated set. E04: 24/24 (incl. both runtime branches live). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add an injectable-response seam to the MOCK_AZURE_OCR path (poll-ocr-results.ts honors MOCK_AZURE_OCR_RESPONSE) so the harness can feed the real submit/poll/extract/gate path a chosen Azure DI analyze response and drive the real confidence gate. Extend the harness with a `di` config. Convert E01's runtime to the harness: real worker + real activities + real DB/blob, stubbing only the DI SDK. Both reviewSwitch branches run live (high → no review; low → humanReview → store), asserting the persisted ocrResult. Also tick R1/R2/T4/T6 in the tracker (all closed) and record the harness-conversion coverage (E01/E02/E04/E05 done; E03 pending CU axios mocks). E01: 19/19. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rness Add an injectable-result seam to the MOCK_AZURE_CU path (azure-cu-analyze honors MOCK_AZURE_CU_RESULT) so the harness can feed the real analyze → mapping → gate path a chosen CuAnalyzeResult and drive the real confidence gate. Extend the harness with a `cu` config (CU is not in production, so a seam in the activity is cleaner than intercepting its axios.create instance). Convert E03's runtime to the harness: real worker + real activities + real DB/blob, stubbing only the CU API (deploy + analyze via the MOCK_AZURE_CU seam). Both reviewSwitch branches run live. Completes the mock-only-paid conversion for all 5 experiment suites (E01–E05). E03: 22/22. CU unit tests: 21/21. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…s/temporal/scripts The 14 experiment/benchmark dev scripts lived under apps/temporal/src/scripts, so they compiled into the worker's dist/ and were coupled to the worker's type-check. Move them to apps/temporal/scripts/ (a sibling of src/), excluded from the worker build and type-checked via a dedicated tsconfig.scripts.json. - git mv src/scripts/*.ts -> scripts/*.ts (14 files) - fix relative imports (../x -> ../src/x) and __dirname path depth - tsconfig.json: exclude scripts/ from the emitted worker build - tsconfig.scripts.json: noEmit type-check of scripts + src - package.json: type-check runs both; add type-check:scripts - update usage/doc references src/scripts/ -> scripts/ Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…, not a hardcoded field set
The schema-aware evaluator hardcoded `PRESENCE_ONLY_FIELDS = {"signature",
"spouse_signature"}` — SDPR-form-specific field names baked into generic
benchmarking infrastructure that must support arbitrary workloads (violates the
"no document-specific implementation" rule).
- evaluator: add `presence` as a first-class matching rule (alongside
exact/fuzzy/numeric/date/boolean) via a `presenceMatch` handler; remove the
hardcoded field-name set and its branch. Which fields use it is now supplied
per-benchmark via `fieldRules`.
- seed: `seedExperimentWorkflows` reads `evaluatorConfig` from each workflow
template's metadata (falling back to an exact-match default), so the
document-specific matching config lives in per-experiment data, not in the
generic seed or the engine.
- templates: the 7 experiment workflow JSONs declare
`metadata.evaluatorConfig.fieldRules` opting signature/spouse_signature into
the presence rule — preserving prior behaviour.
- tests: the presence-only suite now configures the rule via fieldRules.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ration-checklists
- design spec: point the `.env.sample` section (and checklist item 7) at the
root `.env.sample`, noting per-app `apps/{backend-services,temporal}/.env`
were consolidated into a single root `.env` (dbarkowsky review comment).
- rename `experiments/briefs/` → `experiments/integration-checklists/` and
sweep all path references across spec, docs-md, test comment, scripts, and
experiment result logs; the files are per-engine integration checklists,
not results/methodology (dbarkowsky review comment).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-hitl-committed # Conflicts: # .gitignore # docs-md/workflows/02-mistral-doc-ai-azure-OCR.md # docs-md/workflows/03-content-understanding-OCR.md # docs-md/workflows/04-vlm-direct-OCR.md # docs-md/workflows/05-vlm-ocr-hybrid-OCR.md # docs-md/workflows/templates/experiment-01-neural-doc-intelligence-workflow.json # docs-md/workflows/templates/experiment-02-mistral-doc-ai-azure-workflow.json # docs-md/workflows/templates/experiment-03-content-understanding-workflow.json # docs-md/workflows/templates/experiment-04-vlm-direct-workflow.json # docs-md/workflows/templates/experiment-05-vlm-ocr-hybrid-workflow.json # docs-md/workflows/templates/experiment-07-vlm-ocr-hybrid-gpt-4o-workflow.json # docs-md/workflows/templates/experiment-08-vlm-ocr-hybrid-gpt-5.2-workflow.json
…, not a hardcoded field set
The schema-aware evaluator hardcoded `PRESENCE_ONLY_FIELDS = {"signature",
"spouse_signature"}` — SDPR-form-specific field names baked into generic
benchmarking infrastructure that must support arbitrary workloads (violates the
"no document-specific implementation" rule).
- evaluator: add `presence` as a first-class matching rule (alongside
exact/fuzzy/numeric/date/boolean) via a `presenceMatch` handler; remove the
hardcoded field-name set and its branch. Which fields use it is now supplied
per-benchmark via `fieldRules`.
- seed: `seedExperimentWorkflows` reads `evaluatorConfig` from each workflow
template's metadata (falling back to an exact-match default), so the
document-specific matching config lives in per-experiment data, not in the
generic seed or the engine.
- templates: the 7 experiment workflow JSONs declare
`metadata.evaluatorConfig.fieldRules` opting signature/spouse_signature into
the presence rule — preserving prior behaviour.
- tests: the presence-only suite now configures the rule via fieldRules.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…iment/09-sdpr-hitl-committed
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| } | ||
|
|
||
| function escapeCell(s: string): string { | ||
| return s.replace(/\|/g, "\\|").replace(/\n/g, " ⏎ "); |
| } | ||
|
|
||
| function escapeTableCell(s: string): string { | ||
| return s.replace(/\|/g, "\\|").replace(/\n/g, " "); |
| } | ||
| } | ||
| if (changed) { | ||
| fs.writeFileSync(filePath, `${JSON.stringify(gt, null, 4)}\n`); |
|
|
||
| await fs.promises.writeFile( | ||
| path.join(ITERATION_DIR, "last-response.json"), | ||
| JSON.stringify(resultBody, null, 2), |
| elapsedMs, | ||
| rawResponseSummary, | ||
| }); | ||
| await fs.promises.writeFile(path.join(ITERATION_DIR, "last-diff.md"), md); |
| elapsedMs, | ||
| rawResponseSummary, | ||
| }); | ||
| await fs.promises.writeFile(path.join(ITERATION_DIR, "last-diff.md"), md); |
| }; | ||
| await fs.promises.writeFile( | ||
| path.join(ITERATION_DIR, "last-response.json"), | ||
| JSON.stringify(responseForDisk, null, 2), |
| ); | ||
| const outFile = path.join(outDir, "benchmark-run.json"); | ||
| fs.mkdirSync(outDir, { recursive: true }); | ||
| fs.writeFileSync(outFile, JSON.stringify(downloadResp.data, null, 2)); |
Comment on lines
+85
to
+89
| { | ||
| model: "mistral-document-ai-2512", | ||
| document: { type: "document_url", document_url: dataUrl }, | ||
| document_annotation_format: annotationFormat, | ||
| }, |
| "experiment-02", | ||
| `mistral-azure-ocr-response-${sampleId}-smoketest.json`, | ||
| ); | ||
| await fs.promises.writeFile(outPath, JSON.stringify(data, null, 2)); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft — stacked on top of #165 (E08). Base branch is
experiment/08-vlm-ocr-hybrid-gpt-5.2, so this PR shows only the E09 delta over E08.Two commits on top of E08:
hitl: inline canvas-overlay editor, confidence-tier colors, smarter auto-zoom— generic product feature (frontendannotation/hitl, 6 files). Mergeable on its own.EXPERIMENT (DO NOT MERGE): SDPR HITL timing experiment harness— experiment-specific scaffolding (UNC blob storage, experiment OCR loader/field-filter,scripts/sdpr-experiment/). Not for merge.Data-hygiene notes
scripts/sdpr-experiment/output/corrections.csv,sessions.csv): fully removed — they never existed in E08 and are not present in any reachable commit of this branch. Clean.sdpr-monthly-report-sample.jpg,REPORT.md,REPORT.pdf,hybrid-model-research.md): these live in the E08 layer (already onorigin/08). This PR's tip commit deletes them, so they do not appear in the E09 working tree; they remain in the underlying E08 history.🤖 Generated with Claude Code