Develop - #227
Open
alex-struk wants to merge 193 commits into
Open
Conversation
Introduce a repo-focused wiki under docs-md/wiki with topic pages (index, sources, log, system overview, graph-workflows, HITL, auth-and-groups, deployment-and-ops, open-questions) and guidelines (README). Add scripts/scripts: build-docs-wiki.js and validate-wiki.js plus package.json updates to expose the docs:wiki:check script. Add a GitHub Actions workflow (.github/workflows/wiki-check.yml) to run npm run docs:wiki:check on PRs. Integrate generated wiki into the site: update docs build (docs/build.sh) to run the wiki renderer, add nav link and page entries in docs/* HTML, and include generated wiki HTML files (docs/wiki*.html). Also update CLAUDE.md with wiki usage rules. This enables validation, generation, and navigation of the repo wiki from CI and the static docs site.
Add a GitHub Pages workflow and tighten wiki CI checks; introduce AGENTS.md and ARCHIVE.md to canonize repo wiki rules and archival guidance. Rework many docs and READMEs to rename/replace labeling/training with template-models, document graph-workflow and workflow-builder changes, and surface group-scoped API key behavior. Strengthen contributor/copilot guidance (avoid leaking secrets, curl usage with env API_KEY) and update CLAUDE.md accordingly. Ignore generated wiki HTML in .gitignore, remove stale generated wiki HTML files, and update docs build/validation scripts and package.json to support the new pages/wiki workflows.
The canonical Workflow Builder Guide lives under docs-md/workflow-builder/. Co-authored-by: Cursor <cursoragent@cursor.com>
Replace haproxy.router.openshift.io/ip_whitelist with haproxy.router.openshift.io/ip_allowlist in base Route manifests (frontend & backend) and update overlays. Renamed patch files route-whitelist-patch.yml -> route-allowlist-patch.yml and updated kustomization.yml references in prod and test overlays, plus adjusted the JSON patch paths. Updated documentation (KUSTOMIZE_INSTANCE_TEMPLATE.md) to reflect the new annotation name. This aligns the annotation naming from "whitelist" to "allowlist".
Sets up the parent branch (feature/extraction-experiments) for a 5-engine extraction comparison: neural DI + post-processing, Mistral Doc AI on Azure, Azure Content Understanding, VLM-direct, and VLM+OCR hybrid. Defines parent-branch scaffolding (brief library, deployment-selection plumbing, local-folder dataset seed convention) and per-experiment branch deliverables. Stacks on PR #134's neural training capability. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ting-engine audit, dev-loop, future candidates Updates the extraction-experiments design spec per user feedback: - Replaces the AI-feedback 12-item checklist with one derived from this codebase (file paths, types, conventions actually present here). - Documents the dev loop as "implement → run on real document → run benchmark via API → write mock tests when stable", not an accuracy-threshold gate. - Adds existing-engine audit (Azure DI + Mistral) against the new checklist to the provider-architecture doc on the parent branch. - Adds vendor-doc citations confirming Mistral and CU use OCR-first → LLM annotation, which E05 recreates explicitly. - Adds "Candidates for future rounds" section listing trends not in the core 5 experiments (confidence-based routing, per-field calibration, multi-engine voting, agentic correction, open-source VLMs, etc.) so they can be re-pitched after E01–E05 land. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… scaffolding Removes: - Duplicate "Cross-engine audit" section (already in deliverable #3) - "Expected gaps to surface" speculation - "Open questions / assumptions" section (was for pre-approval review) - Goals section (overlaps Context + deliverables) - Brief-list file-content descriptions - Experiment-branch deliverables list (overlaps the integration checklist) - "Why deferred" column from future-candidates table Compresses: - Three Azure resource tables into one - Per-experiment sections to elevator-pitch length - Sequence of work (drops "commit spec" step which is done) - TODO-callout section moved into the E01 brief description Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
gpt-5.5 GlobalStandard quota is 0 in this subscription. gpt-5 (vanilla, 2025-08-07) has 1000 TPM quota in westus, on the same account as gpt-4o, so deployment succeeded immediately and the plumbing stays simple (single endpoint, deployment-name as the only switch). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ure audit Parent-branch deliverables for the extraction-experiments suite: - experiments/briefs/_shared-rules.md — codebase-derived 12-item engine-integration checklist and dev loop discipline (no accuracy threshold gating; "implement → run on real document → benchmark via API → mock-replay tests once stable") - experiments/briefs/01..05-*.md — per-experiment briefs for neural DI, Mistral on Azure, Content Understanding, VLM-direct, VLM+OCR hybrid - docs-md/EXTRACTION_EXPERIMENTS.md — hub doc with per-experiment status table and engine-integration checklist scaffolds - docs-md/EXTRACTION_PROVIDER_ARCHITECTURE.md — current state + decision to defer formal OcrProvider interface + audit of existing Azure DI and Mistral providers against the 12-item checklist (gaps captured as scoped TODOs in the affected experiment briefs) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…convention
- Append new env vars to apps/{backend-services,temporal}/.env.sample for the
experiments not yet wired (E02 Mistral on Foundry, E03 Content Understanding,
E04/E05 selectable Azure OpenAI deployments). User redirects values to their
personal Azure via the existing override file at
~/.config/bcgov-di/{backend-services,temporal}.env (Claude does not read it).
- AZURE_OPENAI_DEPLOYMENT remains the default fallback for backward compatibility;
AZURE_OPENAI_DEPLOYMENTS is the new comma-separated allow-list for per-node
selection.
- Add data/datasets/ as a tracked path with .gitkeep; gitignore
data/datasets/*/private/** so users can drop personal/test docs without
committing them. Convention documented in the design spec.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…-folder dataset seed
Two pieces of parent-branch scaffolding that all 5 experiments depend on:
1. Per-workflow-node Azure OpenAI deployment selection
- apps/temporal/src/activities/enrich-results.ts: read params.azureOpenAiDeployment
with fall-through to AZURE_OPENAI_DEPLOYMENT env var.
- apps/backend-services/src/azure/azure-openai.controller.ts: GET /api/azure-openai/deployments
parses AZURE_OPENAI_DEPLOYMENTS (comma-separated). Mirrors GET /api/models.
Falls back to AZURE_OPENAI_DEPLOYMENT singleton, then empty array.
- DTO + Swagger decorators, registered in AzureModule.
- Tests: 7 controller cases + 2 enrich-results override cases.
2. Local-folder dataset seed extension
- apps/backend-services/src/seed/local-datasets.ts: pure parser scanning
data/datasets/<name>/{public,private}/manifest.json. Validates JSON +
samples array; warns and skips on errors.
- apps/shared/prisma/seed.ts: new seedLocalDatasets() called after the
existing benchmark-data seed. Idempotent upsert of Dataset + DatasetVersion
rows. Skips quietly when data/datasets/ is empty.
- Test: 10 cases covering parser edge cases + id-helper coverage.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extends the local-folder dataset seed to handle the natural drop pattern of <name>.<image-ext> + <name>.json pairs without a manifest.json. When no manifest exists, scan for pairs and auto-derive one, writing it to disk with metadata.autoGenerated = true so subsequent re-seeds refresh it. User-curated manifests (no auto-gen marker) are never touched. Validated end-to-end on a 33-sample drop (samples-mix/private). - New deriveManifestFromFlatPairs(folderPath, folder, log) — pure helper. - Accepts: jpg/jpeg/png/pdf/tif/tiff/webp/bmp/gif as document extensions. - Warns about unpaired documents (missing JSON) and unpaired ground-truth (missing document). - 7 new tests covering flat-pair, regen-on-rerun, user-curated takes precedence, unpaired warnings, empty folders, mixed-case extensions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Captures the quota-request flow + the deploy command + a fallback (run E04 with gpt-4o and gpt-5 first if approval is delayed). Also notes that gpt-5.5 will live in eastus2 (different endpoint than the existing westus account), so the workflow node may need azureOpenAiEndpoint/Key params threaded — current parent-branch plumbing only threads deployment. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ev only)
The seed creates Dataset/DatasetVersion DB records, but the existing benchmark
infrastructure (Sample Preview, BenchmarkRun execution) pulls files from blob
storage. Without uploading the actual files, dropped local datasets show up
in the DB but not in the UI and aren't usable for benchmarks.
Adds LocalDatasetSyncService that runs at backend startup (NestJS
onApplicationBootstrap) and uploads each local dataset's files to blob storage
following the existing convention used by dataset.service.ts uploadFiles:
groups/{groupId}/benchmark/datasets/{datasetId}/{versionId}/inputs/{file}
groups/{groupId}/benchmark/datasets/{datasetId}/{versionId}/ground-truth/{file}
groups/{groupId}/benchmark/datasets/{datasetId}/{versionId}/dataset-manifest.json
The blob dataset-manifest.json follows the schemaVersion=1.0 shape the existing
runtime expects (with `inputs/` / `ground-truth/` path prefixes baked in).
- Idempotent: skips files already in blob storage; always rewrites the manifest
(cheap, lets it pick up local manifest edits between runs).
- Updates DatasetVersion.storagePrefix and manifestPath to the blob paths so
the runtime finds the synced files.
- Gated to NODE_ENV !== production; can be turned off via
SYNC_LOCAL_DATASETS_ON_START=false.
- Pure helper syncLocalDatasetToBlobStorage exported for unit testing — 4 cases
covering first-sync, idempotent re-sync, partial-resync after files added,
and blob-key convention check.
Also updates seedLocalDatasets to write the eventual blob-convention paths
into the DB records (storagePath, storagePrefix, manifestPath) so the records
are correct from the moment the seed completes, even before the sync service
runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…tself PrismaService is a wrapper that exposes the actual PrismaClient via `prismaService.prisma` (see apps/backend-services/src/database/prisma.service.ts). The sync service was casting the wrapper directly to DatasetVersionUpdater, so this.prisma.datasetVersion was undefined and the storagePrefix update threw at runtime. The unit tests passed because they bypass NestJS DI and call the pure syncLocalDatasetToBlobStorage helper with a hand-rolled stub already shaped like a PrismaClient. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… land
Closes two readiness gaps so the user has zero hand-holding once experiment
implementation completes:
1. Seed scaffolding for the benchmark suite, on the parent branch:
- Adds BenchmarkProject id 'seed-experiments-project' ("Extraction
Experiments") that all 5 experiment branches add their BenchmarkDefinition
rows to.
- Adds a Split per local dataset version with id pattern
'seed-local-{folder}-{visibility}-split', type=test, populated with all
sample IDs from the (auto-generated or curated) manifest. Verified: 33
sample IDs land on the split for the user's samples-mix/private drop.
2. Codifies the per-experiment seeding requirement in _shared-rules.md (item 11
Benchmark integration) — each experiment must seed: a WorkflowLineage +
WorkflowVersion, plus a BenchmarkDefinition with the deterministic id
'seed-experiment-{slug}-definition' referencing the parent-seeded project
and split. Once landed, the definition is invokable via the standard API
endpoint POST /api/benchmark/projects/.../definitions/.../runs.
3. Adds scripts/run-experiment-benchmarks.sh — triggers all 5 experiment runs
(or a subset by leading number) using TEST_API_KEY from the override file.
Tags runs with experiment-{slug} for cross-experiment comparison.
4. Updates docs-md/EXTRACTION_EXPERIMENTS.md with a "Run all benchmarks"
section and the on-demand curl pattern.
After E01-E05 land and seed runs:
./scripts/run-experiment-benchmarks.sh
…fires all 5 runs against the user's seeded dataset.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ll dataset
Two refinements after auditing the existing workflow conventions:
1. Drop the Split seeding for local datasets. BenchmarkDefinition.splitId is
optional in the Prisma schema, and benchmarks should run on the full
user-dropped dataset (33 samples), not a subset. Each experiment's
BenchmarkDefinition will reference the full DatasetVersion directly via
datasetVersionId.
2. Align _shared-rules.md with the existing OCR-provider conventions in the
codebase, which the prior version partially missed:
- Add "Required reading" pointing at ADDING_OCR_PROVIDERS.md,
ADDING_GRAPH_NODES_AND_ACTIVITIES.md, DAG_WORKFLOW_ENGINE.md,
GRAPH_TYPES.md, and the templates folder. These are authoritative.
- Update the "files you may edit" list to call out all THREE activity
registries (apps/temporal/src/activity-registry.ts,
apps/temporal/src/activity-types.ts, AND
apps/backend-services/src/workflow/activity-registry.ts) — missing any
one breaks workflow validation or worker resolution.
- Update item 2 (activity registration): three registries, naming
conventions <provider>Ocr.process or <provider>Ocr.{submit,poll,extract}.
- Update item 8 (workflow graph): templates live at
docs-md/graph-workflows/templates/<slug>-workflow.json, validated by
graph-schema-validator.test.ts. Standard node sequence per the docs.
Provider-specific options via initialCtx, not new global env vars.
- Update item 11 (benchmark integration): no Split — reference full
DatasetVersion directly. Pointer to seedLineageVersion pattern.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Seed.ts now scans docs-md/graph-workflows/templates/ for files matching
'experiment-*-workflow.json'. For each file found, idempotently creates:
- WorkflowLineage seed-experiment-{slug}-workflow
- WorkflowVersion wv_seed-experiment-{slug}-workflow (full JSON config)
- BenchmarkDefinition seed-experiment-{slug}-definition in
seed-experiments-project, targeting the (single) local DatasetVersion
Slug derived from filename: experiment-04-vlm-direct-workflow.json → "04-vlm-direct".
If multiple local datasets exist, workflows can target a specific one via
metadata.targetLocalDataset = "{folder}-{visibility}". Defaults to first.
Verified end-to-end with a smoke-test template (experiment-99-smoke-workflow.json):
seed picked it up, created lineage/version/definition targeting
seed-local-samples-mix-private-v1. Removed file after — clean state.
Implication for experiment AIs: drop a workflow JSON, get the full benchmark
plumbing for free. No edits to seed.ts per experiment.
Updates the briefs accordingly:
- _shared-rules.md item 11: explains the auto-discovery contract.
- E01: copy standard-ocr-workflow-with-corrections.json (already wires
cleanup/normalize/character-confusion/spellcheck), change modelId default
to neural model id, optionally add ocr.enrich.
- E02: copy mistral-standard-ocr-workflow.json, swap mistralOcr.process →
mistralAzureOcr.process.
- E03: base on standard-ocr-workflow.json (async pattern: submit/pollUntil/
extract), replace Azure DI nodes with azureContentUnderstanding.deployAnalyzer
+ azureContentUnderstanding.analyze.
- E04: base on mistral-standard-ocr-workflow.json (sync), insert
pdf.renderToImages + vlmDirect.extract.
- E05: base on standard-ocr-workflow.json, replace extraction with
pdf.renderToImages + azureOcr.readPlain + vlmOcrHybrid.extract.
Each variant in E04/E05 can have its own JSON file (one per variant) — each
gets its own BenchmarkDefinition automatically.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… report
Three changes so the user has a populated final-tip branch ready to run all
benchmarks and produce a cross-experiment report:
1. Chained branches (was hub-and-spoke).
- E01 branches from feature/extraction-experiments
- E02 branches from E01
- E03 branches from E02
- E04 branches from E03
- E05 branches from E04
By the time E05 lands, its tip contains every workflow JSON, every
provider folder, and every seed change from E01-E05. Updates the spec
diagram, hub doc, _shared-rules.md, and each per-experiment brief's
"Branch" line to reflect this. Also enumerates the exact `git checkout -b`
commands per experiment in the hub doc.
2. E01 brief simplification (per user).
- modelId default = "test" (the existing trained neural model — no
training step needed in this experiment).
- Drops spellcheck node (`ocr.spellcheck`) from the workflow.
- Drops LLM enrichment (`ocr.enrich`) — out of scope for E01.
- Drops cross-field validation (`document-validate-fields`) — out of scope.
- Final node sequence: file.prepare → azureOcr.submit/poll/extract →
ocr.cleanup → ocr.normalizeFields → ocr.characterConfusion →
ocr.checkConfidence → reviewSwitch → humanReview/storeResults.
- Adds an explicit "Out of scope" section so future readers don't add
these activities back unintentionally.
3. scripts/compare-experiment-benchmarks.sh.
- Lists runs in seed-experiments-project, filters by tag pattern
experiment-*, picks the most recent COMPLETED run per tag.
- Downloads each via the existing GET /runs/:runId/download endpoint to
/tmp/extraction-experiments-YYYYMMDD-HHMMSS/<slug>/run.json.
- Writes COMPARISON.md — a markdown table with status, field/character/
word accuracy, duration, cost, run id per experiment.
- Intentionally simple ("start with something simple" per user); easy to
extend with per-field-class metrics, P95 latency, etc.
- Hub doc updated with the full two-command flow:
./scripts/run-experiment-benchmarks.sh
(wait for runs)
./scripts/compare-experiment-benchmarks.sh
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three references in the brief updated. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ult) The existing standard-ocr-workflow-with-corrections.json template already sets ctx.modelId.defaultValue to sdpr_synth_test, so the experiment AI just copies the template and leaves modelId untouched. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ob sync)
Add docs-md/local-datasets-seeding.md covering the
data/datasets/<name>/{public,private}/ convention, manifest-driven and
flat-pair layouts, seedLocalDatasets() DB seeding, and the dev-only
LocalDatasetSyncService blob upload (SYNC_LOCAL_DATASETS_ON_START /
FORCE_RESYNC_LOCAL_DATASETS).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drops in `docs-md/graph-workflows/templates/experiment-01-neural-doc-intelligence-workflow.json`,
auto-discovered by `seedExperimentWorkflows()` and benched against the user's
`seed-local-samples-mix-private-v1` dataset (33 samples) using the trained
neural model `sdpr_synth_test`.
Chain: file.prepare → azureOcr.submit/poll/extract → ocr.cleanup →
ocr.normalizeFields → ocr.characterConfusion → ocr.checkConfidence →
reviewSwitch → (humanReview)? → ocr.storeResults. Spellcheck dropped per the
brief; LLM enrichment and cross-field validation out of scope.
Real-API benchmark run id 2295feed-1c99-493e-ae20-546499b5d685: pass_rate
0.515, F1 median 0.806, precision 0.90, recall 0.59 over 33 samples in ~83s.
Workflow chain executed end-to-end on every sample.
Tests: 17 static + structural assertions on the template and recorded neural
OCR fixture (apps/temporal/src/experiment-01-neural-doc-intelligence.test.ts).
TestWorkflowEnvironment runtime tests skipped — same temporal.download TLS
blocker as the existing graph-workflow.test.ts; the live benchmark is the
runtime integration coverage.
Parent-shared infra fixes (user-authorized to keep seed-driven and
manual-create dataset paths consistent):
* local-dataset-sync.service.ts: store relative `dataset-manifest.json`
instead of full blob key, matching dataset.service.ts createVersion.
* seed.ts seedLocalDatasets: same relative manifestPath.
* seed.ts seedExperimentWorkflows: switch evaluatorType from the
unregistered `field-accuracy` to `schema-aware`, with a fuzzy default
rule and 0.8 pass threshold.
Outstanding parent gap (NOT fixed here, flagged in SUMMARY):
scripts/run-experiment-benchmarks.sh sends `tags` as an array but
CreateRunDto requires an object — one-line fix in the script.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…mark export
Two unrelated fixes bundled because both surfaced while finishing E01:
1. ts-node-dev hot reload was stopping the worker without respawning.
Root causes:
a. The Rust-backed `NativeConnection` was never explicitly closed,
so its tokio runtime kept the Node event loop alive after
`worker.run()` resolved. ts-node-dev's `--respawn` waits for the
child to exit before starting a new one, so the new instance
never came up. Fixed by `await connection.close()` and an
explicit `process.exit(0)` after the workers stop.
b. `--watch src` told chokidar to watch every file under src/
including JSON fixtures (e.g. the 290 KB neural-OCR poll fixture).
Touching a fixture forced a worker drain. ts-node-dev's
`--ignore-watch` only filters require-loaded files, not
explicitly watched paths, so the only practical fix is to
narrow the watch glob — switched to `'src/**/*.ts'` so JSON
fixture edits no longer trigger a reload.
Verified live:
- touch src/__fixtures__/.../neural-ocr-response-1-81.json → no restart
- touch src/logger.ts → drain → "Worker stopped" → ~600ms later
"Worker initializing" → RUNNING
2. Archive the full benchmark export for run
`2295feed-1c99-493e-ae20-546499b5d685` (the canonical E01 result) at
`experiments/results/01-neural-doc-intelligence/benchmark-run.json`.
Includes per-sample ground truth + prediction + evaluation details
+ per-field results + error-detection analysis. Pulled from
`GET /api/benchmark/projects/.../runs/{runId}/download`. Set up so
each experiment in the stack can drop its `benchmark-run.json`
alongside `SUMMARY.md` and the cross-experiment comparison can read
them locally without re-querying the backend.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three things, all driven by E01 retrospective:
1. Workflow-level runtime tests against the local dev-stack Temporal
(`localhost:7233`). Each test loads the actual JSON template, runs the
real `graphWorkflow` workflow, and replays the recorded neural-OCR
fixture through mocked activities. Covers both `reviewSwitch` branches
— high-confidence skips humanReview; low-confidence routes through
humanReview, gets a `humanApproval` signal, then completes via
`storeResults`. Both tests patch `pollOcrResults` interval/initialDelay
to `1ms` so they finish in ~2s each. 19/19 pass in ~7s. Sidesteps
`TestWorkflowEnvironment.createTimeSkipping()` which TLS-fails on the
binary download; documented as the canonical pattern.
2. `scripts/run-experiment-benchmarks.sh`: send `tags` as an object (not
an array) per the `CreateRunDto` `@IsObject()` validation, and pass
`persistOcrCache: true` by default so each run captures engine
responses for the test fixture.
3. `experiments/briefs/_shared-rules.md` overhaul, incorporating E01
retrospective:
* **No upstream backporting** — fix shared infra bugs in your
experiment branch and let the chained stack carry them upstream.
Replaces the old "stop and raise it back" guidance.
* **Authentication** subsection — how to get and use TEST_API_KEY,
the relevant API endpoints, that the seed-time key is dev-only.
* **Common bugs runbook** — the five footguns E01 hit (script tags
shape, manifestPath value, evaluator name, confidence-threshold
recalibration, hot-reload watcher).
* **Dev loop** rewritten — benchmark-with-persistOcrCache IS the
real-API run, fixture-capture is now an explicit step, full
benchmark export is a deliverable at
`experiments/results/<slug>/benchmark-run.json`.
* **Test layout** — pin per-experiment tests at
`apps/temporal/src/experiment-<slug>.test.ts` with fixtures at
`apps/temporal/src/__fixtures__/experiment-<slug>/`.
* **`metadata.targetLocalDataset` is now required**, not just an
override, so benchmark-target choice is deterministic.
* **Done criteria** — `benchmark-run.json` saved as a deliverable;
runtime tests required, not just static.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CI's temporal-qa.yml runs `npm test` without a Temporal sidecar, so the runtime layer fails to connect to localhost:7233. Gate the runtime `describe` block on `process.env.CI` (set automatically by GitHub Actions). Static suite continues to run on every CI build. Verified: - local: 19 passed - CI=true: 17 passed, 2 skipped Pattern documented in `_shared-rules.md` so subsequent experiments use the same gate. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The TODO asked E01 to verify Azure DI activities don't have APIM-specific path manipulation that would break under direct access. E01 ran the chain end-to-end against direct Azure DI without issue, so the concern is empirically resolved — no path-prefix surgery was needed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Updates from the E01 retrospective + open audit items: - Task 2: clarify it's a single HTTP call (no two-call orchestration in our code). Mistral's API handles the OCR-then-annotation chain server-side via `document_annotation_format` + `document_annotation_prompt` in the body, mirroring how `mistral-ocr-process.ts` already works. (Was an ambiguity that surfaced in the handoff review.) - Task 3: explicit list of all THREE activity registries + activities.ts export, so the next session doesn't miss one and ship a worker that fails save-time validation. - Task 5: require `metadata.targetLocalDataset = "samples-mix-private"` per the updated _shared-rules.md (was "override if needed", now required for deterministic dataset binding). - New Task 6: capture the cross-engine audit Item 5⚠️ (Mistral mapper sets `polygon: []`). Populate per-word polygons from Mistral's bbox data so E05's VLM+OCR hybrid has spatial info. Update existing mapper test. Benefits both public-API and Foundry paths since they share the mapper. - New Task 7: capture the cross-engine audit Item 9⚠️ (Mistral internal preprocessing not documented). Resolve from Mistral docs and capture in the per-engine OCR doc; same doc covers Foundry-vs-public diffs. - Task 8: align with the post-E01 trigger (`./scripts/run-experiment-benchmarks.sh 02`, which now sends `tags` as object and `persistOcrCache: true` by default). Tag is now `{"experiment":"02-mistral-doc-ai-azure"}` (full slug) — supersedes the old `experiment-02-mistral-azure` short form. - New Tasks 9, 10: explicit deliverables — `benchmark-run.json` saved from the download endpoint; one OCR fixture captured to `__fixtures__/experiment-02/` for the test replay. - Task 11: replaces the old "Mock-based tests" line with the canonical two-layer pattern (static + runtime) and the `process.env.CI` gate for the runtime suite, pointing at `experiment-01-neural-doc-intelligence.test.ts` as the reference implementation. - Task 12: SUMMARY.md fields explicit — deployment id, run id, headline metrics, comparison row vs public-API, confidence-distribution observations, infra-fix log, bbox-fix verification. - Cross-engine audit follow-through section: list both⚠️ items (5 and 9) explicitly with the tasks that address them. - Header pointer to `_shared-rules.md` so the canonical patterns and "fix forward" rule are surfaced up front. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…o negative side effects noted.
Update both PostgresCluster base manifests to expire pgBackRest full backups after 14 days, and document the new retention policy in the environment and instance template docs.
AI-1479 Header Spacing Fix
Resolve conflicts: - .gitignore: union of both sides' ignore rules - apps/temporal/package.json: keep PR-branch config; drop dead build:graph-workflow-config script (package removed on develop) - docs-md/workflows/: adopt develop's graph-workflows -> workflows rename for the experiment OCR docs + templates added on this branch
Extraction experiments — cumulative merge (E01–E08 + improvements + review fixes + harness)
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| if isinstance(v, bool): | ||
| return None # bool is int in Python; we don't treat True/False as numbers | ||
| if isinstance(v, (int, float)): | ||
| if v != v or v in (float("inf"), float("-inf")): # NaN / Inf |
| # Use the full A+B-aware locator so verdicts mirror production behavior. | ||
| located_full = locate_table(tables, analyze_result, cfg) | ||
| located = (located_full[0], located_full[1]) if located_full else None | ||
| strategy_label = located_full[4] if located_full else "not-found" |
| """ | ||
|
|
||
| import json | ||
| import os |
| import csv | ||
| import json | ||
| import os | ||
| import shutil |
| # Build a synthesised benchmark-run.json so the comparison report can | ||
| # include E06 alongside the others. | ||
| per_sample = strategy_per_sample[best_name] | ||
| eval_blob = [] |
| rows = sweep_thresholds(preds, fine_thresholds, e.docs_count) | ||
| xs = [r["non_blank_flagged_per_100_docs"] for r in rows] | ||
| ys = [r["recall"] * 100 for r in rows] | ||
| ceiling = rows[-1]["catchable_errors"] / rows[-1]["errors_total"] * 100 if rows[-1]["errors_total"] else 100 |
| if located is None: | ||
| continue | ||
| table, table_index, column_map, row_map, strategy = located | ||
| rule_label = STRATEGY_RULES[strategy] |
|
|
||
| # ---------- Estimate the no-fuzzy accuracy ---------- | ||
| # Strict accuracies (from V2 report Appendix 11.2, neural V2 strict): | ||
| strict_acc = { |
Reduce pgBackRest retention to 14 days
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
AI-###
Changes
Testing
Checklist
By submitting this pull request, I acknowledge that I have attempted to meet the following: