training: versioned TrainedModels per template

[alex-struk]

Summary

Each retrain previously overwrote the prior TrainedModel and Azure model. Benchmarks pinned ctx.modelId.defaultValue: "<model_id>" could no longer tell what they had measured against, and there was no way to look back at "what was v2 trained on?".

This branch makes TrainedModel versioned per template, with a manual-management UI:

• v1 keeps the bare TemplateModel.model_id for backwards compatibility — existing benchmarks/documents that reference it keep resolving.
• Retrains create v2, v3, … with Azure model ids of <base>-v<n>. The new version becomes active automatically; the prior one is demoted but kept.
• A dataset snapshot (labeled docs + labels at training time) is captured on each successful training and viewable per version.
• New "Versions" tab on the template-model detail page exposes per-version actions: view training data, set active, delete. Deletion tombstones the row, removes the Azure artifact, and is blocked when the version is currently active or referenced by a benchmark definition.
• "Copy model ID" on the template-models list and the detail page header now hands out the active version's model id (e.g., km-invoice-v3) instead of the bare stub. Updates without a manual refresh after retrain / activate / delete.

Schema

Two migrations under apps/shared/prisma/migrations/:

1. 20260501225325_add_trained_model_versioning — adds version, is_active, deleted_at, dataset_snapshot on trained_models; replaces the unique on template_model_id with (template_model_id, version) plus a (template_model_id, is_active) index. Existing rows are backfilled to version = 1, is_active = true.
2. 20260501225925_training_job_target_version — adds target_model_id and target_version on training_jobs so the poller knows the versioned Azure name without re-deriving it.

No data destruction; existing single-version templates upgrade transparently.

Test plan

• npm run db:migrate from apps/backend-services applies cleanly (already applied locally).
• First training of a brand-new template produces v1 with the bare model id (e.g., km-invoice); existing benchmarks/documents continue resolving.
• Retraining a template produces v2 (km-invoice-v2), the prior version is demoted, the new one becomes active, and the Versions tab shows both.
• After training completes, the Versions tab updates without a manual refresh; the list page card and detail page header show the new active model id.
• "View training data" on a version opens an Accordion drawer with the labeled-docs list collapsed by default; expanding shows labels as-of-training.
• Deleting an inactive version tombstones it (Status badge: "Deleted"), and the Azure model goes away. Active versions can't be deleted.
• Trying to delete a version still referenced by a benchmark definition fails with a notification stating the count.
• OCR model picker (/api/template-models/...trained-model-ids) excludes tombstoned versions.

Notes

• The Versions tab does not yet support a "Start from vN" workflow (re-hydrating an old version's labeled-doc set into the live editor for tweaking). Easy to add later if useful.
• Benchmark definitions still pin a model id string. Long-term we may want to normalize that to a FK on a versioned ModelVersion row, but that's out of scope here and would be a separate migration.

🤖 Generated with Claude Code

[dbarkowsky]

Nothing breaking, but some comments worth capturing in another ticket if not addressed before merge.

[alex-struk]

Thanks for the review @dbarkowsky — addressed all five in separate commits on this branch:

#	Comment	Commit
1	Wrap demote + create in a transaction	243592c — new TrainingDbService.replaceActiveTrainedModel(templateModelId, data) runs both writes inside prisma.$transaction
2	Fetch-by-id instead of fetch-all-then-.find	f53b6f7 — new findTrainedModelForTemplate(templateModelId, trainedModelId, opts); used in getTrainedVersionSnapshot, setActiveTrainedVersion, deleteTrainedVersion. Kept the implicit templateModelId-scope check explicit in the where
3	Concurrent retrain race on getNextVersionNumber	4a49913 — went with an in-flight job guard in startTraining instead of SELECT FOR UPDATE. Serializing version assignment still allows two parallel Azure runs (wasted cost, plus an orphaned Azure model when the loser's poller hits @@unique). Refusing the second start matches obvious user intent: wait for or cancel the running job
4	Move benchmark check to TrainingDbService to drop the forwardRef	5cd7c9a — just dropped both forwardRef wrappers → reverted in f399153. I was wrong: there is a real cycle, routed through OcrModule (TrainingModule → BenchmarkModule → OcrModule → TrainingModule, because OcrModule imports TrainingModule for trained-model lookup). My earlier grep only checked src/benchmark/ for direct training references and missed the transitive path. Restored both forwardRefs and added a comment in training.module.ts documenting the cycle so this isn't tripped over again. Moving the check into TrainingDbService (your original suggestion) would have made the cycle worse, not better — so leaving the architecture as-is is the right call here
5	activateMutation missing the response.success check	70d46e4 — was a silent-failure bug, not just style. apiService swallows axios errors into { success: false, message }, so activateMutation was treating backend rejections as success. Both mutations now share a small unwrap() helper

Backend suite: 2075 passing (+8 new tests covering replaceActiveTrainedModel, findTrainedModelForTemplate, findInFlightJobForTemplate, and the guard in startTraining). Nest boot verified after the f399153 revert.