fix(ocr): stop documents getting stuck in "Processing" on OCR failure

[alex-struk]

Problem

Documents could be orphaned in ongoing_ocr ("Processing" in the UI) forever. The graph workflow set ongoing_ocr up front, but on failure it re-threw without resetting the status, so a permanently-failing OCR run (e.g. a password-protected PDF that Azure rejects) could neither finish nor be deleted (deleteDocument refuses in-flight docs).

On prod this stranded ~233 documents in "Processing". Investigation showed the dominant trigger is password-protected PDFs: Azure Document Intelligence rejects them with 400 InvalidRequest / UnsupportedContent ("File may be password protected").

Fix — guards so a failed run always reaches a terminal status

1. Reject password-protected PDFs at normalization (pdf-normalization.service.ts) via mupdf needsPassword() → conversion_failed, so they never enter OCR. Only open-password PDFs are rejected; permission-only encrypted PDFs (readable by OCR) still pass, so the pdf-lib pass keeps ignoreEncryption: true.
2. Failure-path status transition (graph-workflow.ts): on failure the document moves from an in-flight status to failed, guarded so a progressed (extracted/awaiting_review) or cancelled doc is never clobbered, and a status-update error never masks the original error.

Azure OCR submit errors (4xx and 5xx alike) stay retryable — the submit activity throws a plain Error, so Temporal applies the activity's retry policy. Once retries are exhausted the workflow fails and guard #2 drives the document to failed, so even a permanently-failing submit reaches a terminal status (after the retry budget) rather than stranding in "Processing".

Plus: surface the specific reason to the upload caller — a password-protected PDF now returns code: "password_protected" with an actionable message instead of the generic conversion_failed.

Both failed and conversion_failed are terminal/purgeable, so the ephemeral cleanup janitor reclaims blobs + Temporal records as usual.

Re-run (reprocess) endpoint

POST /api/documents/:documentId/reprocess re-runs a failed or stuck ongoing_ocr document's workflow from its existing normalized PDF (group-scoped). Guards return 409 for a non-runnable state, missing/purged source file, an in-flight run, or a missing workflow config. Looping this endpoint over IDs clears stuck backlogs.

Verification

• Unit/integration: submit-to-azure-ocr (retryable 4xx/5xx/429), full graph-workflow suite incl. the failure-path hook, pdf-normalization password probe + code, ocr.service/ocr.controller reprocess guards, and document.service/upload.controller code/reason surfacing — all green; both apps type-check clean. (Note: graph-workflow.test.ts needs the time-skipping test server from temporal.download, which is firewalled in some CI/sandboxes — pre-stage that binary if needed.)
• Live (local stack): uploading a genuine open-password PDF returns HTTP 422 code: password_protected with the specific message, lands in conversion_failed, never starts OCR, and the ephemeral janitor purges its blobs from storage.

Docs

docs-md/DOCUMENT_IMAGE_NORMALIZATION.md updated; new docs-md/OCR_FAILURE_HANDLING.md (status lifecycle + the two guards + cleanup interaction). OpenAPI spec (docs/assets/openapi.json) regenerated for the reprocess endpoint and the password_protected upload response.

Follow-up (not in this PR)

Once merged & live, re-run the ~233 already-stuck ongoing_ocr documents via the reprocess endpoint so they reach a terminal status and get cleaned up. Separately, the prod app-pg pgBackRest backup repo is full (failed WAL archiving / DB restarts) — tracked independently.

🤖 Generated with Claude Code