Conversation
Add a short reference and a full requirements spec for document image normalization. Documents added: docs-md/document-image-normalization.md (summary) and feature-docs/007-document-image-normalization/REQUIREMENTS.md (detailed spec). The docs specify storing the original blob and a normalized PDF (documents/{id}/original.{ext} and documents/{id}/normalized.pdf), adding normalized_file_path to the DB, synchronous image→PDF conversion at upload, view endpoint serving the normalized PDF, download serving the original, client-side single PDF viewer contract, conversion failure behavior, lifecycle (delete both blobs), and related out-of-scope items. Notes BLOB_STORAGE.md should be updated to reflect the two-blob pattern.
kmandryk
requested review from
NoorChasib,
alex-struk,
antsand and
dbarkowsky
as code owners
March 12, 2026 21:19
kmandryk
marked this pull request as draft
March 13, 2026 16:55
dbarkowsky
reviewed
Mar 17, 2026
dbarkowsky
left a comment
dbarkowsky
left a comment
Collaborator
There was a problem hiding this comment.
I don't see anything that would keep us from going forward with this. Hard to say for sure until we see how the implementation goes.
Some thoughts that maybe we don't need answers for right now:
- If a document is already a PDF on intake, are we storing an identical copy of that in the normalized path? If not, how do we reconcile where to find it later?
- What's the trade off in compute vs storage? Like, if it's going to be a lot of storage bloat to save these files, could we just convert them as needed? Depends on the cost of conversion, I guess.
- Maybe the originals could be compressed in some way to save space?
- Maybe the normalized files have a lifespan? Like, if it has been a month since any operations were done on this file, do we need to keep the normalized version, or do we just do that process again if it's needed?
Introduce PDF normalization pipeline and persist normalized PDF blobs. Adds a PdfNormalizationService (using pdf-lib and sharp) to validate uploads and convert images to PDF; new PdfNormalizationError type for conversion failures. Persist a normalized_file_path on documents/labeling records and wire it through DB services, DTOs, and tests. DocumentService, LabelingOcrService, GroundTruthGenerationService, HitlDatasetService and related controllers updated to write/read normalized PDFs and to handle conversion failures (status conversion_failed and new response DTO). Add a /:documentId/view endpoint to stream the normalized PDF and derive download MIME from original filename. Added utilities for extension/mime detection (original-blob-key.util, mime-from-filename). Updated tests, Prisma schema/migration, and docs; added runtime deps pdf-lib and sharp.
Apply small formatting and style cleanups across several frontend files. Changes include: condensing the pdfPageDimensions conditional in DocumentViewer, formatting the fetch call and download assignment in DocumentViewerModal, adding spacing around the pdfjs worker import in ReviewWorkspacePage, and rewrapping the labelingDocument type assertion check in ProjectDetailPage. These are non-functional/whitespace edits to improve readability and satisfy linters/formatters.
Clean up and refactor several backend files: removed the unused getMimeType helper from hitl-dataset.service.ts; reorganized and reordered imports (document controller, labeling services and tests) to satisfy linting and clarity; adjusted minor formatting (wrapped blobStorage.read argument, wrapped long ApiForbiddenResponse description, and fixed line breaks in tests). Also corrected DTO import order in labeling-conversion-failed-response and small test/controller call formatting. These are non-functional cleanup changes to improve code style and maintainability.
kmandryk
marked this pull request as ready for review
April 1, 2026 17:26
Resolve apps/backend-services/package.json and package-lock.json: keep pdf-lib and sharp from this branch, add type-check script and @biomejs/biome 2.4.8 from develop, then regenerate the root lockfile with npm install. Made-with: Cursor
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
dbarkowsky
approved these changes
Apr 1, 2026
dbarkowsky
left a comment
dbarkowsky
left a comment
Collaborator
There was a problem hiding this comment.
It works though and I can see the normalized version beside the original:
I think there's opportunities to save storage space and maybe some ways we could expect paths to be a certain format instead of storing them in the database.
I'm also wondering if it makes sense to use the pdf for the frontend viewing when it's so much larger than the original. The difference for my test file was 4x bigger as a pdf.
dbarkowsky
self-requested a review
April 1, 2026 23:16
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
AI-1042
Changes
Implementing the following:
Testing
Checklist
By submitting this pull request, I acknowledge that I have attempted to meet the following: