Merge pull request #7 from MMoney1988/feat/chunk-record-export

feat(chunk): export provenance-preserving records

Author: MMoney1988

README.md

@@ -6,8 +6,9 @@ It takes JSON exported by [Docling](https://github.com/docling-project/docling)
 
 - a small traceable JSON file with text blocks, page numbers, bounding boxes, and provenance
 - a deterministic Markdown quality report that highlights missing fields, broken provenance, invalid bounding boxes, and layout/noise signals
+- provenance-preserving JSONL chunk records from normalized parser output
 
-This project does not try to replace Docling. Docling does the PDF parsing. This repo checks whether the parser output is structured, traceable, and safe enough to review before using it in downstream workflows.
+This project does not try to replace Docling. Docling does the PDF parsing. This repo checks whether the parser output is structured and traceable for human review before downstream use.
 
 It does **not** claim OCR quality, RAG answer quality, table reconstruction, legal extraction, compliance readiness, production extraction accuracy, or enterprise readiness.
 
@@ -83,7 +84,7 @@ python -m pdf_quality_report.report \
 
 ## Export Clean Markdown
 
-Export normalized blocks to clean Markdown with source reference comments for downstream review or RAG ingestion preparation:
+Export normalized blocks to clean Markdown with source reference comments for downstream review or retrieval-oriented preparation:
 
 ```bash
 python -m pdf_quality_report.to_markdown \
@@ -101,6 +102,28 @@ pdf-quality-markdown \
   --output /tmp/page_012.md
 ```
 
+## Export Chunk Records
+
+Export normalized blocks to JSONL chunk records with block IDs, page numbers, bounding-box references, and source metadata
+preserved for downstream review or retrieval-oriented preparation:
+
+```bash
+python -m pdf_quality_report.chunk \
+  --input examples/mdpi_pdf_elements/page_006/normalized_blocks.json \
+  --output /tmp/page_006_chunks.jsonl
+```
+
+Use `--out` as a short alias for `--output`. See [Export Chunk Records](docs/how_to_export_chunk_records.md) for a
+short walkthrough.
+
+If the package entry point is installed:
+
+```bash
+pdf-quality-chunk \
+  --input examples/mdpi_pdf_elements/page_006/normalized_blocks.json \
+  --output /tmp/page_006_chunks.jsonl
+```
+
 ## Included Examples
 
 This repository includes derived example outputs for page 6 and page 12 of the sample article:
@@ -146,7 +169,7 @@ These signals identify page-layout elements that may need review before reusing
 - ambiguous_image_blocks: 1
 ```
 
-`REVIEW` means the hard structure checks passed, but warning details still need human review before Markdown export, RAG ingestion preparation, or manual extraction.
+`REVIEW` means the hard structure checks passed, but warning details still need human review before Markdown export, retrieval-oriented preparation, or manual extraction.
 
 ## License
 

docs/how_to_export_chunk_records.md

@@ -0,0 +1,48 @@
+# Export Chunk Records
+
+## What this does
+
+Converts `normalized_blocks.json` into JSONL chunk records.
+
+This export step comes after conversion. It does not read PDFs directly. It reads normalized blocks that already contain
+text, block IDs, page numbers, bounding boxes, and source provenance.
+
+Use it when you want a compact reviewable artifact that preserves source references for downstream review or
+retrieval-oriented preparation.
+
+## What this does not do
+
+No OCR. No table reconstruction. No embeddings. No vector store. No retrieval pipeline. No LLM answer generation.
+
+## Command
+
+```bash
+python -m pdf_quality_report.chunk \
+  --input examples/mdpi_pdf_elements/page_006/normalized_blocks.json \
+  --output /tmp/page_006_chunks.jsonl
+```
+
+```bash
+pdf-quality-chunk \
+  --input examples/mdpi_pdf_elements/page_006/normalized_blocks.json \
+  --output /tmp/page_006_chunks.jsonl
+```
+
+## Output
+
+The export writes one JSON object per line. Each record contains:
+
+- `doc_id`, equal to the stable chunk ID
+- `text`, joined from one or more normalized text blocks
+- `meta.schema_version`, currently `chunk_record_v0`
+- `meta.block_ids`, `meta.page_numbers`, and `meta.bbox_refs`
+- `meta.section_heading` and `meta.section_heading_block_id` when a heading applies
+- `meta.source_identifier` and `meta.source_pdf_hash`
+
+By default, headers, footers, images, empty text, and very short non-heading fragments are skipped. Tables are exported
+only when the normalized table block already has `content.text`.
+
+## When to use it
+
+Use this after the quality report if the normalized blocks are structurally usable and you need source-preserving chunk
+records for review or retrieval-oriented preparation.

pyproject.toml

@@ -27,6 +27,7 @@ docling = [
 pdf-quality-convert = "pdf_quality_report.convert:main"
 pdf-quality-report = "pdf_quality_report.report:main"
 pdf-quality-markdown = "pdf_quality_report.to_markdown:main"
+pdf-quality-chunk = "pdf_quality_report.chunk:main"
 
 [tool.pytest.ini_options]
 testpaths = ["tests"]