Skip to content

feat(chunk): export provenance-preserving records - #7

Merged
MMoney1988 merged 1 commit into
mainfrom
feat/chunk-record-export
May 13, 2026
Merged

feat(chunk): export provenance-preserving records#7
MMoney1988 merged 1 commit into
mainfrom
feat/chunk-record-export

Conversation

@MMoney1988

Copy link
Copy Markdown
Owner

Summary

  • Add a chunk-record export path from normalized block JSON to JSONL.
  • Preserve source references through block IDs, page numbers, bounding-box refs, source identifier, and source PDF hash.
  • Add CLI support via python module and the pdf-quality-chunk entry point.

Why

  • Provides a small, deterministic export artifact for downstream review or retrieval-oriented preparation.
  • Keeps the scope limited to normalized parser output and provenance preservation.

What

  • Added pdf_quality_report.chunk with load/build/write domain functions and a thin CLI adapter.
  • Added JSONL chunk records with schema_version chunk_record_v0.
  • Added README and walkthrough documentation for the chunk export.
  • Added focused tests for grouping, splitting, filtering, provenance preservation, CLI behavior, and example smoke exports.
  • Does not add retrieval, embeddings, vector stores, LLM use, UI, OCR, or table reconstruction.

Tests

  • .venv/bin/python -m pytest -p no:cacheprovider tests
  • .venv/bin/python -m ruff check src tests
  • git diff --check
  • .venv/bin/python -m pdf_quality_report.chunk --help
  • .venv/bin/python -m pdf_quality_report.chunk --input examples/mdpi_pdf_elements/page_006/normalized_blocks.json --output /tmp/page_006_chunks.jsonl
  • .venv/bin/pdf-quality-chunk --help

Risks

  • The output format is v0 and intentionally small; future format changes should use schema_version.
  • Source metadata can only be preserved when present in the normalized input.
  • The export does not validate semantic correctness or downstream retrieval quality.

Rollback

  • Revert this PR to remove the chunk export module, CLI entry point, tests, and docs.

@MMoney1988
MMoney1988 merged commit 93abf0b into main May 13, 2026
1 check passed
@MMoney1988
MMoney1988 deleted the feat/chunk-record-export branch May 13, 2026 12:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant