Skip to content

feat: dlgrep tool - #696

Open
cau-git wants to merge 19 commits into
mainfrom
cau/dlgrep-support
Open

feat: dlgrep tool#696
cau-git wants to merge 19 commits into
mainfrom
cau/dlgrep-support

Conversation

@cau-git

@cau-git cau-git commented Jul 28, 2026

Copy link
Copy Markdown
Member

What this adds

dlgrep is a grep-style CLI for DocLang documents. It searches semantic units — headings, paragraphs, list items, table cells, captions, formulas, code — rather than lines of text, and returns a reusable XPath address for every match, so results can be fed back into dlgrep show / dlgrep select or into downstream tooling.

$ dlgrep 'GPU|CPU' paper.dclg --within-xpath '/heading[13]' --section -n
/list[4]/ldiv[1]:- AWS EC2 VM ... Nvidia L4 GPU ...
/text[42]:All experiments ... GPU acceleration ... x86 CPU ...

It was developed in a separate repository and depended on unreleased docling-core APIs, which forced a git-branch dependency. This PR moves it in, and adds the deserializer APIs it needs.

Changes to docling-core itself

DocLangSourceMap and DocLangSourceTarget are exported from docling_core.transforms.deserializer, letting a caller map deserialized document items back to their source XML nodes. This is what lets dlgrep report XPath addresses.

How the packaging works

dlgrep lives at packages/dlgrep as a uv workspace member and builds its own dlgrep wheel, published separately to PyPI. It is not merged into the docling-core wheel and docling-core does not depend on it — the dependency runs one way only.

This follows the pattern already used in the docling repo for docling / docling-slim, with one difference: packages/docling there is a dependency-only meta-package, whereas packages/dlgrep carries real source.

Versioning is lockstep. dlgrep shares the docling-core version and pins it exactly (docling-core==<same version>). dlgrep tracks docling-core internals closely enough that an exact pin is honest, and separate versioning would mean a second semantic-release tag stream, which PSR 7.x does not support without significant machinery. The practical consequence: dlgrep gets a release whenever docling-core does, whether or not it changed.

Since the version number now implies a maturity dlgrep does not have, dlgrep --help states that it is experimental and may break in any release.

Release flow. release.sh writes one TARGET_VERSION into three places — the root version, the dlgrep version, and the docling-core== pin — then asserts they agree before anything is committed or tagged. build-packages.sh builds both wheels from that single commit into dist/docling-core/ and dist/dlgrep/, and pypi.yml publishes docling-core first, then dlgrep.

This matters for the cross-cutting case: when a change spans both packages, the dlgrep release pins exactly the docling-core release produced in the same cycle. During development the pin is not consulted at all — the uv workspace source replaces it with the local path, so dlgrep tests always run against the in-tree docling-core.

Two guards, because uv cannot catch a wrong pin on its own (the workspace source drops the version specifier, so a mismatched pin locks and resolves cleanly):

  • .github/scripts/check_lockstep.py asserts version/version/pin agreement. Runs in pre-commit, in the build script, and in release.sh.
  • checks.yml and pypi.yml install both wheels together. A mismatched pin is unsatisfiable against the locally built pair, so it fails before publish rather than shipping an uninstallable wheel.

Repo integration

  • history preserved via git subtree
  • ruff, mypy and pytest configuration inherited from the root; dlgrep is in the dev group so uv sync covers it
  • pre-commit runs dlgrep's tests and the lockstep check
  • CI syncs with --all-packages and smoke-tests the built CLI
  • click and lxml are now declared instead of arriving transitively
  • __version__ reads from package metadata instead of a hardcoded literal

Validation

pre-commit run --all-files passes. Both wheels build, install together into a clean environment, and the CLI runs a real search against a .dclg file.

The failure paths were tested, not assumed: a deliberately mismatched pin fails check_lockstep.py, aborts build-packages.sh, and is rejected at install with No solution found when resolving dependencies — including when the wrongly named version does exist on PyPI.

uvx dlgrep works: dlgrep owns its console script, so no entry-point re-declaration is needed (unlike doclingdocling-slim). Verified against the built wheels.

cau-git added 16 commits July 22, 2026 17:01
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
…59fc7d2'

git-subtree-dir: packages/dlgrep
git-subtree-mainline: 7cd8fa6
git-subtree-split: 46e2f04
dlgrep becomes a uv workspace member under packages/dlgrep, published as
its own wheel and released in lockstep with docling-core: same version,
exact `docling-core==` pin, both wheels built from the same commit. This
removes the git-branch dependency dlgrep needed to track unreleased
docling-core changes.

- workspace wiring; ruff, mypy and pytest config inherited from the root,
  dlgrep in the dev group so `uv sync` covers it
- release.sh writes both versions and the pin from one TARGET_VERSION
- build-packages.sh builds into per-package dist dirs; pypi.yml publishes
  docling-core first, then dlgrep
- check_lockstep.py guards against version drift, which uv cannot catch on
  its own: the workspace source replaces the pin with a local path, so a
  wrong pin resolves and locks cleanly
- checks.yml and pypi.yml install both wheels together, which makes a
  mismatched pin fail before anything is published
- declare click and lxml, previously only transitive
- read __version__ from package metadata instead of a hardcoded literal
- flag dlgrep as experimental in its help output, since the shared version
  number would otherwise imply a maturity it does not have

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@github-actions

Copy link
Copy Markdown
Contributor

DCO Check Passed

Thanks @cau-git, all your commits are properly signed off. 🎉

@mergify

mergify Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Merge Protections

🔴 1 of 2 protections blocking · waiting on 👀 reviews

Protection Waiting on
🔴 Require two reviewer for test updates 👀 reviews
🟢 Enforce conventional commit

🔴 Require two reviewer for test updates

Waiting for

  • #approved-reviews-by >= 2
This rule is failing.

When test data is updated, we require two reviewers

  • #approved-reviews-by >= 2

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

  • title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

@cau-git
cau-git requested review from ceberam and dolfim-ibm July 28, 2026 08:42
@codecov

codecov Bot commented Jul 28, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.28571% with 12 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
docling_core/transforms/deserializer/doclang.py 88.78% 12 Missing ⚠️

📢 Thoughts on this report? Let us know!

cau-git added 2 commits July 28, 2026 13:10
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@cau-git
cau-git marked this pull request as ready for review July 28, 2026 13:36
Comment on lines +33 to +36
"doclang>=0.7,<0.8",
"typer>=0.15.1,<0.25.0",
"click>=8.0.0",
"lxml>=6.0.2",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we move this to a docling-core extra and just make the dlgrep depend on it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants