feat: dlgrep tool - #696
Open
cau-git wants to merge 19 commits into
Open
Conversation
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
dlgrep becomes a uv workspace member under packages/dlgrep, published as its own wheel and released in lockstep with docling-core: same version, exact `docling-core==` pin, both wheels built from the same commit. This removes the git-branch dependency dlgrep needed to track unreleased docling-core changes. - workspace wiring; ruff, mypy and pytest config inherited from the root, dlgrep in the dev group so `uv sync` covers it - release.sh writes both versions and the pin from one TARGET_VERSION - build-packages.sh builds into per-package dist dirs; pypi.yml publishes docling-core first, then dlgrep - check_lockstep.py guards against version drift, which uv cannot catch on its own: the workspace source replaces the pin with a local path, so a wrong pin resolves and locks cleanly - checks.yml and pypi.yml install both wheels together, which makes a mismatched pin fail before anything is published - declare click and lxml, previously only transitive - read __version__ from package metadata instead of a hardcoded literal - flag dlgrep as experimental in its help output, since the shared version number would otherwise imply a maturity it does not have Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Contributor
|
✅ DCO Check Passed Thanks @cau-git, all your commits are properly signed off. 🎉 |
Contributor
Merge Protections🔴 1 of 2 protections blocking · waiting on 👀 reviews
🔴 Require two reviewer for test updatesWaiting for
This rule is failing.When test data is updated, we require two reviewers
Show 1 satisfied protection🟢 Enforce conventional commitMake sure that we follow https://www.conventionalcommits.org/en/v1.0.0/
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
cau-git
marked this pull request as ready for review
July 28, 2026 13:36
dolfim-ibm
reviewed
Jul 29, 2026
Comment on lines
+33
to
+36
| "doclang>=0.7,<0.8", | ||
| "typer>=0.15.1,<0.25.0", | ||
| "click>=8.0.0", | ||
| "lxml>=6.0.2", |
Member
There was a problem hiding this comment.
should we move this to a docling-core extra and just make the dlgrep depend on it?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
dlgrepis a grep-style CLI for DocLang documents. It searches semantic units — headings, paragraphs, list items, table cells, captions, formulas, code — rather than lines of text, and returns a reusable XPath address for every match, so results can be fed back intodlgrep show/dlgrep selector into downstream tooling.It was developed in a separate repository and depended on unreleased
docling-coreAPIs, which forced a git-branch dependency. This PR moves it in, and adds the deserializer APIs it needs.Changes to docling-core itself
DocLangSourceMapandDocLangSourceTargetare exported fromdocling_core.transforms.deserializer, letting a caller map deserialized document items back to their source XML nodes. This is what letsdlgrepreport XPath addresses.How the packaging works
dlgreplives atpackages/dlgrepas a uv workspace member and builds its owndlgrepwheel, published separately to PyPI. It is not merged into thedocling-corewheel anddocling-coredoes not depend on it — the dependency runs one way only.This follows the pattern already used in the
doclingrepo fordocling/docling-slim, with one difference:packages/doclingthere is a dependency-only meta-package, whereaspackages/dlgrepcarries real source.Versioning is lockstep.
dlgrepshares thedocling-coreversion and pins it exactly (docling-core==<same version>).dlgreptracksdocling-coreinternals closely enough that an exact pin is honest, and separate versioning would mean a second semantic-release tag stream, which PSR 7.x does not support without significant machinery. The practical consequence:dlgrepgets a release wheneverdocling-coredoes, whether or not it changed.Since the version number now implies a maturity
dlgrepdoes not have,dlgrep --helpstates that it is experimental and may break in any release.Release flow.
release.shwrites oneTARGET_VERSIONinto three places — the root version, thedlgrepversion, and thedocling-core==pin — then asserts they agree before anything is committed or tagged.build-packages.shbuilds both wheels from that single commit intodist/docling-core/anddist/dlgrep/, andpypi.ymlpublishesdocling-corefirst, thendlgrep.This matters for the cross-cutting case: when a change spans both packages, the
dlgreprelease pins exactly thedocling-corerelease produced in the same cycle. During development the pin is not consulted at all — the uv workspace source replaces it with the local path, sodlgreptests always run against the in-treedocling-core.Two guards, because uv cannot catch a wrong pin on its own (the workspace source drops the version specifier, so a mismatched pin locks and resolves cleanly):
.github/scripts/check_lockstep.pyasserts version/version/pin agreement. Runs in pre-commit, in the build script, and inrelease.sh.checks.ymlandpypi.ymlinstall both wheels together. A mismatched pin is unsatisfiable against the locally built pair, so it fails before publish rather than shipping an uninstallable wheel.Repo integration
git subtreedlgrepis in the dev group souv synccovers itdlgrep's tests and the lockstep check--all-packagesand smoke-tests the built CLIclickandlxmlare now declared instead of arriving transitively__version__reads from package metadata instead of a hardcoded literalValidation
pre-commit run --all-filespasses. Both wheels build, install together into a clean environment, and the CLI runs a real search against a.dclgfile.The failure paths were tested, not assumed: a deliberately mismatched pin fails
check_lockstep.py, abortsbuild-packages.sh, and is rejected at install withNo solution found when resolving dependencies— including when the wrongly named version does exist on PyPI.uvx dlgrepworks:dlgrepowns its console script, so no entry-point re-declaration is needed (unlikedocling→docling-slim). Verified against the built wheels.