feat(dclx): DocLang OPC archive export + conformance harness (issue #32, part 1) by artiz · Pull Request #37 · docling-project/docling.rs

artiz · 2026-07-08T17:03:07Z

Summary

Part 1 of #32 — the .dclx (DocLang OPC archive, new in docling 2.110) export, from the OPC container through the hierarchical/rich-inline serialization DocLang requires.

export_to_doclang() (docling-core): DocLang 0.7 serializer reproducing docling's minidom.toprettyxml output directly.
dclx::save_as_dclx (docling): the OPC container, byte-for-byte as the Python doclang package writes it.
CLI --to dclx; scripts/gen_dclx.py (135 groundtruth archives); scripts/dclx_conformance.sh scoring.

Score progression

	non-PDF avg	byte-EXACT
initial	74%	22
this branch	80%	35

Per-format: csv 100, asciidoc 98, md 90, latex 88, xlsx 87, jats 86, html 83 (was 70), webvtt 81, docx 74 (was 69), uspto 73, odf 66, pptx 55. Target is 90% non-PDF / 50% PDF over 134 fixtures.

What landed

USPTO SGML/HTML entities — legacy pa/pg patents parse.
Code-block language label — <code> emits the DocLang <label> (Linguist) head + block form.
Structured inline model + serializer — InlineRun/Script, Node::InlineGroup/Node::Furniture, reproducing minidom's InlineGroup layout exactly; unit-tested against the Python reference.
HTML rich inline — paragraphs with inline structure become InlineGroups; <title> → furniture heading; DOM-walk runs carry <underline>/<subscript>/<superscript>. formatting.html 28% → 98%.
XML content routing — a JATS article saved as .txt routes to the XML backends by content.
DOCX rich inline — body paragraphs with formatting become InlineGroups (shared inline_paragraph_node).
XLSX table <location> — cell-grid bounding-box tokens, gated to image-free sheets. xlsx_02 byte-exact.
List items — <text> wrapping for items with a nested-list sibling; enumeration <marker>s (ListItem.marker, set by HTML <ol start> and DOCX numbered items, matching docling's backend distinction); adjacent sibling-list splitting. html_description_list 77→97, word_sample 82→97, example_01/02/03 and md/nested → byte-exact.

Markdown/JSON unchanged

All serializer additions are DocLang-only; the HTML/DOCX backends keep the exact former text as the group's md_text/list text. outputs_match_fixtures passes; no previously-exact .dclx fixture regressed across any step.

Known residuals (follow-ups)

PPTX/ODF slide grouping (55/66%) and DOCX/USPTO backend completeness are the largest remaining buckets — backend work, not serialization.
DOCX uniformly-formatted list items (docling stores them with direct formatting, not an inline group) and DOCX multilevel ordered markers need structure the flat model doesn't distinguish.
XLSX picture sheets / title-offset tables stay locationless / approximate until chart geometry and docling's exact table anchoring are ported.
PDF .dclx groundtruth generation is blocked on OCR-model hosts returning 403 from this environment.

🤖 Generated with Claude Code

https://claude.ai/code/session_017fKFbatf55BK5j8vr3dZjj

First cut of .dclx support — the DocLang OPC archive docling 2.110 introduced as an export format: - docling-core: export_to_doclang() — DocLang 0.7 XML serializer over the Node tree, reproducing the Python reference's minidom-pretty output directly (2-space indent, single-text-child inlining, CDATA glue lines, AUTO CDATA/<content> escaping). Inline bold/italic/code/ link markers baked into our text are re-parsed into the structural <bold>/<italic>/<code>/<href> elements DocLang expects; tables emit the OTSL cell stream (ched/fcel/ecel/nl). - docling: dclx::save_as_dclx/to_dclx_bytes — the OPC container ([Content_Types].xml + _rels/.rels byte-for-byte as the doclang package writes them, deflate, reference entry order). - CLI: --to dclx writes <input-stem>.dclx and prints the path. - scripts/gen_dclx.py: groundtruth generator over tests/data via the latest published docling (backend-direct for declarative formats, full pipeline with RapidOCR for PDF/images); 135 archives committed under tests/data/<fmt>/groundtruth_dclx/. - scripts/dclx_conformance.sh: per-fixture document.xml diff scoring with the issue #32 targets (>=90% non-PDF, >=50% PDF). Current score: non-PDF avg similarity 74% (19 fixtures byte-EXACT, incl. wiki.md/csv/uspto grants); PDF groundtruth generation is still blocked on the OCR-model host (403) — pending. Known gaps to close toward the 90% target: PPTX/ODF slide text grouping, USPTO pa/pg legacy schemas, list marker elements, code-language labels beyond the common set. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017fKFbatf55BK5j8vr3dZjj

Legacy USPTO archives (pap-v1, PATDOC grant-v2) reference ISO-8879 and HTML5 named entities — [, ™, &agr;, … — that roxmltree cannot resolve, aborting the parse with "unknown entity reference". Preprocess the source to expand recognized names into characters (folding the ISO Greek names onto their HTML5 counterparts, exactly as docling's skippedEntity handler does) and drop the unrecognized/graphics ones. The XML built-ins and numeric references pass through for the parser. Generated table (crates/docling/src/backend/uspto_entities.rs) is derived from Python's html.entities.html5 plus docling's ISO-8879 map. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017fKFbatf55BK5j8vr3dZjj

…tion `<code>` with a recognized fence language now emits the DocLang `<label value="…"/>` head (folding the raw fence onto the docling CodeLanguageLabel value, then onto its Linguist key) and switches to the block form, exactly as docling's AUTO label_mode does; unknown/absent languages stay inline and unlabeled. Also corrects the pretty-printer: minidom writes each node's indentation prefix and only suppresses the *newline* for glued text (CDATA/plain) children, so the closing tag after a glued CDATA lands indented on the same line (`…]]> </code>`). Emitting the indent unconditionally reproduces this; glue fragments carry depth 0 and contribute none. md/blocks.md is now byte-exact; no previously-exact fixture regressed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017fKFbatf55BK5j8vr3dZjj

CI runs clippy with -D warnings; drop the extra borrow on the parser input and rewrite the entity-lookup match as an if-let. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017fKFbatf55BK5j8vr3dZjj

Adds the docling-core foundation for rich DocLang inline content, without yet emitting it (no backend produces the new variants, so Markdown/JSON and conformance are unchanged): - `InlineRun` (bold/italic/underline/strikethrough/script/code) + `Script`, and two `Node` variants: `InlineGroup { unwrapped, runs, md_text }` and `Furniture(Box<Node>)`. - DocLang serializer for both, reproducing docling's `minidom.toprettyxml` layout exactly: the run-delimiter's leading newline pushes plain text to column 0 while the first `<text>` child stays indented; all-plain wrapped groups collapse to one inline text node; nested styles wrap outermost-last (script ⊃ strikethrough ⊃ underline ⊃ italic ⊃ bold); furniture injects `<layer value="furniture"/>`. Unit-tested against the Python reference. - Markdown/JSON treat `InlineGroup` as a paragraph of its `md_text` and skip `Furniture`, keeping their output byte-identical. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017fKFbatf55BK5j8vr3dZjj

Wires the structured inline path into the HTML backend so DocLang output matches docling's InlineGroup semantics, while Markdown/JSON stay byte-identical (they render the group's `md_text`, exactly the former paragraph text): - A paragraph with inline structure (two+ segments from link/format boundaries, or a single uniformly-formatted segment) becomes an `InlineGroup`; a plain paragraph or lone hyperlink stays a `Paragraph` (keeping the clean `<text>…</text>` / `<href>` forms). Multi-segment groups serialize unwrapped once a body heading precedes them. - The document `<title>` becomes a furniture-layer title heading (`<heading><layer value="furniture"/>…`), excluded from Markdown/JSON. - Structured runs are derived by re-parsing the Markdown (`***`/`**`/`*`/ `~~`/`` ` ``/`[t](u)`) via the new `inline_runs_from_markdown`. Non-PDF conformance 74% → 77%, byte-exact fixtures 25 → 28 (hyperlink_01/ 06 and more now exact); no previously-exact fixture regressed. Underline and sub/superscript have no Markdown marker and remain the known residual. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017fKFbatf55BK5j8vr3dZjj

Builds the structured runs during the HTML DOM walk (rather than re-parsing Markdown), so DocLang now carries the formatting Markdown drops entirely — `<underline>`, `<subscript>`, `<superscript>` — with docling's run segmentation. A ` ` folds the next same-formatting segment into the previous run (`a b` → one item `"a\nb"`), matching docling. A `<content>`-wrapped text body now renders in block form (element child), fixing paragraphs with soft line breaks. Also content-routes XML documents that arrived with a text/Markdown extension (a JATS article saved as `.txt`) to the XML backends, mirroring docling's content detection. Markdown/JSON output is unchanged (the `md` run strings and finalize path are untouched; the structured runs are DocLang-only). formatting.html 73% → 98%; non-PDF conformance 77% → 78%, byte-exact fixtures 28 → 29; no regressions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017fKFbatf55BK5j8vr3dZjj

Reuses the InlineGroup path for the DOCX backend: a body paragraph with inline formatting now becomes an `InlineGroup` (built from docling's `_get_paragraph_elements` run grouping, carrying bold/italic/strikethrough/ underline/sub-/superscript), a plain paragraph stays a `Paragraph`. The docx body is flat (docling parents these on the body group), so the group is always wrapped. The InlineGroup decision is now a shared `docling_core::inline_paragraph_node` helper used by both the HTML and DOCX backends. Markdown/JSON output is unchanged (they render the group's `md_text`). docx avg similarity 69% → 72%; no regressions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017fKFbatf55BK5j8vr3dZjj

Adds the table-level DocLang `<location>` tokens (x0,y0,x1,y1) for the spreadsheet backend. Each table's cell-index bounding box is normalized against the sheet extent and quantized to the 0–511 grid, matching docling's `_create_location_tokens_for_bbox` + `_quantize_to_resolution`. `Node`'s `Table` gains an optional `location`; declarative backends leave it `None` (no coordinates), so their output is unchanged. Because docling's page extent also counts images/charts — whose geometry we don't parse — we set locations only for image-free sheets (where the tables-only extent is exact) and leave them off otherwise. xlsx avg similarity 85% → 87% (xlsx_02 now byte-exact, xlsx_06 99%); no regressions; Markdown/JSON unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017fKFbatf55BK5j8vr3dZjj

docling wraps a list item's content in `<text>` when it has a "segment sibling" — a nested list following it — and otherwise emits it bare. The serializer now detects the nested-list case (the next flat item is deeper) and wraps accordingly: a plain item as `<text>text</text>`, a formatted one as its InlineGroup. Bare items are unchanged. Big gains across list-heavy fixtures: html_description_list 77% → 97%, md/nested 84% → 98%, md/mixed_without_h1 → exact, docx_lists 72% → 78%, hyperlink_03 68% → 86%. non-PDF 78% → 79%, byte-exact 29 → 31; no previously-exact fixture regressed; Markdown/JSON unchanged. A uniformly-formatted item that docling stores with direct formatting (rather than an inline group) is also wrapped upstream, but that backend-structural distinction isn't recoverable from the flat node model. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017fKFbatf55BK5j8vr3dZjj

Adds `marker: Option<String>` to `Node::ListItem` so backends can pass the DocLang enumeration marker, which the serializer emits as `<ldiv><marker>N.</marker></ldiv>` (bare `<ldiv/>` when absent) — matching docling, whose list serializer emits `<marker>` only for a non-empty marker. Only the backends that provide one set it, exactly as docling does: - HTML: an ordered list with an explicit `start` attribute (`<ol start=…>`); a plain `<ol>` carries none. - DOCX: enumerated (numbered) items, from the computed enum marker. Markdown and the other declarative backends leave it `None`, so their `.dclx` (and Markdown/JSON) output is unchanged. word_sample 82% → 97%, docx_lists 78% → 89%, example_01 80% → 93%, textbox 76% → 80%; docx avg 73% → 74%. No previously-exact fixture regressed; Markdown/JSON unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017fKFbatf55BK5j8vr3dZjj

The list serializer now closes and reopens `<list>` at a sibling boundary — the backend flagged a fresh list (`first_in_list`), the kind flips (`<ul>`↔`<ol>`), or an ordered run breaks — mirroring the Markdown serializer's heuristic. Previously all consecutive same-level items merged into one list, mis-typing adjacent lists. With the enumeration markers in place this is a clean win: example_01/02/03 and md/nested reach byte-exact, docx_lists 89% → 91%, latex/example_01 90% → 95%. non-PDF 79% → 80%, byte-exact 31 → 35; no regressions; Markdown/JSON unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017fKFbatf55BK5j8vr3dZjj

docling emits an empty `<text></text>` (and JSON text item) for every blank body paragraph (`skip_empty_text=False`); paragraphs carrying a drawing skip it. The DOCX backend now emits an empty `Node::Paragraph` for such paragraphs, the DocLang serializer renders it as `<text></text>`, and the JSON export as an empty text item — matching docling. Markdown drops them. Empty paragraphs interspersed between list items are absorbed into the list run by all three serializers, so a ListGroup stays contiguous (docling keeps empty text items outside the list) — no spurious list splits. Big DOCX gains: lorem_ipsum/unit_test_headers/tablecell/table_with_equations reach byte-exact, equations 46% → 60%, drawingml 64% → 80%, textbox 80% → 89%. non-PDF byte-exact 35 → 39; no regressions. Markdown unchanged; JSON now carries the empty text items (expected snapshots regenerated). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017fKFbatf55BK5j8vr3dZjj

…ME DocLang Reorganize the flat scripts/ directory into topical subfolders: conformance/ compare, conformance, dclx_conformance, pdf_conformance, pdf_groundtruth, gen_dclx, docling_convert, setup-docling ci/ bump_version, ci_publish, release install/ install, download_dependencies, pdf_setup, quantize_models, export_tableformer, export_layout test/ performance, parse_vs_ours, parser_completeness, dump_parse_cells, dump_word_cells, docling_dump_all _common.sh stays at scripts/ root. Fixes all intra-script path derivation (../_common.sh sourcing, one-level-deeper repo-root cd, PY_RUNNER into conformance/, Python REPO/ROOT parents), cross-script invocations (release -> ci/, install -> install/, pdf_setup -> install/, gen_dclx -> conformance/docling_convert.py), and every external reference in workflows, docs, Rust comments and Cargo.toml. Also document DocLang / `.dclx` output in the README: export_to_doclang(), docling::dclx::save_as_dclx, the CLI `--to dclx`, and the conformance tooling under scripts/conformance/. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017fKFbatf55BK5j8vr3dZjj

Two conformance bugs in the USPTO backend, both visible on ipa20110039701 / ipa20180000016: 1. raw_text() read text only via node.text() (first child) and element.tail(), so any run following a non-element node was dropped. In USPTO `<?in-line-formulae?>R1—CO…` the leading "R" is the processing instruction's tail — it (and in some cases a whole formula paragraph like "(Ca,Sr)S:Eu (V)") vanished. Rewrite raw_text to walk children in order and emit text-node children directly, which also captures text after PIs/comments. 2. The abstract emitted one paragraph per ``, but docling joins the abstract's text into a single text item (dropping the chemistry-drawing `` as empty). Join the paragraphs with a space. Every restored fragment now matches docling's `.dclx` groundtruth. Adds unit tests for both and regenerates the two affected snapshots. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017fKFbatf55BK5j8vr3dZjj

claude added 15 commits July 8, 2026 17:02

This was referenced Jul 9, 2026

DocLang: raise PPTX conformance to ≥90% (currently ~55%) #39

Closed

DocLang: raise ODF conformance to ≥90% (currently ~66%) #40

Closed

artiz merged commit 71dd1d8 into master Jul 9, 2026
3 checks passed

artiz deleted the claude/issue-32-dclx-conformance branch July 9, 2026 09:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(dclx): DocLang OPC archive export + conformance harness (issue #32, part 1) - #37

feat(dclx): DocLang OPC archive export + conformance harness (issue #32, part 1)#37
artiz merged 15 commits into
masterfrom
claude/issue-32-dclx-conformance

artiz commented Jul 8, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

artiz commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Score progression

What landed

Markdown/JSON unchanged

Known residuals (follow-ups)

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

artiz commented Jul 8, 2026 •

edited

Loading