[Question] DocTags parser for docling-core (Rust)?

[pjmalandrino]

We are keen to contribute a DocTags parser to docling.rs.

Our use case is edge compute, so the small VLM path is the one that matters most
to us: Granite-Docling emitting DocTags. On device, a single small model plus a
DocTags parser is a lot lighter than pdfium plus the ONNX layout stack.

We could not find a DocTags parser on the Rust side yet. The format produced by
Granite-Docling has no importer in docling-core. We already have a working
pure-Rust implementation (tolerant lexer plus recursive-descent parser, no native
deps, panic-free on real degenerate output) and are happy to upstream it,
retargeted to DoclingDocument.

Questions:

1. Is a DocTags to DoclingDocument parser in scope for docling-core (Rust)?
2. Would you accept a PR targeting the DoclingDocument model in crates/docling-core?
3. Any preference on placement (docling-core vs a separate crate) and target API?

[artiz]

Great timing — yes to all three, with some context on how this should land.

1. In scope? Definitely. As of VLM pipeline (remote OpenAI-compatible endpoint first) #77 (PR pdf: VLM pipeline — remote OpenAI-compatible vision endpoint (#77) #151) docling.rs has a remote-VLM pipeline that consumes granite-docling output. What it ships today is deliberately minimal: a lexical DocTags→DocLang bridge feeding the existing tolerant DocLang reader. That keeps structure (headings, OTSL tables, lists, furniture) but drops everything a real parser would keep: <loc_*> geometry (provenance/bounding boxes), OTSL span semantics beyond cell kinds, key-value regions, chart/picture classes, inline formatting. A dedicated tolerant parser is exactly the right next step, and your edge use case (small VLM, no pdfium/ONNX) is one we want to support first-class — the whole pdf-text/wasm path exists for the same reason.

2. PR against crates/docling-core? Yes. Ground rules that keep it mergeable:

Conformance-first: fixtures + tests comparing against Python docling-core's DocTagsDocument/load_from_doctags on shared inputs (we can wire that into scripts/conformance/). Tolerant-but-verified is the house style — every reader/serializer here is pinned byte-for-byte or by snapshot against upstream.
No native deps, wasm-safe: docling-core compiles for wasm32-unknown-unknown; pure-Rust + panic-free (as you describe) is exactly right. Prefer zero new dependencies; roxmltree won't help you anyway since DocTags isn't XML.
The usual repo hygiene: cargo fmt/clippy clean, dense why-comments, DCO sign-off on commits.
3. Placement & API: docling-core, as a doctags module next to the existing doclang serializer — not a separate crate (it mirrors Python docling-core, which hosts DocTagsDocument). Suggested surface, mirroring upstream:

// single fragment
pub fn parse(markup: &str) -> DoclingDocument;
// multi-page, mirroring DocTagsDocument.from_doctags_and_image_pairs:
pub fn parse_pages<'a>(pages: impl IntoIterator<Item = &'a str>) -> DoclingDocument;
(returning a document even for garbage input — collect recoverable issues in a side channel if you want, but never fail hard; VLM output is hostile by nature.)

Integration once it lands: the VLM pipeline in docling/src/vlm.rs swaps its lexical bridge for your parser (there are tests there with real granite-docling shapes — loc runs, unclosed OTSL, furniture — that should transfer directly into your test corpus), and .doctags/.dt files could become an InputFormat for free. Happy to review — open the PR against docling-project/docling.rs master.

[artiz]

Hey @pjmalandrino I tried to implement all the reuired logic in #153
Please check and feel free to ask any questions.

[pjmalandrino]

Thanks @artiz, that was fast !

The API is exactly what we hoped for, and parse_pages returning a DoclingDocument fits our multi-page ingestion directly.

We will run it against our edge corpus (real granite-docling output: loc runs,
unclosed OTSL, stamped-letter loops) and report back.

Two things we would like to contribute on top:

1. Figure/chart classification. Right now Picture sets classification: None and
   drops the class tokens (pie_chart, line_chart, logo, ...). For our analytics
   use case the chart type is signal, and you flagged it as a known drop. We
   have the mapping already and can send a small follow-up PR carrying the class
   into Node::Picture.classification.

2. Conformance fixtures. We have a corpus of real degenerate granite-docling
   outputs that should slot into scripts/conformance. Where would you like them?

[artiz]

Thanks @artiz, that was fast !

The API is exactly what we hoped for, and parse_pages returning a DoclingDocument fits our multi-page ingestion directly.

We will run it against our edge corpus (real granite-docling output: loc runs, unclosed OTSL, stamped-letter loops) and report back.

Two things we would like to contribute on top:

1. Figure/chart classification. Right now Picture sets classification: None and
   drops the class tokens (pie_chart, line_chart, logo, ...). For our analytics
   use case the chart type is signal, and you flagged it as a known drop. We
   have the mapping already and can send a small follow-up PR carrying the class
   into Node::Picture.classification.

Feel free to create PR againt this repo, that was main idea of migration project - to optimize docling performance on real uses cases, and actully VLM support was planned intitially.
On figure/chart classification: yes, please send it — and the good news is the model side already exists, so the PR should be tiny:

• Node::Picture.classification is Option<Vec<PictureClass>> with PictureClass { class_name, confidence } — the same field the DocumentFigureClassifier enrichment (feat: optional enrichment models (DocumentPictureClassifier, CodeFormula) #76) fills. The JSON serializer already emits it as docling's classification annotation on the picture item, so once you populate it, JSON output carries the class with zero serializer changes. Markdown/DocLang deliberately ignore it (docling parity).
• The place is parse_picture() in docling-core/src/doctags.rs: the _ => *i += 1 arm is where class tokens (<pie_chart>, <line_chart>, , …) currently die. Capture the first such token instead.
• Suggested mapping: PictureClass { class_name: token, confidence: 1.0 } — a single model-asserted prediction, which is how docling's own DocTags loader treats the class token (there's no distribution to preserve). Keep the token name verbatim rather than validating against the classifier's 26-label set: the vocabularies coincide today, and verbatim keeps the parser tolerant of tokens a future model adds ( included).
• For the conformance bar: one test asserting the parsed classification on a granite-style picture fragment, and ideally a JSON-output check against Python docling-core's load_from_doctags for the same fragment — the classification annotation should come out identical.

2. Conformance fixtures. We have a corpus of real degenerate granite-docling
   outputs that should slot into scripts/conformance. Where would you like them?

For this one please check scripts/conformance/vlm_conformance.sh and add source PDF/images to appropriate folders, or just update it to use separate files set

[artiz]

Hey @pjmalandrino and one more thing, have you tried ONNX engine that is delivered with docling.rs? On my side it outperforms all the external VLMs, especially being run on GPU.

[pjmalandrino]

Thanks, good pointer, and we have not benchmarked the ONNX pipeline yet. I take
your point that on GPU it beats the external VLMs.

For us the constraint is the deployment envelope more than raw accuracy: we
target on-device (eventually wasm/mobile), where the ONNX path pulls in ort plus
pdfium as native runtimes and there is usually no CUDA/DirectML GPU, so the "on
GPU" win does not fully transfer, and docling-pdf is not wasm-safe the way
docling-core is. The small VLM plus DocTags stays the portable path for that
tier, which is also why the parser matters to us.

For a desktop or server tier with a GPU it is clearly the stronger engine, so we
will likely keep both in mind. Out of curiosity, how does it do CPU-only (int8
models, no GPU) on your side? That number is what would decide it for our edge
target.

[artiz]

For us the constraint is the deployment envelope more than raw accuracy: we target on-device (eventually wasm/mobile), where the ONNX path pulls in ort plus pdfium...

There is already non-ONNX WASM module: https://github.com/docling-project/docling.rs/tree/master/crates/docling-wasm
For ONNX in browser I created #157 with top priority

For a desktop or server tier with a GPU it is clearly the stronger engine, so we will likely keep both in mind. Out of curiosity, how does it do CPU-only (int8 models, no GPU) on your side? That number is what would decide it for our edge target.

For non-ML formats performance uplift is 20–60 with ~60× less RAM, more detals: https://github.com/docling-project/docling.rs/blob/master/README.md#performance
For PDF on CPU with int8 quantized models - layout inference 2.4× faster; 1.83× end-to-end on a 1913-page PDF (0.74 → 0.40 s/page)

More info: https://github.com/docling-project/docling.rs/blob/master/docs/PDF_CONFORMANCE.md

[pjmalandrino]

Thanks for the quick answer and the insights I'll check it out !

[artiz]

Thanks for the quick answer and the insights I'll check it out !

I merged #157

Now ONNX is available in browser, tested on Samsung S22, it works like a charm