From d37f80025c007580da3f020b20fa128e135b8a39 Mon Sep 17 00:00:00 2001 From: artiz Date: Thu, 23 Jul 2026 17:15:20 +0000 Subject: [PATCH 1/3] conformance: VLM corpus-comparison harness (#153) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The follow-up #77 left open: measure docling.rs's --pipeline vlm against Python docling's VlmPipeline over the PDF corpus. scripts/conformance/vlm_conformance.sh — drives both sides against the SAME OpenAI-compatible endpoint (the granite shim; llama.cpp-family servers strip the DocTags structure tokens, see #77) so model and server are held constant and diffs isolate rendering/parsing/assembly. Reports per-fixture whitespace-normalized character similarity + byte-exactness and a corpus mean; outputs cached under target/vlm-conformance/ for triage (delete to regenerate — each side costs minutes of GPU per fixture). A measurement, not a gate: always exits 0 with numbers for PDF_CONFORMANCE.md. FIXTURES env restricts the corpus subset. scripts/conformance/vlm_convert.py — the reference side: docling's VlmPipeline via ApiVlmOptions (DOCTAGS response format, temperature 0, import-path compatible across docling minor versions). granite_vlm_server.py grows corpus-run ergonomics: a persistent TRITON_CACHE_DIR (kernel recompilation dominated per-page latency in the #77 bring-up) and --warmup to pay first-request compilation before the corpus starts. docs/PDF_CONFORMANCE.md: "VLM pipeline conformance" section marking the measurement pending and naming the known render-scale asymmetry so drift triage starts from the right bucket. Refs #153 Signed-off-by: artiz Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT --- docs/PDF_CONFORMANCE.md | 12 ++++ scripts/conformance/vlm_conformance.sh | 92 ++++++++++++++++++++++++++ scripts/conformance/vlm_convert.py | 74 +++++++++++++++++++++ scripts/dev/granite_vlm_server.py | 30 +++++++++ 4 files changed, 208 insertions(+) create mode 100755 scripts/conformance/vlm_conformance.sh create mode 100755 scripts/conformance/vlm_convert.py diff --git a/docs/PDF_CONFORMANCE.md b/docs/PDF_CONFORMANCE.md index 1c838cd4..14ba6ca0 100644 --- a/docs/PDF_CONFORMANCE.md +++ b/docs/PDF_CONFORMANCE.md @@ -98,6 +98,18 @@ bounded by these, so raising it further is a model problem, not a serialization one: every laid-out block kind now carries provenance, so the ±2 figure sits at the geometry-ignored ceiling. +## VLM pipeline conformance (#153 — measurement pending) + +The remote-VLM pipeline (#77, `--pipeline vlm`) has its own comparison +harness: `scripts/conformance/vlm_conformance.sh` runs the PDF corpus through +docling.rs *and* Python docling's `VlmPipeline`, both against the same +endpoint (`scripts/dev/granite_vlm_server.py` — the only server class that +keeps granite-docling's DocTags tokens intact), and reports per-fixture +whitespace-normalized similarity plus byte-exactness. Requires a GPU for +sane runtimes; numbers land here once the corpus run happens. Known accepted +asymmetry: each side renders pages at its own scale, so some drift is +render-induced rather than parser-induced — triage before attributing. + ## Enrichment models (opt-in) docling's optional enrichment stages are ported behind the same flags diff --git a/scripts/conformance/vlm_conformance.sh b/scripts/conformance/vlm_conformance.sh new file mode 100755 index 00000000..5f7bbff0 --- /dev/null +++ b/scripts/conformance/vlm_conformance.sh @@ -0,0 +1,92 @@ +#!/usr/bin/env bash +# VLM corpus comparison (#153): run the PDF corpus through docling.rs's +# `--pipeline vlm` AND Python docling's VlmPipeline — both against the SAME +# OpenAI-compatible endpoint (scripts/dev/granite_vlm_server.py keeps +# granite-docling's DocTags tokens intact; llama.cpp-family servers strip +# them) — and report per-fixture similarity between the two Markdown outputs. +# +# This is a measurement, not a gate: model output legitimately varies across +# checkpoints/servers, so the script always exits 0 and prints numbers for +# docs/PDF_CONFORMANCE.md. Outputs are kept under target/vlm-conformance/ for +# triage (rust/ vs python/ per fixture). +# +# DOCLING_RS_VLM_ENDPOINT=http://localhost:8000/v1 (default) +# DOCLING_RS_VLM_MODEL=granite-docling (default) +# FIXTURES="a.pdf b.pdf" restrict to specific corpus files (basenames) +# +# Prereqs: the shim serving (see its header for the venv recipe), pdfium for +# the Rust side, and Python docling with VLM extras for the reference side +# (scripts/conformance/setup-docling.sh). +set -euo pipefail +cd "$(dirname "$0")/../.." # docling.rs/ + +ENDPOINT="${DOCLING_RS_VLM_ENDPOINT:-http://localhost:8000/v1}" +MODEL="${DOCLING_RS_VLM_MODEL:-granite-docling}" +OUT="target/vlm-conformance" +export PDFIUM_DYNAMIC_LIB_PATH="${PDFIUM_DYNAMIC_LIB_PATH:-$(pwd)/.pdfium/lib}" + +# Reachable? Any HTTP status will do (the shim only answers POST); only a +# connection failure means "no server". +if ! curl -s -o /dev/null --max-time 5 -X POST "$ENDPOINT/chat/completions" -d '{}'; then + echo "no VLM endpoint at $ENDPOINT — start scripts/dev/granite_vlm_server.py first" >&2 + exit 1 +fi +python3 -c "import docling" 2>/dev/null || { + echo "python docling not importable — run scripts/conformance/setup-docling.sh" >&2 + exit 1 +} + +[ -x target/release/docling-rs ] || cargo build --release -q -p docling-cli +mkdir -p "$OUT/rust" "$OUT/python" + +# Whitespace-normalized character-level similarity of two Markdown files, +# 0–100 (line-level scoring would zero a whole paragraph over one differing +# word). Model output is not byte-stable across renderers, so exactness is +# reported separately. +similarity() { + python3 - "$1" "$2" <<'EOF' +import difflib, re, sys +def norm(p): + text = open(p, encoding="utf-8").read() + return "\n".join( + re.sub(r"\s+", " ", l).strip() for l in text.splitlines() if l.strip() + ) +a, b = norm(sys.argv[1]), norm(sys.argv[2]) +print(f"{difflib.SequenceMatcher(a=a, b=b, autojunk=False).ratio() * 100:.1f}") +EOF +} + +total=0; exact=0; sum="0" +printf "%-38s %8s %s\n" "fixture" "sim%" "exact" +for src in tests/data/pdf/sources/*.pdf; do + name="$(basename "$src")" + if [ -n "${FIXTURES:-}" ] && ! grep -qw "$name" <<<"$FIXTURES"; then + continue + fi + rust_md="$OUT/rust/$name.md" + py_md="$OUT/python/$name.md" + # Cached across reruns (each side costs minutes of GPU time per fixture); + # delete target/vlm-conformance/ to force regeneration. + if [ ! -s "$rust_md" ]; then + ./target/release/docling-rs --pipeline vlm --no-stream \ + --vlm-endpoint "$ENDPOINT" --vlm-model "$MODEL" "$src" > "$rust_md" \ + || { echo " rust side failed on $name" >&2; rm -f "$rust_md"; continue; } + fi + if [ ! -s "$py_md" ]; then + python3 scripts/conformance/vlm_convert.py \ + --endpoint "$ENDPOINT" --model "$MODEL" "$src" "$py_md" \ + || { echo " python side failed on $name" >&2; rm -f "$py_md"; continue; } + fi + sim="$(similarity "$py_md" "$rust_md")" + is_exact="no" + diff -q "$py_md" "$rust_md" >/dev/null 2>&1 && { is_exact="yes"; exact=$((exact + 1)); } + printf "%-38s %8s %s\n" "$name" "$sim" "$is_exact" + total=$((total + 1)) + sum="$(python3 -c "print($sum + $sim)")" +done + +if [ "$total" -gt 0 ]; then + mean="$(python3 -c "print(f'{$sum / $total:.1f}')")" + echo "VLM corpus similarity (rust vs python docling, $MODEL): mean $mean% over $total fixtures, $exact byte-exact" + echo "outputs kept in $OUT/ for triage" +fi diff --git a/scripts/conformance/vlm_convert.py b/scripts/conformance/vlm_convert.py new file mode 100755 index 00000000..6c9615f4 --- /dev/null +++ b/scripts/conformance/vlm_convert.py @@ -0,0 +1,74 @@ +#!/usr/bin/env python3 +"""Convert a PDF to Markdown through Python docling's **VlmPipeline** (#153). + +The reference side of the VLM corpus comparison: docling drives the *same* +OpenAI-compatible endpoint the Rust `--pipeline vlm` uses (ApiVlmOptions, +DOCTAGS response format), so the model and server are held constant and the +diff isolates what #153 is after — page rendering, DocTags parsing and +document assembly differences between the two implementations. (One known, +accepted asymmetry: each side renders pages with its own scale/DPI, which can +nudge the model's output; triage such diffs as "render", not "parser".) + +Usage: + vlm_convert.py --endpoint http://localhost:8000/v1 --model granite-docling \ + [output.md] +""" + +import argparse +import sys +from pathlib import Path + +from docling.datamodel.base_models import InputFormat + +# ApiVlmOptions moved between docling minor versions; support both homes. +try: + from docling.datamodel.pipeline_options_vlm_model import ( + ApiVlmOptions, + ResponseFormat, + ) +except ImportError: # older docling + from docling.datamodel.pipeline_options import ApiVlmOptions, ResponseFormat + +from docling.datamodel.pipeline_options import VlmPipelineOptions +from docling.document_converter import DocumentConverter, PdfFormatOption +from docling.pipeline.vlm_pipeline import VlmPipeline + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--endpoint", required=True, help="…/v1 base or full …/chat/completions URL") + ap.add_argument("--model", required=True) + ap.add_argument("--max-tokens", type=int, default=8192) + ap.add_argument("input") + ap.add_argument("output", nargs="?") + args = ap.parse_args() + + url = args.endpoint.rstrip("/") + if not url.endswith("/chat/completions"): + url = f"{url}/chat/completions" + + vlm_options = ApiVlmOptions( + url=url, + params={"model": args.model, "max_tokens": args.max_tokens, "temperature": 0}, + prompt="Convert this page to docling.", + timeout=600, + response_format=ResponseFormat.DOCTAGS, + ) + pipeline_options = VlmPipelineOptions(vlm_options=vlm_options, enable_remote_services=True) + converter = DocumentConverter( + format_options={ + InputFormat.PDF: PdfFormatOption( + pipeline_cls=VlmPipeline, pipeline_options=pipeline_options + ) + } + ) + md = converter.convert(Path(args.input)).document.export_to_markdown() + if args.output: + Path(args.output).write_text(md + ("\n" if not md.endswith("\n") else "")) + else: + sys.stdout.write(md) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/dev/granite_vlm_server.py b/scripts/dev/granite_vlm_server.py index d2dfa153..66caa641 100644 --- a/scripts/dev/granite_vlm_server.py +++ b/scripts/dev/granite_vlm_server.py @@ -133,8 +133,38 @@ def main(): ap.add_argument("--port", type=int, default=8000) ap.add_argument("--model", default=DEFAULT_MODEL) ap.add_argument("--cpu", action="store_true", help="force CPU inference") + ap.add_argument( + "--warmup", + action="store_true", + help="run one dummy generation at startup so the first real page " + "doesn't pay the kernel-compilation cost (useful before corpus runs)", + ) args = ap.parse_args() + # Persist triton's JIT kernel cache across restarts — without it every + # server start (and on some setups every request) recompiles, which + # dominated per-page latency in the #77 bring-up. + import os + + os.environ.setdefault( + "TRITON_CACHE_DIR", + os.path.expanduser("~/.cache/docling-rs/triton"), + ) Handler.processor, Handler.model, Handler.device = load(args.model, args.cpu) + if args.warmup: + from PIL import Image + + buf = io.BytesIO() + Image.new("RGB", (64, 64), "white").save(buf, format="PNG") + started = time.time() + generate( + Handler.processor, + Handler.model, + Handler.device, + buf.getvalue(), + "Convert this page to docling.", + 8, + ) + print(f"warmup done in {time.time() - started:.1f}s", flush=True) print(f"serving on http://127.0.0.1:{args.port}/v1/chat/completions", flush=True) HTTPServer(("127.0.0.1", args.port), Handler).serve_forever() From 75a13d44694c5f28f65de2d5d0880e12d4b833c0 Mon Sep 17 00:00:00 2001 From: artiz Date: Thu, 23 Jul 2026 17:28:37 +0000 Subject: [PATCH 2/3] core: DocTags parser in docling-core; VLM pipeline switches to it (#152) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The DocTags→DoclingDocument parser #152 asks about, as a docling-core module (mirroring Python docling-core, which hosts DocTagsDocument): a tolerant lexer + recursive-descent walk over the token stream the docling VLMs emit. DocTags is not XML — loc tokens, OTSL cell markers, picture classes and checkboxes are unclosed, captions nest inside , and generation truncates mid-element — so the grammar is parsed natively instead of being lexically rewritten into DocLang (#77's interim bridge, now deleted): - runs become real layout provenance: Node::Located wrappers and per-node locations (table/list/formula), which the lexical bridge threw away; - OTSL tables keep their span/header structure (TableStructure: ched/rhed headers, lcel/ucel/xcel continuations), not just the text grid, with the in-otsl caption emitted as a paragraph before the table (docling reading order); - section_header_level_K → heading K+1, title → 1 (docling Markdown parity, pinned against the pg9 groundtruth in #77); - page_header/page_footer → Node::PageFurniture (Markdown/JSON omit); - code recovers its <_lang_> token as the language; formulas map to Node::Formula; - everything unrecognized degrades: unknown tokens are transparent, stray block-level text becomes paragraphs (the token-stripping-server shape from the #77 bring-up), truncated elements stop at the next block start instead of swallowing the page. parse() never fails. API: doctags::parse(&str) / parse_pages(iter) — pure Rust, no new dependencies, panic-free, wasm32-safe (checked). docling/src/vlm.rs: fragments now route by grammar — DocTags-shaped answers (loc/otsl/section_header markers) go to the new parser, DocLang XML keeps the existing reader path, untagged prose keeps the line-per-paragraph fallback. The interim DocTags→DocLang lexical translator and its tests are gone; unit tests cover the routing + wrapper stripping, docling-core tests pin the full markup semantics (real granite shapes: spans, truncation, degraded loc+text, entities), and a new mock-server e2e exercises the DocTags route end to end. Refs #152, #77 Signed-off-by: artiz Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT --- crates/docling-core/src/doctags.rs | 690 +++++++++++++++++++++++++++++ crates/docling-core/src/lib.rs | 1 + crates/docling/src/vlm.rs | 380 ++++------------ crates/docling/tests/vlm.rs | 20 + 4 files changed, 796 insertions(+), 295 deletions(-) create mode 100644 crates/docling-core/src/doctags.rs diff --git a/crates/docling-core/src/doctags.rs b/crates/docling-core/src/doctags.rs new file mode 100644 index 00000000..c96f11d4 --- /dev/null +++ b/crates/docling-core/src/doctags.rs @@ -0,0 +1,690 @@ +//! DocTags reader (issues #152 / #77): the token markup docling's VLMs +//! (granite-docling / SmolDocling) emit, parsed into a [`DoclingDocument`]. +//! +//! DocTags is **not XML** — it is a stream of special tokens interleaved with +//! text: `` location tokens and OTSL table-cell markers +//! (`Text…`) are unclosed, captions nest inside ``, +//! and a model under sampling pressure emits stray text, truncated elements +//! and unknown tokens. This module is therefore a tolerant lexer plus a +//! recursive-descent walk: every recognized structure maps onto the document +//! model (headings, paragraphs, lists, OTSL tables *with* span structure, +//! pictures, page furniture, code, formulas), `` runs become layout +//! provenance ([`Node::Located`] / per-node locations — DocTags' 0–500 grid +//! is carried as-is, matching the 0–511 DocLang convention closely enough +//! for provenance), and anything unrecognized degrades to text or is skipped +//! — never an error. Garbage in, best-effort document out: VLM output is +//! hostile by nature, and a parser that fails hard would fail every second +//! page. +//! +//! docling-parity choices, pinned by the VLM pipeline's live tests against +//! real granite-docling output: +//! - `` → heading level `K+1` (`#` is the document +//! title, sections start at `##` — matches docling's Markdown export); +//! - an in-`` `` becomes a paragraph *before* the table +//! (docling's reading order); +//! - ``/`` become [`Node::PageFurniture`], which +//! the Markdown/JSON exports omit like every other furniture. + +use crate::document::{DoclingDocument, Node, Table, TableStructure}; + +/// Parse one DocTags fragment (typically one page's model output). +pub fn parse(markup: &str) -> DoclingDocument { + parse_pages([markup]) +} + +/// Parse multiple per-page fragments into one document, mirroring Python +/// docling-core's `DocTagsDocument.from_doctags_and_image_pairs` + +/// `DoclingDocument.load_from_doctags` (sans images). A [`Node::PageBreak`] +/// separates pages in the model; Markdown/JSON exports omit it. +pub fn parse_pages<'a>(pages: impl IntoIterator) -> DoclingDocument { + let mut doc = DoclingDocument::new("doctags"); + let mut first = true; + for page in pages { + if !first { + doc.nodes.push(Node::PageBreak); + } + first = false; + let toks = lex(page); + let mut i = 0; + parse_blocks(&toks, &mut i, &mut doc.nodes, 0); + } + doc +} + +// --------------------------------------------------------------------------- +// Lexer. +// --------------------------------------------------------------------------- + +#[derive(Debug, Clone, PartialEq)] +enum Tok<'a> { + /// `` (attributes, if any, are irrelevant to DocTags and dropped). + Open(&'a str), + /// `` + Close(&'a str), + /// `` + Loc(u16), + /// Raw text between tokens (entities decoded). + Text(String), +} + +/// Split the markup into tags and text. A `<` that doesn't start a plausible +/// tag is literal text (`a < b`). +fn lex(markup: &str) -> Vec> { + let mut toks = Vec::new(); + let mut rest = markup; + let mut text = String::new(); + while let Some(lt) = rest.find('<') { + let (before, tag_on) = rest.split_at(lt); + text.push_str(before); + let Some(gt) = tag_on.find('>') else { + text.push('<'); + rest = &tag_on[1..]; + continue; + }; + let inner = tag_on[1..gt].trim(); + let close = inner.starts_with('/'); + let name = inner + .trim_start_matches('/') + .trim_end_matches('/') + .split_whitespace() + .next() + .unwrap_or_default(); + if name.is_empty() + || !name.chars().next().is_some_and(|c| c.is_ascii_alphabetic()) + || !name.chars().all(|c| c.is_ascii_alphanumeric() || c == '_') + || inner.contains('<') + { + // Not a tag (`a < b`, or a stray `<` right before a real tag): + // the `<` is literal text. + text.push('<'); + rest = &tag_on[1..]; + continue; + } + if !text.trim().is_empty() { + toks.push(Tok::Text(decode_entities(text.trim()))); + } + text.clear(); + if let Some(v) = name + .strip_prefix("loc_") + .and_then(|v| v.parse::().ok()) + { + toks.push(Tok::Loc(v)); + } else if close { + toks.push(Tok::Close(name)); + } else { + toks.push(Tok::Open(name)); + } + rest = &tag_on[gt + 1..]; + } + if !rest.trim().is_empty() { + text.push_str(rest); + } + if !text.trim().is_empty() { + toks.push(Tok::Text(decode_entities(text.trim()))); + } + toks +} + +/// The model may emit XML-style entities or raw characters; accept both. +fn decode_entities(s: &str) -> String { + if !s.contains('&') { + return s.to_string(); + } + s.replace("&", "&") + .replace("<", "<") + .replace(">", ">") + .replace(""", "\"") + .replace("'", "'") + .replace("'", "'") +} + +// --------------------------------------------------------------------------- +// Parser. +// --------------------------------------------------------------------------- + +/// Element names that start a block — inline text collection stops there so a +/// missing closer (truncated generation) doesn't swallow the next block. +fn is_block_start(name: &str) -> bool { + matches!( + name, + "text" + | "paragraph" + | "title" + | "caption" + | "footnote" + | "code" + | "formula" + | "otsl" + | "picture" + | "chart" + | "ordered_list" + | "unordered_list" + | "list_item" + | "page_header" + | "page_footer" + | "page_break" + | "doctag" + | "doctags" + ) || name.starts_with("section_header_level_") +} + +/// Wrap `node` in [`Node::Located`] when a `loc_l,t,r,b` run was collected. +fn located(node: Node, loc: Option<[u16; 4]>) -> Node { + match loc { + Some(location) => Node::Located { + location, + inner: Box::new(node), + }, + None => node, + } +} + +/// Collect an element's inline content: text joins with spaces, the first +/// four `` become its location, unknown tokens are transparent. +/// Consumes the matching closer when present; stops (without consuming) at a +/// block-start tag, so truncated output loses only its own element. +fn collect_inline(toks: &[Tok], i: &mut usize, name: &str) -> (String, Option<[u16; 4]>) { + let mut text = String::new(); + let mut locs: Vec = Vec::new(); + while *i < toks.len() { + match &toks[*i] { + Tok::Close(n) if *n == name => { + *i += 1; + break; + } + Tok::Open(n) if is_block_start(n) => break, + Tok::Close(n) if is_block_start(n) => break, + Tok::Loc(v) => { + if locs.len() < 4 { + locs.push(*v); + } + *i += 1; + } + Tok::Text(t) => { + if !text.is_empty() { + text.push(' '); + } + text.push_str(t); + *i += 1; + } + // Unknown inline token (checkbox states, styling a future model + // might emit): transparent. + Tok::Open(_) | Tok::Close(_) => *i += 1, + } + } + let loc = (locs.len() == 4).then(|| [locs[0], locs[1], locs[2], locs[3]]); + (text, loc) +} + +fn parse_blocks(toks: &[Tok], i: &mut usize, out: &mut Vec, list_level: u8) { + // Location tokens seen at block level (the degraded no-structure shape + // servers produce when they strip special tokens) attach to the next + // bare-text paragraph. + let mut pending_loc: Vec = Vec::new(); + while *i < toks.len() { + match &toks[*i] { + Tok::Loc(v) => { + if pending_loc.len() < 4 { + pending_loc.push(*v); + } + *i += 1; + } + Tok::Text(t) => { + // Stray block-level text (no wrapping element): a paragraph. + let loc = (pending_loc.len() == 4).then(|| { + [ + pending_loc[0], + pending_loc[1], + pending_loc[2], + pending_loc[3], + ] + }); + pending_loc.clear(); + out.push(located(Node::Paragraph { text: t.clone() }, loc)); + *i += 1; + } + Tok::Close(_) => { + // Stray closer (ours ended already, or the model's nesting is + // off): skip. Returning here would drop the rest of the page. + *i += 1; + } + Tok::Open(name) => { + let name = *name; + pending_loc.clear(); + *i += 1; + if let Some(level) = name.strip_prefix("section_header_level_") { + let level: u8 = level.parse().unwrap_or(1); + let (text, loc) = collect_inline(toks, i, name); + if !text.is_empty() { + out.push(located( + Node::Heading { + // docling parity: section level K renders as + // heading K+1 ("#" is the document title). + level: level.saturating_add(1).clamp(2, 6), + text, + }, + loc, + )); + } + continue; + } + match name { + "doctag" | "doctags" => {} + "page_break" => out.push(Node::PageBreak), + "title" => { + let (text, loc) = collect_inline(toks, i, name); + if !text.is_empty() { + out.push(located(Node::Heading { level: 1, text }, loc)); + } + } + "text" | "paragraph" | "footnote" | "caption" => { + let (text, loc) = collect_inline(toks, i, name); + if !text.is_empty() { + out.push(located(Node::Paragraph { text }, loc)); + } + } + "page_header" | "page_footer" => { + let (text, loc) = collect_inline(toks, i, name); + if !text.is_empty() { + out.push(Node::PageFurniture { + footer: name == "page_footer", + location: loc.unwrap_or([0; 4]), + text, + }); + } + } + "code" => { + let (mut text, loc) = collect_inline(toks, i, name); + // DocTags carries the language as a `<_lang_>` token, + // which the lexer keeps as literal text (a leading + // underscore is no tag name); peel it off here. + let mut language = None; + if let Some(rest) = text.strip_prefix("<_") { + if let Some((lang, body)) = rest.split_once("_>") { + language = Some(lang.to_string()).filter(|l| !l.is_empty()); + text = body.trim_start().to_string(); + } + } + if !text.is_empty() { + out.push(located( + Node::Code { + language, + text, + orig: None, + }, + loc, + )); + } + } + "formula" => { + let (latex, loc) = collect_inline(toks, i, name); + if !latex.is_empty() { + out.push(Node::Formula { + orig: latex.clone(), + latex, + location: loc, + }); + } + } + "ordered_list" | "unordered_list" => { + parse_list(toks, i, out, name == "ordered_list", list_level); + } + // A bare item outside a list wrapper (truncated output). + "list_item" => { + let (text, loc) = collect_inline(toks, i, name); + if !text.is_empty() { + out.push(Node::ListItem { + ordered: false, + number: 1, + first_in_list: true, + text, + level: list_level, + marker: None, + location: loc, + dclx: None, + href: None, + layer: None, + }); + } + } + "otsl" => parse_otsl(toks, i, out), + "picture" | "chart" => parse_picture(toks, i, out, name), + // Unknown block token: transparent (its text will surface + // as stray block-level text above). + _ => {} + } + } + } + } +} + +fn parse_list(toks: &[Tok], i: &mut usize, out: &mut Vec, ordered: bool, level: u8) { + let close = if ordered { + "ordered_list" + } else { + "unordered_list" + }; + let mut number: u64 = 0; + let mut first = true; + while *i < toks.len() { + match &toks[*i] { + Tok::Close(n) if *n == close => { + *i += 1; + break; + } + Tok::Open("list_item") => { + *i += 1; + let (text, loc) = collect_inline(toks, i, "list_item"); + if !text.is_empty() { + number += 1; + out.push(Node::ListItem { + ordered, + number, + first_in_list: std::mem::take(&mut first), + text, + level, + marker: None, + location: loc, + dclx: None, + href: None, + layer: None, + }); + } + } + Tok::Open(n @ ("ordered_list" | "unordered_list")) => { + let nested_ordered = *n == "ordered_list"; + *i += 1; + parse_list(toks, i, out, nested_ordered, level + 1); + } + // Anything else (locs, stray text between items): skip. + Tok::Open(n) if is_block_start(n) && *n != "list_item" => break, + _ => *i += 1, + } + } +} + +/// OTSL cell-marker kinds, in DocTags' unclosed-token form. +fn cell_kind(name: &str) -> Option<&'static str> { + match name { + "fcel" => Some("fcel"), + "ched" => Some("ched"), + "rhed" => Some("rhed"), + "ecel" => Some("ecel"), + "lcel" => Some("lcel"), + "ucel" => Some("ucel"), + "xcel" => Some("xcel"), + _ => None, + } +} + +fn parse_otsl(toks: &[Tok], i: &mut usize, out: &mut Vec) { + let mut locs: Vec = Vec::new(); + let mut caption: Option = None; + let mut rows: Vec> = Vec::new(); + let mut kinds: Vec> = Vec::new(); + let mut row: Vec = Vec::new(); + let mut kind_row: Vec<&'static str> = Vec::new(); + let mut cell: Option<(&'static str, String)> = None; + + fn close_cell( + cell: &mut Option<(&'static str, String)>, + row: &mut Vec, + kind_row: &mut Vec<&'static str>, + ) { + if let Some((kind, text)) = cell.take() { + row.push(text.trim().to_string()); + kind_row.push(kind); + } + } + fn close_row( + cell: &mut Option<(&'static str, String)>, + row: &mut Vec, + kind_row: &mut Vec<&'static str>, + rows: &mut Vec>, + kinds: &mut Vec>, + ) { + close_cell(cell, row, kind_row); + if !row.is_empty() { + rows.push(std::mem::take(row)); + kinds.push(std::mem::take(kind_row)); + } + } + + while *i < toks.len() { + match &toks[*i] { + Tok::Close("otsl") => { + *i += 1; + break; + } + Tok::Open("nl") => { + close_row(&mut cell, &mut row, &mut kind_row, &mut rows, &mut kinds); + *i += 1; + } + Tok::Open("caption") => { + *i += 1; + let (text, _) = collect_inline(toks, i, "caption"); + if !text.is_empty() { + caption = Some(text); + } + } + Tok::Open(n) => { + if let Some(kind) = cell_kind(n) { + close_cell(&mut cell, &mut row, &mut kind_row); + cell = Some((kind, String::new())); + *i += 1; + } else if is_block_start(n) { + // Truncated table (no ): don't eat the next block. + break; + } else { + *i += 1; + } + } + Tok::Loc(v) => { + if cell.is_none() && locs.len() < 4 { + locs.push(*v); + } + *i += 1; + } + Tok::Text(t) => { + if let Some((_, text)) = cell.as_mut() { + if !text.is_empty() { + text.push(' '); + } + text.push_str(t); + } + *i += 1; + } + Tok::Close(_) => *i += 1, + } + } + close_row(&mut cell, &mut row, &mut kind_row, &mut rows, &mut kinds); + + // docling's reading order puts the caption paragraph before the table. + if let Some(text) = caption { + out.push(Node::Paragraph { text }); + } + if rows.is_empty() { + return; + } + // Span/header structure from the marker kinds. `rows` keeps empty + // strings for continuation cells (the span text is in the anchor cell), + // which is exactly how the Markdown/DocLang serializers expect the grid. + let structure = TableStructure { + header_row: kinds + .iter() + .map(|k| { + let mut headers = 0usize; + let mut filled = 0usize; + for kind in k { + match *kind { + "ched" => { + headers += 1; + filled += 1; + } + "fcel" | "rhed" => filled += 1, + _ => {} + } + } + filled > 0 && headers == filled + }) + .collect(), + col_continuation: kinds + .iter() + .map(|k| { + k.iter() + .map(|kind| matches!(*kind, "lcel" | "xcel")) + .collect() + }) + .collect(), + row_continuation: kinds + .iter() + .map(|k| { + k.iter() + .map(|kind| matches!(*kind, "ucel" | "xcel")) + .collect() + }) + .collect(), + row_header: kinds + .iter() + .map(|k| k.iter().map(|kind| *kind == "rhed").collect()) + .collect(), + col_header: kinds + .iter() + .map(|k| k.iter().map(|kind| *kind == "ched").collect()) + .collect(), + }; + out.push(Node::Table(Table { + rows, + location: (locs.len() == 4).then(|| [locs[0], locs[1], locs[2], locs[3]]), + structure: Some(structure), + cell_blocks: None, + })); +} + +fn parse_picture(toks: &[Tok], i: &mut usize, out: &mut Vec, close: &str) { + let mut locs: Vec = Vec::new(); + let mut caption: Option = None; + while *i < toks.len() { + match &toks[*i] { + Tok::Close(n) if *n == close => { + *i += 1; + break; + } + Tok::Open("caption") => { + *i += 1; + let (text, _) = collect_inline(toks, i, "caption"); + if !text.is_empty() { + caption = Some(text); + } + } + Tok::Open("otsl") => { + // A chart's tabular payload: emit the picture first (below), + // then the data table after it, like the chart backends do. + break; + } + Tok::Open(n) if is_block_start(n) => break, + Tok::Loc(v) => { + if locs.len() < 4 { + locs.push(*v); + } + *i += 1; + } + // Picture-class tokens (``, ``, …) and stray + // text: dropped — docling's picture body text is not extracted. + _ => *i += 1, + } + } + out.push(located( + Node::Picture { + caption, + image: None, + classification: None, + }, + (locs.len() == 4).then(|| [locs[0], locs[1], locs[2], locs[3]]), + )); +} + +#[cfg(test)] +mod tests { + use super::*; + + /// The exact shape live granite-docling emits (loc runs, unclosed OTSL + /// markers with span cells, in-otsl caption, furniture headers) — from + /// the #77 bring-up against a real endpoint. + #[test] + fn granite_page_parses_with_structure() { + let page = "Running Title\n\ +Intro paragraph.\n\ +5.1 Optimization\n\ +AB123\ +Table 1. Caption text.\n\ +FirstSecond\n\ +Fig 1.\n\ +"; + let doc = parse(page); + let md = doc.export_to_markdown(); + assert!(md.contains("Intro paragraph."), "md: {md:?}"); + assert!(md.contains("## 5.1 Optimization"), "md: {md:?}"); + // Caption precedes the table. + assert!( + md.find("Table 1. Caption text.").unwrap() < md.find("| A").unwrap_or(usize::MAX), + "md: {md:?}" + ); + assert!( + md.contains("- First") && md.contains("- Second"), + "md: {md:?}" + ); + // Furniture stays out of the body. + assert!(!md.contains("Running Title"), "md: {md:?}"); + // Geometry survives as provenance on the table. + let table = doc + .nodes + .iter() + .find_map(|n| match n { + Node::Table(t) => Some(t), + _ => None, + }) + .expect("table parsed"); + assert_eq!(table.location, Some([114, 212, 388, 297])); + let s = table.structure.as_ref().expect("structure"); + assert_eq!(s.header_row, vec![true, false]); + assert_eq!(s.col_continuation[0], vec![false, false, true]); + } + + /// The degraded shape token-stripping servers produce: loc runs + bare + /// text, no structure tags at all. Text must survive as paragraphs. + #[test] + fn degraded_loc_and_text_becomes_paragraphs() { + let doc = parse( + "First line.\nSecond line.", + ); + let md = doc.export_to_markdown(); + assert!(md.contains("First line."), "md: {md:?}"); + assert!(md.contains("Second line."), "md: {md:?}"); + } + + /// Truncated generation: an element with no closer must not swallow the + /// following block. + #[test] + fn truncated_element_does_not_eat_next_block() { + let doc = parse("Cut off\nNext"); + let md = doc.export_to_markdown(); + assert!(md.contains("Cut off"), "md: {md:?}"); + assert!(md.contains("## Next"), "md: {md:?}"); + } + + #[test] + fn entities_and_stray_angle_brackets() { + let doc = parse("A & B & a < b"); + let md = doc.export_to_markdown(); + assert!(md.contains("A & B & a < b"), "md: {md:?}"); + } + + #[test] + fn pages_join_with_page_breaks() { + let doc = parse_pages(["One.", "Two."]); + assert!(matches!(doc.nodes[1], Node::PageBreak)); + let md = doc.export_to_markdown(); + assert!(md.contains("One.") && md.contains("Two.")); + } +} diff --git a/crates/docling-core/src/lib.rs b/crates/docling-core/src/lib.rs index 605a8010..38df0b87 100644 --- a/crates/docling-core/src/lib.rs +++ b/crates/docling-core/src/lib.rs @@ -13,6 +13,7 @@ pub mod base64; pub mod chunker; mod doclang; +pub mod doctags; mod document; mod json; mod labels; diff --git a/crates/docling/src/vlm.rs b/crates/docling/src/vlm.rs index 1cfe41ad..2a86cfa3 100644 --- a/crates/docling/src/vlm.rs +++ b/crates/docling/src/vlm.rs @@ -4,10 +4,12 @@ //! (`ApiVlmOptions`): each PDF page is rendered to an image, sent to an //! OpenAI-compatible `chat/completions` endpoint (LM Studio, Ollama, vLLM, or //! a hosted service) together with a DocLang-eliciting prompt, and the -//! returned markup is parsed by the existing DocLang reader -//! (`backend::doclang`) into a [`DoclingDocument`]. Local ONNX inference of a -//! docling VLM is a later enhancement — this module deliberately contains no -//! model code, just the request loop. +//! returned markup is parsed into a [`DoclingDocument`] — DocTags answers +//! (granite-docling-class models) via `docling_core::doctags`' tolerant +//! parser (#152), DocLang XML via the existing reader (`backend::doclang`), +//! and untagged prose via a line-per-paragraph fallback. Local ONNX +//! inference of a docling VLM is a later enhancement — this module +//! deliberately contains no model code, just the request loop. //! //! HTTP goes over `ureq`, the crate's existing blocking client //! (`fetch-images` pulls the same one, keeping a single HTTP stack in the @@ -143,7 +145,7 @@ pub fn convert_vlm( encode_png(&page.image).and_then(|png| request_page(&agent, opts, &png)); match step { Ok(markup) => { - fragments.push(doclang_fragment(&markup)); + fragments.push(strip_wrappers(&markup)); Ok(()) } Err(e) => { @@ -162,7 +164,7 @@ pub fn convert_vlm( // The image file is already the page; no re-encode, no pdfium. let markup = request_page(&agent, opts, &source.bytes) .map_err(|e| ConversionError::Parse(format!("vlm: {e}")))?; - fragments.push(doclang_fragment(&markup)); + fragments.push(strip_wrappers(&markup)); } other => { return Err(ConversionError::Parse(format!( @@ -170,15 +172,24 @@ pub fn convert_vlm( ))); } } - // One document out of the per-page fragments, through the tolerant - // DocLang reader — exactly what a `.dclg` file would take. - let xml = format!( - "\n{}\n", - fragments.join("\n") - ); - let synthetic = - SourceDocument::from_bytes(&source.name, InputFormat::XmlDoclang, xml.into_bytes()); - let doc = DoclangBackend.convert(&synthetic)?; + // Two grammars come back from the wire. DocTags (granite-docling-class + // models: loc tokens, unclosed OTSL markers) goes to docling-core's + // dedicated tolerant parser, which keeps geometry and span structure + // (#152). Proper DocLang XML — or plain prose — takes the DocLang + // reader path a `.dclg` file would. One DocTags-shaped page routes the + // whole document: mixing grammars page-to-page is model misbehavior, + // and the DocTags parser degrades to paragraphs on non-DocTags input. + let doc = if fragments.iter().any(|f| looks_like_doctags(f)) { + let mut doc = docling_core::doctags::parse_pages(fragments.iter().map(String::as_str)); + doc.name = source.name.clone(); + doc + } else { + let body: Vec = fragments.iter().map(|f| prose_fallback(f)).collect(); + let xml = format!("\n{}\n", body.join("\n")); + let synthetic = + SourceDocument::from_bytes(&source.name, InputFormat::XmlDoclang, xml.into_bytes()); + DoclangBackend.convert(&synthetic)? + }; if doc.nodes.is_empty() { // The request loop succeeded, so this is a content problem, not a // transport one: the model answered with nothing our reader keeps. @@ -317,32 +328,34 @@ fn request_page(agent: &ureq::Agent, opts: &VlmOptions, image: &[u8]) -> Result< Err(format!("giving up after 4 attempts: {last_err}")) } -/// Reduce one model response to DocLang *body* markup, ready to concatenate -/// under a single `` root. -/// -/// Models wrap their answer unpredictably: Markdown code fences, a full -/// `` document, a legacy `` root, or a bare fragment of -/// block elements. The wrappers strip first, then the body goes through the -/// DocTags→DocLang lexical translation ([`doctags_to_doclang`]) — a no-op -/// for already-DocLang answers, load-bearing for granite-docling-class -/// models whose raw DocTags token stream is not XML at all. -fn doclang_fragment(response: &str) -> String { - let translated = doctags_to_doclang(&strip_wrappers(response)); - // A model that ignored the markup instruction entirely (plain prose / - // Markdown, no tags) still carries the page text — wrap it as one - // paragraph per non-empty line instead of silently dropping everything. - if !translated.contains('<') && !translated.trim().is_empty() { - return translated +/// Is this fragment DocTags rather than DocLang? Granite-class models always +/// carry `loc_` tokens; the OTSL/section-header vocabularies clinch it when a +/// (hypothetical) model omits locations. +fn looks_like_doctags(fragment: &str) -> bool { + fragment.contains("") + || fragment.contains("") + || fragment.contains("` per non-empty +/// line, instead of silently dropping the page's content. +fn prose_fallback(fragment: &str) -> String { + if !fragment.contains('<') && !fragment.trim().is_empty() { + return fragment .lines() .filter(|l| !l.trim().is_empty()) .map(|l| format!("{}", l.trim())) .collect::>() .join("\n"); } - translated + fragment.to_string() } -/// The wrapper-stripping half of [`doclang_fragment`]. +/// Strip the wrappers models put around their answer: Markdown code fences +/// and a ``/`` document root (the body keeps its grammar — +/// DocTags routing happens later on the stripped fragment). fn strip_wrappers(response: &str) -> String { let mut text = response.trim(); // ```xml … ``` / ``` … ``` fences. @@ -371,233 +384,6 @@ fn strip_wrappers(response: &str) -> String { text.to_string() } -/// Translate a legacy **DocTags** fragment (what granite-docling-class models -/// actually emit) into well-formed DocLang the XML reader accepts. -/// -/// DocTags is a token stream, not XML: `` location tokens, OTSL cell -/// markers (`Text…`), picture-class and checkbox tokens are -/// all *unclosed*. The DocLang reader already speaks OTSL — its `` -/// parser takes self-closed `` markers with the text between them — -/// so the translation is purely lexical, one pass over the tags: -/// -/// - `` and `` tokens drop (the reader's geometry comes -/// from ``, which VLM output doesn't carry); -/// - `` → `
`, `` → `` (docling parity: section level 1 renders `##`; plain `#` -/// is the document title), `` → `<heading level="1">`, -/// `<paragraph>` → `<text>`, `<ordered_list>`/`<unordered_list>` → -/// `<list>`, `<list_item>` → the `<ldiv/>` item marker DocLang lists use; -/// - a `<caption>` inside `<otsl>` hoists to a `<text>` paragraph *before* -/// the `<table>` (docling's reading order); a standalone `<caption>` -/// becomes `<text>`; a `<picture>`'s caption stays where the picture -/// parser expects it; -/// - `<page_header>`/`<page_footer>` → `<text>` opening with a -/// `<layer value="furniture"/>` head, so headers/footers stay out of the -/// Markdown body exactly like the ML pipeline's furniture; -/// - any other tag that never sees a matching closer in the fragment — -/// OTSL cell markers, picture classes, checkbox states, whatever a future -/// model invents — is emitted self-closed, which the tolerant reader -/// recurses through harmlessly; -/// - stray `&` (not an entity) and dangling `<` escape to entities, since -/// model text is not XML-escaped. -/// -/// A fragment that is already DocLang passes through intact: every paired -/// element has its closer, none of the rename names exist in DocLang, and -/// proper entities are left alone. -fn doctags_to_doclang(fragment: &str) -> String { - // Tag names that appear in closing form — those pairs are kept as-is; - // everything else unknown is a single marker token. - let mut closers: std::collections::HashSet<&str> = std::collections::HashSet::new(); - let bytes = fragment.as_bytes(); - let mut i = 0; - while let Some(off) = fragment[i..].find("</") { - let start = i + off + 2; - let end = fragment[start..] - .find('>') - .map(|e| start + e) - .unwrap_or(fragment.len()); - closers.insert(fragment[start..end].trim()); - i = end; - } - - let mut out = String::with_capacity(fragment.len()); - let mut i = 0; - // Lexical nesting state for caption hoisting: where the current - // `<table>` opened (to insert the hoisted caption before it), and - // whether we're inside a `<picture>` (whose caption is native DocLang). - let mut table_open_at: Option<usize> = None; - let mut picture_depth = 0usize; - while i < bytes.len() { - let c = bytes[i] as char; - if c != '<' { - if c == '&' { - // Escape a bare ampersand; keep real entities (& {). - let rest = &fragment[i + 1..]; - let is_entity = rest - .split_once(';') - .map(|(name, _)| { - !name.is_empty() - && name.len() <= 8 - && name - .chars() - .all(|ch| ch.is_ascii_alphanumeric() || ch == '#') - }) - .unwrap_or(false); - out.push_str(if is_entity { "&" } else { "&" }); - } else { - out.push(c); - } - i += c.len_utf8(); - continue; - } - // A tag candidate: `<`, optional `/`, then a name. Anything else is - // literal text (`a < b`) and escapes. - let close = bytes.get(i + 1) == Some(&b'/'); - let name_start = i + 1 + usize::from(close); - let Some(end) = fragment[i..].find('>').map(|e| i + e) else { - out.push_str("<"); - i += 1; - continue; - }; - let inner = fragment[name_start..end].trim(); - let name = inner - .split([' ', '\t', '\n', '/']) - .next() - .unwrap_or_default(); - if name.is_empty() - || !name - .chars() - .next() - .is_some_and(|ch| ch.is_ascii_alphabetic()) - { - out.push_str("<"); - i += 1; - continue; - } - let self_closed = inner.ends_with('/'); - i = end + 1; - // Dropped tokens. - if name.starts_with("loc_") || name == "page_break" || name == "doctag" || name == "doctags" - { - continue; - } - // Renames (open and close forms). - let level = name - .strip_prefix("section_header_level_") - .and_then(|v| v.parse::<u8>().ok()); - // A table caption in DocTags sits inside <otsl>; DocLang wants it as - // a paragraph before the table. Capture the pair verbatim (dropping - // nested tokens) and splice it in front of the <table> opener. - if let Some(at) = table_open_at.filter(|_| !close && name == "caption") { - let inner_end = fragment[i..].find("</caption>").map(|e| i + e); - let (raw, next) = match inner_end { - Some(e) => (&fragment[i..e], e + "</caption>".len()), - None => (&fragment[i..], fragment.len()), - }; - let text = strip_tokens(raw); - if !text.trim().is_empty() { - out.insert_str(at, &format!("<text>{}</text>\n", text.trim())); - } - i = next; - continue; - } - if close { - match name { - _ if level.is_some() => out.push_str("</heading>"), - "title" => out.push_str("</heading>"), - "otsl" => { - table_open_at = None; - out.push_str("</table>"); - } - "picture" => { - picture_depth = picture_depth.saturating_sub(1); - out.push_str("</picture>"); - } - "caption" if picture_depth == 0 => out.push_str("</text>"), - // An item's text runs until the next `<ldiv/>`; the closer is - // structural noise in DocLang. - "list_item" => {} - "paragraph" | "page_header" | "page_footer" => out.push_str("</text>"), - "ordered_list" | "unordered_list" => out.push_str("</list>"), - _ => { - out.push_str("</"); - out.push_str(name); - out.push('>'); - } - } - continue; - } - match name { - _ if level.is_some() => { - // docling parity: section_header level K renders as heading - // K+1 in Markdown ("## 5.1 …"); "#" is the document title. - out.push_str(&format!( - "<heading level=\"{}\">", - (level.unwrap().saturating_add(1)).clamp(2, 6) - )); - } - "title" => out.push_str("<heading level=\"1\">"), - "otsl" => { - table_open_at = Some(out.len()); - out.push_str("<table>"); - } - "picture" if closers.contains("picture") => { - picture_depth += 1; - out.push('<'); - out.push_str(inner); - out.push('>'); - } - "caption" if picture_depth == 0 && closers.contains("caption") => { - out.push_str("<text>") - } - // DocLang list items are `<ldiv/>` markers followed by the item's - // text, not wrapper elements. - "list_item" => out.push_str("<ldiv/>"), - "paragraph" => out.push_str("<text>"), - "page_header" | "page_footer" => out.push_str("<text><layer value=\"furniture\"/>"), - "ordered_list" => out.push_str("<list class=\"ordered\">"), - "unordered_list" => out.push_str("<list>"), - _ if self_closed || closers.contains(name) => { - // A proper pair (DocLang vocabulary) or already self-closed: - // pass through with attributes intact. - out.push('<'); - out.push_str(inner); - out.push('>'); - } - _ => { - // A bare marker token with no closer anywhere — OTSL cells, - // picture classes, checkbox states. Self-close it. - out.push('<'); - out.push_str(name); - out.push_str("/>"); - } - } - } - out -} - -/// Drop every `<...>` token from a captured span, keeping only its text — -/// used for caption bodies, whose loc tokens carry no reader-usable geometry. -fn strip_tokens(raw: &str) -> String { - let mut text = String::new(); - let mut rest = raw; - // Skip the opening tag itself. - if let Some(gt) = rest.find('>') { - rest = &rest[gt + 1..]; - } - while let Some(lt) = rest.find('<') { - text.push_str(&rest[..lt]); - match rest[lt..].find('>') { - Some(gt) => rest = &rest[lt + gt + 1..], - None => { - rest = ""; - } - } - } - text.push_str(rest); - text -} - /// PNG-encode a rendered page (the wire format every OpenAI-compatible /// server accepts as a data URI). fn encode_png(image: &image::RgbImage) -> Result<Vec<u8>, String> { @@ -610,64 +396,66 @@ fn encode_png(image: &image::RgbImage) -> Result<Vec<u8>, String> { #[cfg(test)] mod tests { - use super::doclang_fragment; + use super::{looks_like_doctags, prose_fallback, strip_wrappers}; #[test] - fn fragment_normalization() { + fn wrapper_stripping() { // Bare fragment passes through. - assert_eq!(doclang_fragment("<text>hi</text>"), "<text>hi</text>"); + assert_eq!(strip_wrappers("<text>hi</text>"), "<text>hi</text>"); // Fenced answer is unwrapped. assert_eq!( - doclang_fragment("```xml\n<text>hi</text>\n```"), + strip_wrappers("```xml\n<text>hi</text>\n```"), "<text>hi</text>" ); // A full document root is stripped down to its body. assert_eq!( - doclang_fragment("<doclang version=\"0.7\"><text>hi</text></doclang>"), + strip_wrappers("<doclang version=\"0.7\"><text>hi</text></doclang>"), "<text>hi</text>" ); - // Legacy doctag root likewise. assert_eq!( - doclang_fragment("<doctag><text>hi</text></doctag>"), + strip_wrappers("<doctag><text>hi</text></doctag>"), "<text>hi</text>" ); - // Prose around a fenced fragment is ignored by the fence rule only - // when the fence comes first; a root element wins anywhere. + // Prose around a root element: the root wins anywhere. assert_eq!( - doclang_fragment("Here you go:\n<doclang><heading level=\"1\">T</heading></doclang>"), + strip_wrappers("Here you go:\n<doclang><heading level=\"1\">T</heading></doclang>"), "<heading level=\"1\">T</heading>" ); } - /// Raw granite-docling output is DocTags — loc tokens, unclosed OTSL - /// markers, section_header naming — and must come out as parseable - /// DocLang (this exact shape produced the "expected 'loc_420' tag" - /// failure against a live Ollama endpoint). + /// Routing: granite-style DocTags goes to the docling-core parser (whose + /// own tests pin the full markup semantics); DocLang and prose do not. + #[test] + fn doctags_routing() { + assert!(looks_like_doctags( + "<text><loc_1><loc_2><loc_3><loc_4>Body</text>" + )); + assert!(looks_like_doctags("<otsl><ched>A<nl></otsl>")); + assert!(looks_like_doctags( + "<section_header_level_1>T</section_header_level_1>" + )); + assert!(!looks_like_doctags("<heading level=\"2\">T</heading>")); + assert!(!looks_like_doctags("Just prose, no markup.")); + } + + /// End-to-end through the same assembly the pipeline uses: the exact + /// shape live granite-docling emits (from the #77 bring-up) renders with + /// docling-parity structure. #[test] - fn doctags_translation() { + fn doctags_end_to_end_markdown() { let page = "<doctag><picture><loc_15><loc_10><loc_240><loc_60><other></picture>\ <section_header_level_1><loc_57><loc_70><loc_420><loc_78>Optimized Table Tokenization</section_header_level_1>\ <text><loc_57><loc_84><loc_420><loc_98>Body with A & B & C.</text>\ <unordered_list><list_item><loc_60><loc_100><loc_420><loc_108>First item</list_item></unordered_list>\ <otsl><loc_57><loc_120><loc_420><loc_160><caption><loc_57><loc_115><loc_420><loc_119>Table 1. HPO results.</caption><ched>Col A<ched>Col B<nl><fcel>1<fcel>2<nl></otsl>\ -<page_footer><loc_57><loc_280><loc_420><loc_288>7</page_footer><page_break></doctag>"; - let fragment = doclang_fragment(page); - let xml = format!("<doclang version=\"0.7\">{fragment}</doclang>"); - let doc = crate::backend::DeclarativeBackend::convert( - &crate::backend::doclang::DoclangBackend, - &crate::source::SourceDocument::from_bytes( - "page", - crate::format::InputFormat::XmlDoclang, - xml.into_bytes(), - ), - ) - .expect("translated DocTags must parse"); - let md = doc.export_to_markdown(); +<page_footer><loc_57><loc_280><loc_420><loc_288>7</page_footer></doctag>"; + let fragment = strip_wrappers(page); + assert!(looks_like_doctags(&fragment)); + let md = docling_core::doctags::parse(&fragment).export_to_markdown(); // Section level 1 → "##" (docling parity; bare "#" is the title). assert!(md.contains("## Optimized Table Tokenization"), "md: {md:?}"); assert!(!md.contains("### Optimized"), "md: {md:?}"); - // The in-otsl caption hoists to a paragraph before the table. - assert!(md.contains("Table 1. HPO results."), "md: {md:?}"); + // The in-otsl caption becomes a paragraph before the table. assert!( md.find("Table 1. HPO results.").unwrap() < md.find("Col A").unwrap(), "caption must precede the table: {md:?}" @@ -675,15 +463,17 @@ mod tests { assert!(md.contains("Body with A & B & C."), "md: {md:?}"); assert!(md.contains("- First item"), "md: {md:?}"); assert!(md.contains("Col A |"), "md: {md:?}"); - assert!(md.contains("1 |"), "md: {md:?}"); // Furniture (page footer) stays out of the Markdown body. assert!(!md.contains("\n7\n"), "md: {md:?}"); } - /// DocLang-native fragments survive the translation byte-for-byte. #[test] - fn doclang_passthrough() { - let native = "<heading level=\"2\">T</heading>\n<text>Hi & bye <bold>b</bold></text>\n<table>\n<fcel/>a\n<nl/>\n</table>"; - assert_eq!(doclang_fragment(native), native); + fn prose_fallback_wraps_untagged_lines() { + assert_eq!( + prose_fallback("First line.\n\nSecond line."), + "<text>First line.</text>\n<text>Second line.</text>" + ); + // Tagged content is left for the DocLang reader untouched. + assert_eq!(prose_fallback("<text>hi</text>"), "<text>hi</text>"); } } diff --git a/crates/docling/tests/vlm.rs b/crates/docling/tests/vlm.rs index 1e433be4..99b74366 100644 --- a/crates/docling/tests/vlm.rs +++ b/crates/docling/tests/vlm.rs @@ -149,6 +149,26 @@ fn vlm_converts_an_image_without_pdfium() { assert!(doc.export_to_markdown().contains("From an image.")); } +/// A granite-style DocTags answer routes through docling-core's DocTags +/// parser end to end (image leg: no pdfium needed, runs everywhere). +#[test] +fn vlm_parses_doctags_answers() { + let (endpoint, served, handle) = mock_openai(vec![ + "<doctag><section_header_level_1><loc_1><loc_2><loc_3><loc_4>Section</section_header_level_1>\ +<text><loc_1><loc_5><loc_3><loc_6>Body text.</text>\ +<otsl><loc_1><loc_7><loc_3><loc_9><ched>H<nl><fcel>v<nl></otsl></doctag>" + .into(), + ]); + let source = SourceDocument::from_bytes("page.png", InputFormat::Image, image_bytes()); + let doc = convert_vlm(&source, &opts(endpoint)).expect("vlm conversion"); + handle.join().expect("mock server"); + assert_eq!(served.load(Ordering::SeqCst), 1); + let md = doc.export_to_markdown(); + assert!(md.contains("## Section"), "md: {md:?}"); + assert!(md.contains("Body text."), "md: {md:?}"); + assert!(md.contains("| H"), "md: {md:?}"); +} + #[test] fn vlm_rejects_non_visual_formats() { let source = SourceDocument::from_bytes("x.md", InputFormat::Md, b"# hi".to_vec()); From fe0ff00a99b70b3161dba0a03c840d1a1d7ce1c6 Mon Sep 17 00:00:00 2001 From: artiz <artem.kustikov@gmail.com> Date: Thu, 23 Jul 2026 19:33:31 +0000 Subject: [PATCH 3/3] =?UTF-8?q?core:=20complete=20the=20DocTags=20reader?= =?UTF-8?q?=20=E2=80=94=20checkboxes,=20key-value=20regions,=20.doctags=20?= =?UTF-8?q?input=20format=20(#152)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rounds out the DocTags parser to the full vocabulary Python docling-core's DocTagsDocument covers: - <checkbox_selected>/<checkbox_unselected> → Node::CheckboxItem (Markdown task-list rendering), both as block-level tokens and inside a <text> wrapper; - <key_value_region> (DocTags) / <field_region> (DocLang naming) → Node::FieldRegion, tolerant over numbered DocTags pairs (<key_1>…<value_1>…) and the DocLang <field_item> shape, including <unmatched_value>; - .doctags/.dt files are now a first-class InputFormat (DocTags) wired through the converter to docling_core::doctags::parse — a saved model transcript converts like any other document; name maps added to the Python/Node bindings' format tables. vlm_conformance.sh from the first live corpus run: per-side progress lines (a multi-page fixture takes minutes per page and looked hung), and cached outputs write through a temp file + rename so Ctrl+C never caches a half-written result. README notes the DocTags input; the #152 scope — parser in docling-core, model-typed API, tolerance, geometry — is now fully implemented. Refs #152 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT --- README.md | 5 +- crates/docling-core/src/doctags.rs | 127 ++++++++++++++++++++++++- crates/docling-node/src/lib.rs | 1 + crates/docling-py/src/lib.rs | 1 + crates/docling/src/converter.rs | 21 ++++ crates/docling/src/format.rs | 5 + scripts/conformance/vlm_conformance.sh | 31 ++++-- 7 files changed, 181 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index fe81c848..4eb3393a 100644 --- a/README.md +++ b/README.md @@ -17,7 +17,10 @@ mapping, and per-format conformance. The public API works end to end across **Markdown, CSV, HTML, AsciiDoc, DOCX, PPTX, XLSX, legacy DOC/XLS/PPT, EPUB, ODF, WebVTT, Email, MHTML, JATS, USPTO, XBRL, LaTeX, JSON, PDF, images, METS, audio and video** — plus Markdown / docling-JSON output and image -extraction. MHTML is a docling.rs-only extension (docling has no MHTML +extraction. Raw **DocTags** (`.doctags`/`.dt` — the token markup docling's VLMs emit) reads +in through `docling-core`'s tolerant DocTags parser (#152), the same one the +VLM pipeline uses for model responses. +MHTML is a docling.rs-only extension (docling has no MHTML backend): saved-webpage `.mhtml`/`.mht` archives are parsed as a MIME message with [`mail-parser`](https://crates.io/crates/mail-parser) (which conforms to [RFC 2557](https://datatracker.ietf.org/doc/html/rfc2557), the MHTML spec) and diff --git a/crates/docling-core/src/doctags.rs b/crates/docling-core/src/doctags.rs index c96f11d4..ef2badac 100644 --- a/crates/docling-core/src/doctags.rs +++ b/crates/docling-core/src/doctags.rs @@ -25,7 +25,7 @@ //! - `<page_header>`/`<page_footer>` become [`Node::PageFurniture`], which //! the Markdown/JSON exports omit like every other furniture. -use crate::document::{DoclingDocument, Node, Table, TableStructure}; +use crate::document::{DoclingDocument, FieldItem, Node, Table, TableStructure}; /// Parse one DocTags fragment (typically one page's model output). pub fn parse(markup: &str) -> DoclingDocument { @@ -163,6 +163,10 @@ fn is_block_start(name: &str) -> bool { | "page_header" | "page_footer" | "page_break" + | "checkbox_selected" + | "checkbox_unselected" + | "key_value_region" + | "field_region" | "doctag" | "doctags" ) || name.starts_with("section_header_level_") @@ -349,6 +353,22 @@ fn parse_blocks(toks: &[Tok], i: &mut usize, out: &mut Vec<Node>, list_level: u8 } "otsl" => parse_otsl(toks, i, out), "picture" | "chart" => parse_picture(toks, i, out, name), + // A form checkbox: the token precedes its label text. + "checkbox_selected" | "checkbox_unselected" => { + let (text, loc) = collect_inline(toks, i, name); + if !text.is_empty() { + out.push(located( + Node::CheckboxItem { + checked: name == "checkbox_selected", + text, + }, + loc, + )); + } + } + // A form key-value region (DocTags' key_value_region / + // DocLang's field_region). + "key_value_region" | "field_region" => parse_field_region(toks, i, out, name), // Unknown block token: transparent (its text will surface // as stray block-level text above). _ => {} @@ -604,6 +624,83 @@ fn parse_picture(toks: &[Tok], i: &mut usize, out: &mut Vec<Node>, close: &str) )); } +/// A form key-value region: tolerant over both the DocTags shape +/// (`<key_1>…</key_1><value_1>…</value_1>` numbered pairs) and the DocLang +/// shape (`<field_item><marker/><key/><value/></field_item>`). A `key` +/// starts a new item; `value`/`marker` attach to the current one. +fn parse_field_region(toks: &[Tok], i: &mut usize, out: &mut Vec<Node>, close: &str) { + let mut items: Vec<FieldItem> = Vec::new(); + let mut current: Option<FieldItem> = None; + while *i < toks.len() { + match &toks[*i] { + Tok::Close(n) if *n == close => { + *i += 1; + break; + } + Tok::Open(n @ ("field_item" | "unmatched_value")) => { + let n = *n; + if let Some(item) = current.take() { + items.push(item); + } + current = Some(FieldItem::default()); + *i += 1; + // `<unmatched_value>` (a value with no key) carries its text + // directly rather than nested key/value children. + if n == "unmatched_value" { + let (text, _) = collect_inline(toks, i, n); + if let (Some(item), false) = (current.as_mut(), text.is_empty()) { + item.value = Some(text); + } + } + } + Tok::Close("field_item") => { + if let Some(item) = current.take() { + items.push(item); + } + *i += 1; + } + Tok::Open(n) => { + let n = *n; + let is_key = n == "key" || n.starts_with("key_"); + let is_value = n == "value" || n.starts_with("value_"); + let is_marker = n == "marker"; + if !(is_key || is_value || is_marker) { + if is_block_start(n) { + break; + } + *i += 1; + continue; + } + *i += 1; + let (text, _) = collect_inline(toks, i, n); + if text.is_empty() { + continue; + } + if is_key { + // A key starts a new pair (numbered DocTags pairs come + // key-then-value with no item wrapper). + if current.as_ref().is_some_and(|c| c.key.is_some()) { + items.push(current.take().unwrap_or_default()); + } + current.get_or_insert_with(FieldItem::default).key = Some(text); + } else if is_value { + current.get_or_insert_with(FieldItem::default).value = Some(text); + } else { + current.get_or_insert_with(FieldItem::default).marker = Some(text); + } + } + _ => *i += 1, + } + } + if let Some(item) = current.take() { + items.push(item); + } + items.retain(|it| it.marker.is_some() || it.key.is_some() || it.value.is_some()); + if !items.is_empty() { + out.push(Node::FieldRegion { items }); + } +} + #[cfg(test)] mod tests { use super::*; @@ -680,6 +777,34 @@ mod tests { assert!(md.contains("A & B & a < b"), "md: {md:?}"); } + #[test] + fn checkboxes_and_key_value_regions() { + let doc = parse( + "<text><checkbox_selected>Done item</text>\ +<checkbox_unselected><loc_1><loc_2><loc_3><loc_4>Todo item\ +<key_value_region><key_1><loc_1><loc_2><loc_3><loc_4>Name</key_1><value_1>John</value_1>\ +<key_2>City</key_2><value_2>Berlin</value_2></key_value_region>", + ); + // Both forms produce checkbox items (the located one sits inside a + // Node::Located wrapper, so assert via the Markdown rendering). + let md = doc.export_to_markdown(); + assert!(md.contains("- [x] Done item"), "md: {md:?}"); + assert!(md.contains("- [ ] Todo item"), "md: {md:?}"); + let items = doc + .nodes + .iter() + .find_map(|n| match n { + Node::FieldRegion { items } => Some(items), + _ => None, + }) + .expect("field region parsed"); + assert_eq!(items.len(), 2); + assert_eq!(items[0].key.as_deref(), Some("Name")); + assert_eq!(items[0].value.as_deref(), Some("John")); + assert_eq!(items[1].key.as_deref(), Some("City")); + assert_eq!(items[1].value.as_deref(), Some("Berlin")); + } + #[test] fn pages_join_with_page_breaks() { let doc = parse_pages(["<text>One.</text>", "<text>Two.</text>"]); diff --git a/crates/docling-node/src/lib.rs b/crates/docling-node/src/lib.rs index 2362d57c..072378a8 100644 --- a/crates/docling-node/src/lib.rs +++ b/crates/docling-node/src/lib.rs @@ -1303,6 +1303,7 @@ fn parse_format(s: &str) -> Result<InputFormat> { "xml_xbrl" | "xbrl" => InputFormat::XmlXbrl, "json_docling" => InputFormat::JsonDocling, "xml_doclang" | "doclang" => InputFormat::XmlDoclang, + "doctags" | "dt" => InputFormat::DocTags, "mets_gbs" => InputFormat::MetsGbs, "email" => InputFormat::Email, "latex" => InputFormat::Latex, diff --git a/crates/docling-py/src/lib.rs b/crates/docling-py/src/lib.rs index ae769f38..0c4d8149 100644 --- a/crates/docling-py/src/lib.rs +++ b/crates/docling-py/src/lib.rs @@ -318,6 +318,7 @@ fn parse_format(name: &str) -> Option<docling::InputFormat> { "xml_jats" => XmlJats, "xml_xbrl" => XmlXbrl, "xml_doclang" => XmlDoclang, + "doctags" => DocTags, "mets_gbs" => MetsGbs, "json_docling" => JsonDocling, "audio" => Audio, diff --git a/crates/docling/src/converter.rs b/crates/docling/src/converter.rs index 3232d6c5..30143c92 100644 --- a/crates/docling/src/converter.rs +++ b/crates/docling/src/converter.rs @@ -500,6 +500,13 @@ impl DocumentConverter { InputFormat::XmlDoclang | InputFormat::Dclx => { crate::backend::DoclangBackend.convert(&source)? } + // Raw DocTags (VLM token markup, #152): the tolerant docling-core + // parser — never fails, best-effort document out. + InputFormat::DocTags => { + let mut doc = docling_core::doctags::parse(source.text()?); + doc.name = source.name.clone(); + doc + } #[cfg(feature = "pdf")] InputFormat::Pdf => docling_pdf::convert_with_options( &source.bytes, @@ -624,6 +631,20 @@ mod tests { assert_eq!(result.document.export_to_markdown(), "# Hello\n\nWorld.\n"); } + #[test] + fn doctags_input_converts() { + // Raw DocTags markup (#152) — the VLM token stream — as a first-class + // input format (.doctags/.dt), through the tolerant docling-core + // parser. + let markup = b"<doctag><section_header_level_1><loc_1><loc_2><loc_3><loc_4>Intro</section_header_level_1><text>Body.</text></doctag>" + .to_vec(); + let src = SourceDocument::from_bytes("page.doctags", InputFormat::DocTags, markup); + let result = DocumentConverter::new().convert(src).unwrap(); + let md = result.document.export_to_markdown(); + assert!(md.contains("## Intro"), "{md}"); + assert!(md.contains("Body."), "{md}"); + } + #[test] fn doclang_xml_round_trips() { // Every input format now has a backend; DocLang XML reads back in and diff --git a/crates/docling/src/format.rs b/crates/docling/src/format.rs index 0ddd3d9d..cd1d584d 100644 --- a/crates/docling/src/format.rs +++ b/crates/docling/src/format.rs @@ -29,6 +29,9 @@ pub enum InputFormat { XmlJats, XmlXbrl, XmlDoclang, + /// Raw DocTags markup (`.doctags`/`.dt`) — the token stream docling's + /// VLMs emit, parsed by `docling_core::doctags` (#152). + DocTags, /// A DocLang OPC archive (`.dclx`, the format `--to dclx` writes). Dclx, MetsGbs, @@ -70,6 +73,7 @@ impl InputFormat { InputFormat::XmlJats => "xml_jats", InputFormat::XmlXbrl => "xml_xbrl", InputFormat::XmlDoclang => "xml_doclang", + InputFormat::DocTags => "doctags", InputFormat::Dclx => "dclx", InputFormat::MetsGbs => "mets_gbs", InputFormat::JsonDocling => "json_docling", @@ -96,6 +100,7 @@ impl InputFormat { "html" | "htm" | "xhtml" => InputFormat::Html, "xml" | "nxml" => InputFormat::XmlJats, "dclg" => InputFormat::XmlDoclang, + "doctags" | "dt" => InputFormat::DocTags, "dclx" => InputFormat::Dclx, "jpg" | "jpeg" | "png" | "tif" | "tiff" | "bmp" | "webp" => InputFormat::Image, "adoc" | "asciidoc" | "asc" => InputFormat::Asciidoc, diff --git a/scripts/conformance/vlm_conformance.sh b/scripts/conformance/vlm_conformance.sh index 5f7bbff0..2b2dc6d2 100755 --- a/scripts/conformance/vlm_conformance.sh +++ b/scripts/conformance/vlm_conformance.sh @@ -65,17 +65,32 @@ for src in tests/data/pdf/sources/*.pdf; do fi rust_md="$OUT/rust/$name.md" py_md="$OUT/python/$name.md" - # Cached across reruns (each side costs minutes of GPU time per fixture); - # delete target/vlm-conformance/ to force regeneration. + # Cached across reruns (each side costs minutes of GPU time PER PAGE — + # multi-page fixtures take a while; the shim's terminal shows per-page + # progress). Writes go through a temp file so an interrupted run never + # caches a half-written output; delete target/vlm-conformance/ to force + # regeneration. if [ ! -s "$rust_md" ]; then - ./target/release/docling-rs --pipeline vlm --no-stream \ - --vlm-endpoint "$ENDPOINT" --vlm-model "$MODEL" "$src" > "$rust_md" \ - || { echo " rust side failed on $name" >&2; rm -f "$rust_md"; continue; } + echo "[$name] rust side converting (watch the shim terminal for per-page progress) ..." >&2 + if ./target/release/docling-rs --pipeline vlm --no-stream \ + --vlm-endpoint "$ENDPOINT" --vlm-model "$MODEL" "$src" > "$rust_md.tmp"; then + mv "$rust_md.tmp" "$rust_md" + else + echo " rust side failed on $name" >&2 + rm -f "$rust_md.tmp" + continue + fi fi if [ ! -s "$py_md" ]; then - python3 scripts/conformance/vlm_convert.py \ - --endpoint "$ENDPOINT" --model "$MODEL" "$src" "$py_md" \ - || { echo " python side failed on $name" >&2; rm -f "$py_md"; continue; } + echo "[$name] python docling side converting ..." >&2 + if python3 scripts/conformance/vlm_convert.py \ + --endpoint "$ENDPOINT" --model "$MODEL" "$src" "$py_md.tmp"; then + mv "$py_md.tmp" "$py_md" + else + echo " python side failed on $name" >&2 + rm -f "$py_md.tmp" + continue + fi fi sim="$(similarity "$py_md" "$rust_md")" is_exact="no"