From e74aa9c333f4dd609f9e5e98c84e17c8cc839b1f Mon Sep 17 00:00:00 2001 From: artiz Date: Sun, 26 Jul 2026 19:11:47 +0000 Subject: [PATCH 01/14] wasm: one demo page with the whole pipeline (#157) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The browser demo had grown into three pages: index.html (declarative only, no PDF/image, no output-format parity) plus ocr.html and scan.html, which were really OCR test harnesses. Fold everything into index.html and delete the other two. The single page now covers what the module can actually do: any supported document, PDF via its text layer, and — falling back automatically when a PDF has no text layer — scanned pages and images through the ONNX pipeline. Output picks between Markdown, docling JSON and DocLang XML; a Markdown preview renders the result. The OCR half (ORT, pdf.js, models) loads lazily, so a visitor converting a DOCX downloads none of it. Supporting changes: - convert() takes `images` ("placeholder" | "embedded"), mirroring docling-serve's option, so pictures can ride along as data URIs — there is no `referenced` mode because a page cannot write files. - ScannedConverter::finish and convert_scanned_image accept "doclang" too, so both paths offer the same three output grammars; the format is threaded through worker.js and pipeline.js. - A dead module worker (blocked CDN, offline, strict extension) used to leave every pending RPC unsettled, so the page sat on "loading …" forever. onerror/onmessageerror now reject the waiters with a message that names the cause. Verified: the page reports the failure in ~30 s instead of hanging. - Drop the temporary layout-region diagnostic probe and its JS plumbing, and the per-page console timing logs. - Inline favicon so a static host never 404s on /favicon.ico. Verified in headless Chromium against the real wasm module: DOCX → md/json/ doclang, images=placeholder vs embedded (data URI present), a corpus text-layer PDF, the Markdown preview, and cross-origin isolation via coi.js (crossOriginIsolated = true, so ORT gets threads). Refs #157 Signed-off-by: artiz Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT --- crates/docling-wasm/README.md | 73 +++- crates/docling-wasm/src/lib.rs | 76 +++- crates/docling-wasm/src/ocr.rs | 2 +- crates/docling-wasm/src/scanned.rs | 49 +-- crates/docling-wasm/src/tableformer.rs | 2 +- crates/docling-wasm/www/index.html | 496 ++++++++++++++++++++++--- crates/docling-wasm/www/ocr.html | 219 ----------- crates/docling-wasm/www/pipeline.js | 10 +- crates/docling-wasm/www/scan.html | 347 ----------------- crates/docling-wasm/www/worker.js | 13 +- 10 files changed, 606 insertions(+), 681 deletions(-) delete mode 100644 crates/docling-wasm/www/ocr.html delete mode 100644 crates/docling-wasm/www/scan.html diff --git a/crates/docling-wasm/README.md b/crates/docling-wasm/README.md index 6a5011fe..623ff1b9 100644 --- a/crates/docling-wasm/README.md +++ b/crates/docling-wasm/README.md @@ -26,12 +26,59 @@ converters and the PDF text parser are pure Rust. ## API ```ts -convert(bytes: Uint8Array, filename: string, to?: "md" | "json" | "doclang"): string +convert( + bytes: Uint8Array, + filename: string, + to?: "md" | "json" | "doclang", // default "md" + images?: "placeholder" | "embedded", // default "placeholder", Markdown only +): string supported_extensions(): string // JSON array, e.g. for version(): string ``` -The filename's extension drives format detection, same as the CLI. +The filename's extension drives format detection, same as the CLI. `images` +mirrors docling-serve's option: `placeholder` emits docling's ``, +`embedded` inlines the picture as a `data:` URI so the Markdown is +self-contained (a page has no filesystem, so there is no `referenced` mode). + +### Convert a file the user picked + +```js +import init, { convert } from "./pkg/docling_wasm.js"; +await init(); + +const file = input.files[0]; +const bytes = new Uint8Array(await file.arrayBuffer()); +const markdown = convert(bytes, file.name, "md"); +const json = convert(bytes, file.name, "json"); +const withPics = convert(bytes, file.name, "md", "embedded"); +``` + +### Scanned pages (OCR) + +Scanned PDFs and images need the ML models; everything else runs with no +network at all. The full wiring — model resolution, Web Worker, pdf.js +rasterization — is [`www/index.html`](./www/index.html); the short version: + +```js +import { createOcr } from "./pipeline.js"; + +const ocr = createOcr({ onStatus: (msg) => console.log(msg) }); +await ocr.boot(); // wasm + layout model + +// A standalone image is its own page. +const md = await ocr.convertImage(await file.arrayBuffer(), file.name, "en", "md"); + +// A scanned PDF: feed rasterized pages in order (2 px/point = the native +// pipeline's RENDER_SCALE), then finish. +await ocr.startDoc("en", /* TableFormer */ false); +for (const page of pages) await ocr.addPage(page.rgba, page.w, page.h, 2.0); +const doc = ocr.finishDoc(file.name, "md"); +``` + +Models resolve **device file → local `./models/` → Hugging Face**, so a page can +ship with no models and still work: `ocr.setProvidedModels({ "layout_heron_int8.onnx": buf })` +takes files the user picked, and anything not provided is fetched. ## Build @@ -49,15 +96,20 @@ wasm-bindgen --target web --out-dir crates/docling-wasm/www/pkg \ ## Demo -[`www/index.html`](./www/index.html) is a drop-a-file demo page over the -module (output selector, conversion timing, automated-test hook). After the -`wasm-bindgen` step above: +[`www/index.html`](./www/index.html) is the whole thing on one page: drop a +file, pick the output (Markdown / JSON / DocLang), pick how images render, and +optionally turn on OCR for scanned pages. After the `wasm-bindgen` step above: ```bash python3 -m http.server -d crates/docling-wasm/www 8901 # open http://127.0.0.1:8901/ ``` +It is a plain static page — copy `www/` behind any web server (or into a mobile +app's webview) and it works as-is. The OCR half loads lazily, so a visitor who +only converts a DOCX never downloads ONNX Runtime, pdf.js or any model. PDFs +try their text layer first and fall back to OCR only when there isn't one. + Verified end-to-end in headless Chromium: Markdown/DOCX→md, DOCX→JSON, a corpus PDF→md through the text-layer path, and the scanned-PDF error path all exercised through the real wasm module. @@ -113,10 +165,9 @@ Stage 3 is the same but `conv.addPageTf(rgba, w, h, 2.0, layout, rec, tf)`, where `tf` is a stateful session over the three TableFormer graphs (the heavy cross-attention K/V and the growing decoder KV-cache stay on the JS side, so each decode step marshals only the last tag and gets back logits+hidden — see -[`www/pipeline.js`](./www/pipeline.js)'s `JsTfSession`). The demos wire all of -this up: [`www/ocr.html`](./www/ocr.html) (stage 1), -[`www/scan.html`](./www/scan.html) (stages 2–3, pdf.js + a TableFormer toggle + -a device model-file picker). +[`www/pipeline.js`](./www/pipeline.js)'s `JsTfSession`). +[`www/index.html`](./www/index.html) wires all three stages up behind the OCR +language selector, the TableFormer toggle and the model picker. ## Run it on your phone @@ -136,8 +187,8 @@ same-origin, from a CORS host, **or straight from files on your device**. Pages or githack.) Build the wasm `pkg/` first (see **Build** above). 3. **Provide the models to the page**, either: - - **from your device** — tap the *"Models from device"* picker in - `scan.html` and select `layout_heron_int8.onnx` and, for TableFormer, + - **from your device** — tap the model picker under *"Scanned PDFs and + images"* and select `layout_heron_int8.onnx` and, for TableFormer, `encoder.onnx`, `decoder_kv.onnx` (+`.data`), `bbox.onnx` (+`.data`). Each file is read to an `ArrayBuffer` on the client and used directly — no download, and a single allocation per file (a 225 MB encoder over diff --git a/crates/docling-wasm/src/lib.rs b/crates/docling-wasm/src/lib.rs index c80f37fc..60cb52c3 100644 --- a/crates/docling-wasm/src/lib.rs +++ b/crates/docling-wasm/src/lib.rs @@ -16,7 +16,7 @@ //! const md = convert(new Uint8Array(await file.arrayBuffer()), file.name, "md"); //! ``` -use docling::{DocumentConverter, InputFormat, SourceDocument}; +use docling::{DocumentConverter, ImageMode, InputFormat, SourceDocument}; use wasm_bindgen::prelude::*; #[cfg(feature = "ocr")] @@ -39,7 +39,12 @@ fn start() { /// The whole conversion body, host-testable (`JsError` can only be /// constructed on the wasm target, so the JS boundary stays a thin shim). -fn convert_impl(bytes: &[u8], filename: &str, to: Option<&str>) -> Result { +fn convert_impl( + bytes: &[u8], + filename: &str, + to: Option<&str>, + images: Option<&str>, +) -> Result { let ext = filename.rsplit('.').next().unwrap_or_default(); let format = InputFormat::from_extension(ext) .ok_or_else(|| format!("unknown or unsupported extension: {filename:?}"))?; @@ -47,8 +52,15 @@ fn convert_impl(bytes: &[u8], filename: &str, to: Option<&str>) -> Result Ok(result.document.export_to_markdown()), + // `Referenced` is deliberately unreachable here: it hands the caller + // loose image files to write next to the Markdown, which a page with no + // filesystem cannot do — the browser equivalent is `embedded`. + "md" | "markdown" => Ok(result + .document + .export_to_markdown_with_images(image_mode, "artifacts") + .0), "json" => Ok(result.document.export_to_json()), "doclang" => Ok(result.document.export_to_doclang()), other => Err(format!( @@ -57,13 +69,36 @@ fn convert_impl(bytes: &[u8], filename: &str, to: Option<&str>) -> Result`) or `embedded` +/// (`![Image](data:…;base64,…)`, self-contained — the only way to carry pixels +/// out of a page that cannot write files). +fn image_mode(images: Option<&str>) -> Result { + match images.unwrap_or("placeholder") { + "placeholder" => Ok(ImageMode::Placeholder), + "embedded" => Ok(ImageMode::Embedded), + other => Err(format!( + "unknown images={other:?} (expected \"placeholder\" or \"embedded\")" + )), + } +} + /// Convert a document (as bytes + filename, the extension drives format /// detection) to `to`: `"md"` (Markdown, default), `"json"` (docling-core's /// `DoclingDocument` wire format, schema 1.10.0) or `"doclang"` (docling's /// DocLang XML serialization). +/// +/// `images` controls how pictures render in Markdown — `"placeholder"` +/// (default) or `"embedded"` (base64 data URIs), the same option +/// docling-serve exposes. #[wasm_bindgen] -pub fn convert(bytes: &[u8], filename: &str, to: Option) -> Result { - convert_impl(bytes, filename, to.as_deref()).map_err(|e| JsError::new(&e)) +pub fn convert( + bytes: &[u8], + filename: &str, + to: Option, + images: Option, +) -> Result { + convert_impl(bytes, filename, to.as_deref(), images.as_deref()).map_err(|e| JsError::new(&e)) } /// The file extensions this build can convert, as a JSON string array — @@ -98,16 +133,16 @@ mod tests { #[test] fn markdown_roundtrip() { let md = b"# Title\n\nHello *world*\n"; - let out = convert_impl(md, "note.md", None).unwrap(); + let out = convert_impl(md, "note.md", None, None).unwrap(); assert!(out.contains("# Title")); - let json = convert_impl(md, "note.md", Some("json")).unwrap(); + let json = convert_impl(md, "note.md", Some("json"), None).unwrap(); assert!(json.contains("\"schema_name\"")); } #[test] fn ml_formats_rejected() { // Images still need the full ML pipeline. - let err = convert_impl(&[0x89, b'P', b'N', b'G'], "scan.png", None).unwrap_err(); + let err = convert_impl(&[0x89, b'P', b'N', b'G'], "scan.png", None, None).unwrap_err(); assert!( err.contains("unknown or unsupported") || err.contains("pdf"), "should reject the ML-only format: {err}" @@ -129,7 +164,7 @@ mod tests { "/../../tests/data/pdf/sources/code_and_formula.pdf" )) .expect("corpus pdf"); - let out = convert_impl(&bytes, "code_and_formula.pdf", None).unwrap(); + let out = convert_impl(&bytes, "code_and_formula.pdf", None, None).unwrap(); assert!(!out.trim().is_empty(), "text layer should extract"); } @@ -141,7 +176,7 @@ mod tests { if docling::PDF_ML_COMPILED { return; } - let err = convert_impl(b"%PDF-1.4\n%%EOF", "scan.pdf", None).unwrap_err(); + let err = convert_impl(b"%PDF-1.4\n%%EOF", "scan.pdf", None, None).unwrap_err(); assert!( err.contains("text layer") || err.contains("OCR"), "should point at the missing text layer: {err}" @@ -156,10 +191,29 @@ mod tests { "/../docling/tests/data/docx/sources/docx_lists.docx" )) .expect("corpus docx"); - let out = convert_impl(&bytes, "docx_lists.docx", None).unwrap(); + let out = convert_impl(&bytes, "docx_lists.docx", None, None).unwrap(); assert!(!out.trim().is_empty()); } + /// `images=embedded` inlines picture bytes as data URIs (docling-serve's + /// option, and the only way a page with no filesystem can carry pixels + /// out); the default stays docling's `` placeholder. + #[test] + fn embedded_images_inline_as_data_uris() { + let bytes = std::fs::read(concat!( + env!("CARGO_MANIFEST_DIR"), + "/../docling/tests/data/docx/sources/word_image_anchors.docx" + )) + .expect("corpus docx with images"); + let placeholder = convert_impl(&bytes, "word_image_anchors.docx", None, None).unwrap(); + assert!(placeholder.contains(""), "{placeholder}"); + let embedded = + convert_impl(&bytes, "word_image_anchors.docx", None, Some("embedded")).unwrap(); + assert!(embedded.contains("](data:image/"), "expected a data URI"); + let err = convert_impl(&bytes, "word_image_anchors.docx", None, Some("nope")).unwrap_err(); + assert!(err.contains("unknown images="), "{err}"); + } + #[test] fn extensions_json_parses() { let v: Vec = serde_json::from_str(&supported_extensions()).unwrap(); diff --git a/crates/docling-wasm/src/ocr.rs b/crates/docling-wasm/src/ocr.rs index 3b0d030e..234d4803 100644 --- a/crates/docling-wasm/src/ocr.rs +++ b/crates/docling-wasm/src/ocr.rs @@ -27,7 +27,7 @@ //! }, //! }); //! ``` -//! (see `www/ocr.html` for the complete wiring, including output-name +//! (see `www/index.html` for the complete wiring, including output-name //! discovery and model/dict caching.) use docling_core::{DoclingDocument, Node}; diff --git a/crates/docling-wasm/src/scanned.rs b/crates/docling-wasm/src/scanned.rs index 6e322737..ac5c3c7d 100644 --- a/crates/docling-wasm/src/scanned.rs +++ b/crates/docling-wasm/src/scanned.rs @@ -18,7 +18,7 @@ //! } //! const markdown = conv.finish("scan.pdf", "md"); //! ``` -//! (`www/scan.html` is the complete wiring.) +//! (`www/index.html` is the complete wiring.) use docling_pdf::layout::{decode_layout, layout_input, SIDE}; use docling_pdf::ocr_prep::{ @@ -33,14 +33,6 @@ use wasm_bindgen::prelude::*; use crate::ocr::{tensor_parts, RecSession}; use crate::tableformer::TfSession; -#[wasm_bindgen] -extern "C" { - // On-page diagnostic sink (defined by the host — worker.js forwards it to - // the main thread, which shows it; console isn't reachable on a phone). - #[wasm_bindgen(js_name = __docling_diag)] - fn diag(s: &str); -} - #[wasm_bindgen] extern "C" { /// The JS-side layout session: a wrapper around an `ort.InferenceSession` @@ -188,19 +180,6 @@ impl ScannedConverter { let regions = decode_layout(&logits, &boxes, q, c, page_w, page_h); let regions = refine_regions(regions, &[], page_w, page_h); - // Diagnostic: the region-label histogram, so it's visible (browser - // console) whether the layout model flagged any `table` region on this - // page — tables only render when it does (geometric or TableFormer). - { - let mut hist: std::collections::BTreeMap<&str, usize> = - std::collections::BTreeMap::new(); - for r in ®ions { - *hist.entry(r.label).or_default() += 1; - } - let summary: Vec = hist.iter().map(|(l, n)| format!("{l}×{n}")).collect(); - diag(&format!("layout regions: {}", summary.join(", "))); - } - // OCR the text regions (same gather/batch/decode as native ocr_page). let (bboxes, lines) = prep_region_lines(&img, ®ions, scale); let texts = self.ocr_lines(rec, &lines).await?; @@ -268,16 +247,26 @@ impl ScannedConverter { } /// Assemble the accumulated pages into the final document and render it - /// as `"md"` (default) or `"json"`. Resets the converter. + /// as `"md"` (default), `"json"` or `"doclang"` — the same three the + /// declarative [`crate::convert`] entry point offers. Resets the converter. pub fn finish(&mut self, name: &str, to: Option) -> Result { let doc = finish_document(name, std::mem::take(&mut self.pages)); - match to.as_deref().unwrap_or("md") { - "md" | "markdown" => Ok(doc.export_to_markdown()), - "json" => Ok(doc.export_to_json()), - other => Err(JsError::new(&format!( - "unknown output format {other:?} (expected \"md\" or \"json\")" - ))), - } + render(&doc, to.as_deref()) + } +} + +/// Render an assembled document in one of the three output grammars. The OCR +/// pipeline recovers no picture *pixels* (regions are recognized, not cropped +/// out), so Markdown always uses docling's `` placeholder — there +/// is nothing to embed, unlike the declarative path's `images` option. +fn render(doc: &docling_core::DoclingDocument, to: Option<&str>) -> Result { + match to.unwrap_or("md") { + "md" | "markdown" => Ok(doc.export_to_markdown()), + "json" => Ok(doc.export_to_json()), + "doclang" => Ok(doc.export_to_doclang()), + other => Err(JsError::new(&format!( + "unknown output format {other:?} (expected \"md\", \"json\" or \"doclang\")" + ))), } } diff --git a/crates/docling-wasm/src/tableformer.rs b/crates/docling-wasm/src/tableformer.rs index 4b23862d..bd65ca42 100644 --- a/crates/docling-wasm/src/tableformer.rs +++ b/crates/docling-wasm/src/tableformer.rs @@ -9,7 +9,7 @@ //! encoder's constant cross-attention K/V and `enc_out`, and the growing //! decoder KV-cache, entirely on the JS side. Each decode step sends only the //! last tag (one int) and gets back `logits` + `hidden` (525 floats). See -//! `www/scan.html` for the JS wiring (the `decoder_kv` graph: N_LAYERS=6, +//! `www/index.html` for the JS wiring (the `decoder_kv` graph: N_LAYERS=6, //! KV_HEADS=8, head_dim=64, cross length 784). use docling_pdf::pdfium_backend::TextCell; diff --git a/crates/docling-wasm/www/index.html b/crates/docling-wasm/www/index.html index 169a9b6c..20521204 100644 --- a/crates/docling-wasm/www/index.html +++ b/crates/docling-wasm/www/index.html @@ -3,92 +3,492 @@ - docling.rs — in-browser document conversion (wasm) + + + docling.rs — document conversion in your browser (wasm) + + -

docling.rs → wasm: convert documents in your browser

-

- DOCX / HTML / Markdown / XLSX / PPTX / CSV / EPUB / ODF / LaTeX / - digital PDF / … are converted to Markdown or docling JSON - entirely client-side — the file never leaves this page. - PDFs convert via their embedded text layer (flat paragraphs, like the - native --no-ocr); scanned PDFs, images and audio need the - native ML pipeline and are not part of the wasm build. +

docling.rs in your browser

+

+ Documents convert to Markdown, docling JSON or DocLang XML + entirely client-side — nothing is uploaded, no server is + involved. Everything below runs from this one page.

+ +
Drop a file here, or click to choose
+ +
-
-
Drop a document here, or click to pick a file
- + +
+ + + scanned pages need models — see below +
+ +

 
+  

What converts without any setup

+

+ DOCX, PPTX, XLSX, HTML, Markdown, CSV, AsciiDoc, EPUB, ODF, LaTeX, JATS, + email, MHTML, JSON, DocLang — and PDFs that carry a text layer + (the same extraction the native --no-ocr flag does). These need + no models and no network: the converters are compiled into the wasm module. +

+ +

Scanned PDFs and images (optional models)

+

+ A scanned page has no text layer, so it needs the ML pipeline: RT-DETR + layout detection + PP-OCR recognition (+ TableFormer for tables), running on + ONNX Runtime Web + with all pre/post-processing in Rust/wasm — the same code as the native + pipeline. Provide the models any one of these three ways: +

+
    +
  1. + From your device — pick the .onnx files: + +
    Layout needs layout_heron_int8.onnx; TableFormer also needs + encoder.onnx, decoder_kv.onnx + .data, + bbox.onnx + .data. Each file is read + locally — no download, and it avoids the memory spike a 225 MB fetch + causes on a phone. +
  2. +
  3. + From a models/ folder next to this page — + served same-origin, fetched automatically: +
    cd crates/docling-wasm/www/
    +curl -fsSL https://raw.githubusercontent.com/docling-project/docling.rs/master/scripts/install/download_dependencies.sh | sh
    +
  4. +
  5. + From Hugging Face — used automatically when neither of the + above has the file. (GitHub Release assets can't be fetched cross-origin: + they send no Access-Control-Allow-Origin.) +
  6. +
+

+ Resolution order per file is device → local models/ → + Hugging Face, so any mix works. The recognition model (~10 MB) + streams from Hugging Face on demand; layout is ~68 MB and TableFormer + ~380 MB, so on mobile prefer picking files from the device. +

+ diff --git a/crates/docling-wasm/www/ocr.html b/crates/docling-wasm/www/ocr.html deleted file mode 100644 index 7716c8cd..00000000 --- a/crates/docling-wasm/www/ocr.html +++ /dev/null @@ -1,219 +0,0 @@ - - - - - - - - docling.rs — in-browser OCR (wasm + ONNX Runtime Web) - - - -

docling.rs → wasm: OCR a scanned image in your browser

-

- Stage 1 of #157: - the PP-OCRv3 recognition model (~10 MB) runs via - ONNX Runtime Web; - line segmentation and CTC decoding run in Rust/wasm — the same code as the - native pipeline. The image never leaves this page. The model + dictionary - are fetched once and cached by the browser. Single-column scans work best - (layout detection is stage 2). -

-
drop a scanned image here (PNG/JPEG/TIFF/WebP) or click to pick
- -

- Recognition language: - - -

- -
- - - - diff --git a/crates/docling-wasm/www/pipeline.js b/crates/docling-wasm/www/pipeline.js index c6d54015..b0d497f4 100644 --- a/crates/docling-wasm/www/pipeline.js +++ b/crates/docling-wasm/www/pipeline.js @@ -100,7 +100,7 @@ export function createOcr({ onStatus }) { const status = (msg, spinning = true) => onStatus && onStatus(msg, spinning); // Model files the user picked from the device (basename → ArrayBuffer), - // used instead of any network fetch — see scan.html's file picker. Reading a + // used instead of any network fetch — see index.html's model picker. Reading a // File to an ArrayBuffer is a single allocation, so it also sidesteps the // double-buffering peak fetchProgress hits on the 225 MB encoder. let provided = {}; @@ -269,16 +269,16 @@ export function createOcr({ onStatus }) { await cur.conv.add_page(rgba, w, h, scale, layout, cur.rec); } } - function finishDoc(name) { - const md = cur.conv.finish(name, "md"); + function finishDoc(name, to) { + const md = cur.conv.finish(name, to || "md"); cur = null; return md; } // Standalone image: the wasm side decodes it (no canvas needed). - async function convertImage(bytes, name, lang) { + async function convertImage(bytes, name, lang, to) { const { dict, rec } = await recFor(lang); - return convert_scanned_image(new Uint8Array(bytes), name, dict, layout, rec, "md"); + return convert_scanned_image(new Uint8Array(bytes), name, dict, layout, rec, to || "md"); } return { diff --git a/crates/docling-wasm/www/scan.html b/crates/docling-wasm/www/scan.html deleted file mode 100644 index 48623064..00000000 --- a/crates/docling-wasm/www/scan.html +++ /dev/null @@ -1,347 +0,0 @@ - - - - - - - - docling.rs — scanned PDFs in the browser (lite profile) - - - -

docling.rs → wasm: scanned PDF / image, full lite profile

-

- Stage 2 of #157: - RT-DETR layout detection + PP-OCRv3 recognition via ONNX Runtime Web; - pdf.js rasterizes PDF pages at the native pipeline's 2 px/point; - region refinement, OCR and reading-order assembly run in Rust/wasm — the - same code as the native pipeline. Tables come out via the geometric - reconstruction (TableFormer is stage 3). Nothing leaves this page. -

-
- The layout model is served from this page's own origin — fetch it once into ./models/ -

- Unlike the recognition model (fetched from Hugging Face, which sends CORS - headers), the RT-DETR layout export lives on this repo's GitHub Release, - and GitHub Release assets can't be fetched cross-origin from a - browser (no Access-Control-Allow-Origin). So this - page loads the layout model same-origin, from a - models/ directory next to it. Run the installer from - this www/ directory — it writes ./models/ - relative to the current directory: -

-
cd crates/docling-wasm/www/
-curl -fsSL https://raw.githubusercontent.com/docling-project/docling.rs/master/scripts/install/download_dependencies.sh | sh
-

- That populates ./models/ (this page reads - ./models/layout_heron_int8.onnx, ~68 MB, falling back to - ./models/layout_heron.onnx, ~172 MB — int8 needs a - release that hosts it, otherwise fp32 is fetched). These are the exact - conformance-validated models the native pipeline uses; both decode - identically here (same pixel_values → logitspred_boxes - contract). The recognition model still streams from Hugging Face on demand. -

-
-
drop a scanned PDF or image here, or click to pick
- -

- Recognition language: - - - -

-

- Models from device (optional — skips the download): - -
pick layout_heron_int8.onnx and, for TableFormer, encoder.onnx, - decoder_kv.onnx(+.data), bbox.onnx(+.data). -

- -
-
- - - - diff --git a/crates/docling-wasm/www/worker.js b/crates/docling-wasm/www/worker.js index f4dffe9b..97dbb41b 100644 --- a/crates/docling-wasm/www/worker.js +++ b/crates/docling-wasm/www/worker.js @@ -7,17 +7,14 @@ // // RPC: every request carries an id; the reply is {type:"ok", id, ...data} or // {type:"error", id, msg}. Progress is a broadcast {type:"status", msg, -// spinning}. Requests: boot{lang} | rec{lang} | doc-start{lang} | -// doc-page{rgba,w,h,scale} | doc-finish{name} | convert-image{bytes,name,lang}. +// spinning}. Requests: set-models{models} | boot{lang} | rec{lang} | +// doc-start{lang,useTf} | doc-page{rgba,w,h,scale} | doc-finish{name,to} | +// convert-image{bytes,name,lang,to}. import { createOcr, THREADS } from "./pipeline.js"; const post = (type, extra) => self.postMessage({ type, ...extra }); -// On-page diagnostic sink the wasm calls (see src/scanned.rs) — forward to the -// main thread, which renders it (a phone can't open the console). -globalThis.__docling_diag = (msg) => post("diag", { msg }); - const ocr = createOcr({ onStatus: (msg, spinning) => post("status", { msg, spinning }), }); @@ -45,9 +42,9 @@ async function handle(m) { await ocr.addPage(new Uint8Array(m.rgba), m.w, m.h, m.scale); return {}; case "doc-finish": - return { md: ocr.finishDoc(m.name) }; + return { md: ocr.finishDoc(m.name, m.to) }; case "convert-image": - return { md: await ocr.convertImage(m.bytes, m.name, m.lang) }; + return { md: await ocr.convertImage(m.bytes, m.name, m.lang, m.to) }; default: throw new Error(`unknown request ${m.type}`); } From 753057846aed04a7742ae5844295596466e86e59 Mon Sep 17 00:00:00 2001 From: artiz Date: Sun, 26 Jul 2026 19:30:31 +0000 Subject: [PATCH 02/14] wasm: crop pictures on the OCR path so images=embedded has real bytes (#157) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `images=embedded` did nothing for scanned input: the browser pipeline built its page with `PdfPage::from_cells`, which carries no bitmap, so every picture region came out as `` with no pixels to inline — the option looked broken to anyone converting a scan. The native pipeline already crops each `picture` region out of the rendered page; only the feature gate stopped the browser from doing the same. Since `ocr-prep` is a subset of `ml`, moving the page-bitmap field, its import and `crop_region` from `ml` to `ocr-prep` gives the wasm build the identical code path and leaves both the native (`ml`) and text-layer-only (no `ocr-prep`, no field, no crop) builds exactly as they were. The browser hands its rasterized page over through the new `PdfPage::from_cells_with_image`, and the `images` choice is threaded through `ScannedConverter::finish` / `convert_scanned_image` and the JS layers, so both conversion paths now offer the same output grammars *and* the same picture handling. Verified: a new unit test builds a page from a host-supplied bitmap and asserts the assembled picture carries a PNG cropped to region × scale (100x60); the native docling/docling-pdf/docling-core suites (117 tests) stay green, and the declarative path re-checked in headless Chromium (placeholder vs embedded, md/json/doclang, preview, text-layer PDF). Refs #157 Signed-off-by: artiz Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT --- crates/docling-pdf/src/assemble.rs | 49 ++++++++++++++++++++++-- crates/docling-pdf/src/pdfium_backend.rs | 27 +++++++++++-- crates/docling-pdf/src/textparse.rs | 2 +- crates/docling-wasm/README.md | 4 ++ crates/docling-wasm/src/scanned.rs | 45 +++++++++++++++------- crates/docling-wasm/www/index.html | 14 +++---- crates/docling-wasm/www/pipeline.js | 10 +++-- crates/docling-wasm/www/worker.js | 8 ++-- 8 files changed, 124 insertions(+), 35 deletions(-) diff --git a/crates/docling-pdf/src/assemble.rs b/crates/docling-pdf/src/assemble.rs index 916c14cc..8d39ce2c 100644 --- a/crates/docling-pdf/src/assemble.rs +++ b/crates/docling-pdf/src/assemble.rs @@ -1043,7 +1043,7 @@ pub fn crop_region_scaled(page: &PdfPage, bbox: [f32; 4], target_scale: f32) -> /// Crop a layout region from the rendered page image and encode it as PNG (the /// figure bytes docling stores on a `PictureItem`). Region coordinates are page /// points; the image is rendered at `page.scale`. -#[cfg(feature = "ml")] +#[cfg(feature = "ocr-prep")] fn crop_region(page: &PdfPage, region: &Region) -> Option { let s = page.scale; let (iw, ih) = (page.image.width(), page.image.height()); @@ -1305,9 +1305,9 @@ pub fn assemble_page( }; // Without the page render (text-layer-only build) a picture keeps // its caption/classification but carries no cropped pixels. - #[cfg(feature = "ml")] + #[cfg(feature = "ocr-prep")] let image = crate::timing::timed("crop_region", || crop_region(page, region)); - #[cfg(not(feature = "ml"))] + #[cfg(not(feature = "ocr-prep"))] let image: Option = None; nodes.push(located( loc, @@ -1657,6 +1657,49 @@ mod tests { use crate::pdfium_backend::{LinkAnnot, PdfPage, TextCell}; use docling_core::Node; + /// A `picture` region is cropped out of the rendered page, whatever built + /// that page. The browser pipeline (#157) has no pdfium but does hand over + /// the rasterized bitmap through `from_cells_with_image`, so it must get + /// the same figure bytes the native path does — that is what makes + /// `images = "embedded"` inline real pixels instead of a placeholder. + #[cfg(feature = "ocr-prep")] + #[test] + fn picture_regions_are_cropped_from_a_host_supplied_page_image() { + let mut img = image::RgbImage::new(200, 200); + // Paint the figure area so the crop is distinguishable from the page. + for y in 100..160 { + for x in 20..120 { + img.put_pixel(x, y, image::Rgb([255, 0, 0])); + } + } + // scale 2.0: the region is in page points, the bitmap in pixels. + let page = PdfPage::from_cells_with_image(100.0, 100.0, 2.0, Vec::new(), img); + let region = Region { + label: "picture", + score: 0.9, + l: 10.0, + t: 50.0, + r: 60.0, + b: 80.0, + }; + let (nodes, _) = super::assemble_page(&page, vec![region], &[None], &[None]); + // Layout-derived nodes carry provenance, so the picture arrives wrapped. + let image = nodes + .iter() + .find_map(|n| match n { + Node::Located { inner, .. } => match &**inner { + Node::Picture { image, .. } => image.as_ref(), + _ => None, + }, + Node::Picture { image, .. } => image.as_ref(), + _ => None, + }) + .expect("a picture node with cropped pixels"); + assert_eq!(image.mimetype, "image/png"); + assert_eq!((image.width, image.height), (100, 60), "region × scale"); + assert!(!image.data.is_empty(), "PNG bytes were encoded"); + } + #[test] fn link_anchors_split_a_shared_word_cell_between_adjacent_links() { // A common header layout: one text run holds several pipe-separated diff --git a/crates/docling-pdf/src/pdfium_backend.rs b/crates/docling-pdf/src/pdfium_backend.rs index f2d8a932..612eb9ad 100644 --- a/crates/docling-pdf/src/pdfium_backend.rs +++ b/crates/docling-pdf/src/pdfium_backend.rs @@ -9,7 +9,7 @@ //! loop is driven through the raw `PdfiumLibraryBindings` FFI on a second handle //! to the same bytes (no fork; stays publishable). -#[cfg(feature = "ml")] +#[cfg(feature = "ocr-prep")] use image::RgbImage; #[cfg(feature = "ml")] use pdfium_render::prelude::*; @@ -46,7 +46,11 @@ pub struct PdfPage { /// Per-word cells (one per word, not joined into lines) for TableFormer cell /// matching. pub word_cells: Vec, - #[cfg(feature = "ml")] + /// The rendered page bitmap. Present whenever pixels are available at all + /// (`ocr-prep` ⊂ `ml`): the native pipeline renders it with pdfium, the + /// browser pipeline receives it from the host canvas. Picture regions are + /// cropped out of it. + #[cfg(feature = "ocr-prep")] pub image: RgbImage, /// Hyperlink annotations on the page (rect in top-left page coords + target /// URI), restricted to web/mail/tel schemes. Used only by strict Markdown. @@ -68,11 +72,28 @@ impl PdfPage { cells, code_cells: Vec::new(), word_cells: Vec::new(), - #[cfg(feature = "ml")] + #[cfg(feature = "ocr-prep")] image: RgbImage::new(0, 0), links: Vec::new(), } } + + /// Same as [`from_cells`](Self::from_cells) but carrying the rendered page + /// bitmap, so picture regions can be cropped out of it (#157: the browser + /// pipeline gets the same figure bytes the native one does). + #[cfg(feature = "ocr-prep")] + pub fn from_cells_with_image( + width: f32, + height: f32, + scale: f32, + cells: Vec, + image: RgbImage, + ) -> Self { + Self { + image, + ..Self::from_cells(width, height, scale, cells) + } + } } /// A PDF link annotation: its rectangle (top-left page coordinates, matching diff --git a/crates/docling-pdf/src/textparse.rs b/crates/docling-pdf/src/textparse.rs index 040ee09f..3da21342 100644 --- a/crates/docling-pdf/src/textparse.rs +++ b/crates/docling-pdf/src/textparse.rs @@ -757,7 +757,7 @@ pub fn pdf_text_pages(bytes: &[u8]) -> Vec { cells: prose, code_cells: crate::pdfium_backend::code_cells_from_glyphs(&glyphs, h), word_cells: words, - #[cfg(feature = "ml")] + #[cfg(feature = "ocr-prep")] image: image::RgbImage::new(1, 1), links: Vec::new(), } diff --git a/crates/docling-wasm/README.md b/crates/docling-wasm/README.md index 623ff1b9..8d75c22c 100644 --- a/crates/docling-wasm/README.md +++ b/crates/docling-wasm/README.md @@ -76,6 +76,10 @@ for (const page of pages) await ocr.addPage(page.rgba, page.w, page.h, 2.0); const doc = ocr.finishDoc(file.name, "md"); ``` +Scanned pictures are cropped out of the rendered page just like the native +pipeline, so `images: "embedded"` inlines real figure bytes on this path too +(`ocr.finishDoc(name, "md", "embedded")`). + Models resolve **device file → local `./models/` → Hugging Face**, so a page can ship with no models and still work: `ocr.setProvidedModels({ "layout_heron_int8.onnx": buf })` takes files the user picked, and anything not provided is fetched. diff --git a/crates/docling-wasm/src/scanned.rs b/crates/docling-wasm/src/scanned.rs index ac5c3c7d..20bf0838 100644 --- a/crates/docling-wasm/src/scanned.rs +++ b/crates/docling-wasm/src/scanned.rs @@ -4,7 +4,9 @@ //! reconstruction and reading-order assembly in Rust — the same code as the //! native pipeline (`docling_pdf::{layout, ocr_prep, scanned}`). TableFormer //! and the enrichment models are stage 3; table regions fall back to the -//! geometric reconstruction the native `--no-table-former` flag uses. +//! geometric reconstruction the native `--no-table-former` flag uses. Picture +//! regions are cropped out of the page bitmap, like the native pipeline, so +//! `images = "embedded"` inlines real figure bytes. //! //! Pages arrive as raw RGBA bitmaps: for a scanned PDF the host page renders //! them with pdf.js (`page.getViewport({scale: 2})` — 2 px per PDF point, @@ -16,7 +18,7 @@ //! for (const bitmap of pages) { //! await conv.add_page(bitmap.data, bitmap.width, bitmap.height, 2.0, layout, rec); //! } -//! const markdown = conv.finish("scan.pdf", "md"); +//! const markdown = conv.finish("scan.pdf", "md", "embedded"); //! ``` //! (`www/index.html` is the complete wiring.) @@ -232,7 +234,10 @@ impl ScannedConverter { Vec::new() // assemble_page_with_tables resizes to all-None }; - let page = PdfPage::from_cells(page_w, page_h, scale, cells); + // Hand the page bitmap over: assemble crops each `picture` region out + // of it, so the browser pipeline produces the same figure bytes the + // native one does (`images=embedded` then inlines them). + let page = PdfPage::from_cells_with_image(page_w, page_h, scale, cells, img); self.pages .push(assemble_page_with_tables(&page, regions, table_rows)); Ok(()) @@ -248,20 +253,33 @@ impl ScannedConverter { /// Assemble the accumulated pages into the final document and render it /// as `"md"` (default), `"json"` or `"doclang"` — the same three the - /// declarative [`crate::convert`] entry point offers. Resets the converter. - pub fn finish(&mut self, name: &str, to: Option) -> Result { + /// declarative [`crate::convert`] entry point offers. `images` picks how + /// cropped figures render in Markdown (`"placeholder"` | `"embedded"`), + /// like [`crate::convert`]. Resets the converter. + pub fn finish( + &mut self, + name: &str, + to: Option, + images: Option, + ) -> Result { let doc = finish_document(name, std::mem::take(&mut self.pages)); - render(&doc, to.as_deref()) + render(&doc, to.as_deref(), images.as_deref()) } } -/// Render an assembled document in one of the three output grammars. The OCR -/// pipeline recovers no picture *pixels* (regions are recognized, not cropped -/// out), so Markdown always uses docling's `` placeholder — there -/// is nothing to embed, unlike the declarative path's `images` option. -fn render(doc: &docling_core::DoclingDocument, to: Option<&str>) -> Result { +/// Render an assembled document in one of the three output grammars, with the +/// same `images` choice the declarative path offers — picture regions are +/// cropped out of the rendered page, so `embedded` has real bytes to inline. +fn render( + doc: &docling_core::DoclingDocument, + to: Option<&str>, + images: Option<&str>, +) -> Result { match to.unwrap_or("md") { - "md" | "markdown" => Ok(doc.export_to_markdown()), + "md" | "markdown" => { + let mode = crate::image_mode(images).map_err(|e| JsError::new(&e))?; + Ok(doc.export_to_markdown_with_images(mode, "artifacts").0) + } "json" => Ok(doc.export_to_json()), "doclang" => Ok(doc.export_to_doclang()), other => Err(JsError::new(&format!( @@ -281,6 +299,7 @@ pub async fn convert_scanned_image( layout: &LayoutSession, rec: &RecSession, to: Option, + images: Option, ) -> Result { let img = image::load_from_memory(bytes) .map_err(|e| JsError::new(&format!("decode image: {e}")))? @@ -288,7 +307,7 @@ pub async fn convert_scanned_image( let (w, h) = img.dimensions(); let mut conv = ScannedConverter::new(dict); conv.add_page(img.as_raw(), w, h, 1.0, layout, rec).await?; - conv.finish(name, to) + conv.finish(name, to, images) } // Silence the unused warning for SIDE re-export path (the JS side sizes its diff --git a/crates/docling-wasm/www/index.html b/crates/docling-wasm/www/index.html index 20521204..82a13c15 100644 --- a/crates/docling-wasm/www/index.html +++ b/crates/docling-wasm/www/index.html @@ -231,9 +231,9 @@

Scanned PDFs and images (optional models)

rec: (l) => rpc("rec", { lang: l }), docStart: (l, useTf) => rpc("doc-start", { lang: l, useTf }), addPage: (rgba, w, h, scale) => rpc("doc-page", { rgba, w, h, scale }, [rgba]), - docFinish: (name, fmt) => rpc("doc-finish", { name, to: fmt }).then((r) => r.md), - convertImage: (bytes, name, l, fmt) => - rpc("convert-image", { bytes, name, lang: l, to: fmt }, [bytes]).then((r) => r.md), + docFinish: (name, fmt, img) => rpc("doc-finish", { name, to: fmt, images: img }).then((r) => r.md), + convertImage: (bytes, name, l, fmt, img) => + rpc("convert-image", { bytes, name, lang: l, to: fmt, images: img }, [bytes]).then((r) => r.md), setModels: (m) => rpc("set-models", { models: m }, Object.values(m)), }; } @@ -253,8 +253,8 @@

Scanned PDFs and images (optional models)

rec: async (l) => { await p.recFor(l); await p.warmup(l); return {}; }, docStart: (l, useTf) => p.startDoc(l, useTf), addPage: (rgba, w, h, scale) => p.addPage(new Uint8Array(rgba), w, h, scale), - docFinish: (name, fmt) => Promise.resolve(p.finishDoc(name, fmt)), - convertImage: (bytes, name, l, fmt) => p.convertImage(bytes, name, l, fmt), + docFinish: (name, fmt, img) => Promise.resolve(p.finishDoc(name, fmt, img)), + convertImage: (bytes, name, l, fmt, img) => p.convertImage(bytes, name, l, fmt, img), setModels: (m) => p.setProvidedModels(m), }; } @@ -355,7 +355,7 @@

Scanned PDFs and images (optional models)

await drv.addPage(cur.rgba, cur.w, cur.h, SCALE); elapsed += performance.now() - t; } - return drv.docFinish(name, to.value); + return drv.docFinish(name, to.value, images.value); } async function convertFile(bytes, name) { @@ -363,7 +363,7 @@

Scanned PDFs and images (optional models)

// OCR when there is none (that is exactly what a scanned PDF is). if (isImage(name)) { const drv = await ensureOcr(); - return drv.convertImage(bytes.slice(0).buffer, name, lang.value, to.value); + return drv.convertImage(bytes.slice(0).buffer, name, lang.value, to.value, images.value); } try { return convert(bytes, name, to.value, images.value); diff --git a/crates/docling-wasm/www/pipeline.js b/crates/docling-wasm/www/pipeline.js index b0d497f4..923f1846 100644 --- a/crates/docling-wasm/www/pipeline.js +++ b/crates/docling-wasm/www/pipeline.js @@ -269,16 +269,18 @@ export function createOcr({ onStatus }) { await cur.conv.add_page(rgba, w, h, scale, layout, cur.rec); } } - function finishDoc(name, to) { - const md = cur.conv.finish(name, to || "md"); + function finishDoc(name, to, images) { + const md = cur.conv.finish(name, to || "md", images || "placeholder"); cur = null; return md; } // Standalone image: the wasm side decodes it (no canvas needed). - async function convertImage(bytes, name, lang, to) { + async function convertImage(bytes, name, lang, to, images) { const { dict, rec } = await recFor(lang); - return convert_scanned_image(new Uint8Array(bytes), name, dict, layout, rec, to || "md"); + return convert_scanned_image( + new Uint8Array(bytes), name, dict, layout, rec, to || "md", images || "placeholder", + ); } return { diff --git a/crates/docling-wasm/www/worker.js b/crates/docling-wasm/www/worker.js index 97dbb41b..fc11e3e9 100644 --- a/crates/docling-wasm/www/worker.js +++ b/crates/docling-wasm/www/worker.js @@ -8,8 +8,8 @@ // RPC: every request carries an id; the reply is {type:"ok", id, ...data} or // {type:"error", id, msg}. Progress is a broadcast {type:"status", msg, // spinning}. Requests: set-models{models} | boot{lang} | rec{lang} | -// doc-start{lang,useTf} | doc-page{rgba,w,h,scale} | doc-finish{name,to} | -// convert-image{bytes,name,lang,to}. +// doc-start{lang,useTf} | doc-page{rgba,w,h,scale} | doc-finish{name,to,images} | +// convert-image{bytes,name,lang,to,images}. import { createOcr, THREADS } from "./pipeline.js"; @@ -42,9 +42,9 @@ async function handle(m) { await ocr.addPage(new Uint8Array(m.rgba), m.w, m.h, m.scale); return {}; case "doc-finish": - return { md: ocr.finishDoc(m.name, m.to) }; + return { md: ocr.finishDoc(m.name, m.to, m.images) }; case "convert-image": - return { md: await ocr.convertImage(m.bytes, m.name, m.lang, m.to) }; + return { md: await ocr.convertImage(m.bytes, m.name, m.lang, m.to, m.images) }; default: throw new Error(`unknown request ${m.type}`); } From 70272708bf5e21de10c214575479ced0fcdcff37 Mon Sep 17 00:00:00 2001 From: artiz Date: Sun, 26 Jul 2026 19:49:44 +0000 Subject: [PATCH 03/14] wasm demo: freeze the preview toggle during a run, and stop it reconverting MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two reports, one cause. The preview checkbox was the only control left live while a conversion ran, and its handler was wired straight to run(). Clicking it mid-conversion therefore started a second run on top of the first; the first then reached finishDoc, which clears the pipeline's single in-flight document, and the second dereferenced it — surfacing as "Cannot read properties of null (reading 'tf')". - The busy/ready freeze now covers every control (a named CONTROLS list, so a future addition can't be forgotten again), preview included. - run() takes a re-entrancy guard, so the UI is no longer the only thing standing between us and two concurrent documents. - Toggling preview only re-renders the output we already have. It never reconverted meaningfully before either, but on a scanned document doing so costs another ~8 s per page — now it is free. - addPage/finishDoc reject with a real message when no document is open, instead of a null dereference. Verified in headless Chromium during an actual long-running conversion: all seven controls read disabled, clicking preview while frozen leaves exactly one run in flight, toggling preview afterwards leaves the timing line untouched (no reconversion) and renders the embedded images. The browser also caught a TDZ slip in the first cut of this change (CONTROLS used above its declaration), fixed here. Refs #157 Signed-off-by: artiz Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT --- crates/docling-wasm/www/index.html | 31 ++++++++++++++++++++++------- crates/docling-wasm/www/pipeline.js | 5 +++++ 2 files changed, 29 insertions(+), 7 deletions(-) diff --git a/crates/docling-wasm/www/index.html b/crates/docling-wasm/www/index.html index 82a13c15..bd138e20 100644 --- a/crates/docling-wasm/www/index.html +++ b/crates/docling-wasm/www/index.html @@ -160,6 +160,11 @@

Scanned PDFs and images (optional models)

const [meta, out, preview, drop, pick, to, images, render, lang, tf, modelsInput, ocrState] = ["meta", "out", "preview", "drop", "pick", "to", "images", "render", "lang", "tf", "models", "ocrState"].map($); + // Everything the user can touch, frozen for the whole of a conversion: + // a control left live lets a second run start on top of the first, and the + // two then fight over the pipeline's single in-flight document. + const CONTROLS = [pick, to, images, render, lang, tf, modelsInput]; + await init(); const exts = JSON.parse(supported_extensions()); pick.accept = [...exts, ...IMAGE_EXT].map((e) => "." + e).join(","); @@ -173,12 +178,12 @@

Scanned PDFs and images (optional models)

} function busy(msg) { document.body.classList.add("busy"); - for (const el of [pick, to, images, lang, tf, modelsInput]) el.disabled = true; + for (const el of CONTROLS) el.disabled = true; status(msg, true); } function ready(msg) { document.body.classList.remove("busy"); - for (const el of [pick, to, images, lang, tf, modelsInput]) el.disabled = false; + for (const el of CONTROLS) el.disabled = false; status(msg, false); } @@ -320,7 +325,8 @@

Scanned PDFs and images (optional models)

} // ---- conversion ------------------------------------------------------ - let last = null; // {bytes, name} — re-rendered when an option changes + let last = null; // {bytes, name} — reconverted when an option changes + let lastText = ""; // its rendered output, so previewing costs nothing function isImage(name) { return IMAGE_EXT.includes(name.toLowerCase().split(".").pop()); @@ -376,19 +382,28 @@

Scanned PDFs and images (optional models)

} } + let running = false; async function run() { - if (!last) return; + // The controls are frozen during a run, but never rely on the UI alone: + // a stray re-entry would start a second document while the first is mid + // flight, and the OCR pipeline holds exactly one. + if (!last || running) return; + running = true; out.textContent = ""; preview.style.display = "none"; busy(`converting ${last.name} …`); const t0 = performance.now(); try { const text = await convertFile(last.bytes, last.name); - show(text || "(nothing extracted)"); + lastText = text || "(nothing extracted)"; + show(lastText); ready(`${last.name} → ${to.value} in ${fmtTime(performance.now() - t0)}`); } catch (e) { - show(String((e && e.message) || e)); + lastText = String((e && e.message) || e); + show(lastText); ready(`${last.name} — failed`); + } finally { + running = false; } } @@ -449,7 +464,9 @@

Scanned PDFs and images (optional models)

last = { bytes: new Uint8Array(await file.arrayBuffer()), name: file.name }; run(); } - to.onchange = images.onchange = render.onchange = run; + to.onchange = images.onchange = run; + // Previewing only re-renders what we already have — no reconversion. + render.onchange = () => lastText && show(lastText); // Switching the recognition language reloads that model; re-run if a // scanned document is already on screen. lang.onchange = async () => { diff --git a/crates/docling-wasm/www/pipeline.js b/crates/docling-wasm/www/pipeline.js index 923f1846..cfebee25 100644 --- a/crates/docling-wasm/www/pipeline.js +++ b/crates/docling-wasm/www/pipeline.js @@ -263,6 +263,10 @@ export function createOcr({ onStatus }) { cur = { conv: new ScannedConverter(dict), rec, tf: tfSess }; } async function addPage(rgba, w, h, scale) { + // Guard the single in-flight document: without it a stray addPage/finish + // after (or before) the lifecycle reads `cur` as null and the host sees an + // opaque "Cannot read properties of null". + if (!cur) throw new Error("addPage called with no document open (startDoc first)"); if (cur.tf) { await cur.conv.addPageTf(rgba, w, h, scale, layout, cur.rec, cur.tf); } else { @@ -270,6 +274,7 @@ export function createOcr({ onStatus }) { } } function finishDoc(name, to, images) { + if (!cur) throw new Error("finishDoc called with no document open (startDoc first)"); const md = cur.conv.finish(name, to || "md", images || "placeholder"); cur = null; return md; From 3defa1c200fb0f4dd761457355788a67eff55469 Mon Sep 17 00:00:00 2001 From: artiz Date: Mon, 27 Jul 2026 07:02:26 +0000 Subject: [PATCH 04/14] wasm: run TableFormer only where geometry fails, and publish the demo (#157) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit **Selective TableFormer.** The encoder runs once per table region and costs seconds, but many tables need no model at all. Each table is now first reconstructed geometrically — free, from the OCR cell positions — and TableFormer is invoked only when that grid looks unreliable. The test comes from how `reconstruct_table` derives columns: it clusters cell *left edges*, exact on a clean grid but prone to splitting one real column into several when entries are not left-aligned, leaving a wide mostly-empty table (the "spurious empty columns" a scanned invoice produced). So `geometric_table_is_reliable` trusts a grid only when it is dense (>=60% of cells carry text) and has no column that a single row uses; everything else, including degenerate shapes, goes to the model. Deliberately one-sided: it only ever skips work on grids that are plainly well-formed. This lives in the browser path alone — the native pipeline runs TableFormer on every region as before, so its conformance output is untouched. The predicate and the reconstruction are shared code (`assemble` is now public under `ocr-prep`) and covered by a unit test over both shapes: a genuine 4-column grid is trusted, the split-column invoice shape is not. **GitHub Pages.** A workflow builds the wasm and publishes `www/` on every push to master, so the demo is reachable from any device with no checkout; the root README and the crate README link to it. No models are published — the page resolves them at runtime (device file, same-origin ./models/, or Hugging Face), so declarative conversion and text-layer PDFs work on arrival and OCR works as soon as models are supplied. Pages cannot send COOP/COEP, but www/coi.js installs the service worker that does, so ORT still gets threads. Enabling it is one repo setting: Settings -> Pages -> Source: "GitHub Actions". Verified: docling/docling-pdf/docling-core/docling-wasm suites green (152 tests) so the shared-code change moved nothing native; pdf-text-only and no-default feature builds still compile; the page re-checked end to end in headless Chromium. Refs #157 Signed-off-by: artiz Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT --- .github/workflows/pages.yml | 99 ++++++++++++++++++++++++++++++ README.md | 7 +++ crates/docling-pdf/src/assemble.rs | 95 +++++++++++++++++++++++++++- crates/docling-pdf/src/lib.rs | 5 ++ crates/docling-wasm/README.md | 23 ++++++- crates/docling-wasm/src/scanned.rs | 28 +++++---- crates/docling-wasm/www/index.html | 2 +- 7 files changed, 244 insertions(+), 15 deletions(-) create mode 100644 .github/workflows/pages.yml diff --git a/.github/workflows/pages.yml b/.github/workflows/pages.yml new file mode 100644 index 00000000..7f4581f0 --- /dev/null +++ b/.github/workflows/pages.yml @@ -0,0 +1,99 @@ +# Publish the in-browser demo (crates/docling-wasm/www) to GitHub Pages, so +# docling.rs can be tried from any device without a checkout or a toolchain. +# +# The site is fully static: the wasm module is built here and copied in next to +# the page. No models are published — they are large, and the page resolves them +# at runtime (device file → same-origin ./models/ → Hugging Face), so the +# declarative converters and text-layer PDFs work immediately and OCR works as +# soon as the visitor supplies models. +# +# GitHub Pages cannot send COOP/COEP headers, which ONNX Runtime Web needs for +# threads; www/coi.js installs a service worker that adds them, so the deployed +# page still gets multi-threaded wasm. +# +# One-time setup in the repository: Settings → Pages → Build and deployment → +# Source: "GitHub Actions". +name: pages + +on: + push: + branches: [master] + paths: + - "crates/docling-wasm/**" + - "crates/docling/**" + - "crates/docling-core/**" + - "crates/docling-pdf/**" + - ".github/workflows/pages.yml" + # Deploy on demand too (e.g. the first run, before any matching push). + workflow_dispatch: + +permissions: + contents: read + pages: write + id-token: write + +# One deployment at a time; let a running one finish so the site is never +# left half-published. +concurrency: + group: pages + cancel-in-progress: false + +jobs: + build: + name: build the demo + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v7 + + - name: Install Rust (stable + wasm32 target) + run: | + rustup toolchain install stable --profile minimal + rustup default stable + rustup target add wasm32-unknown-unknown + + - uses: actions/cache@v4 + with: + path: | + ~/.cargo/registry/index + ~/.cargo/registry/cache + ~/.cargo/git/db + target + key: ${{ runner.os }}-cargo-v2-wasm-${{ hashFiles('**/Cargo.lock') }} + restore-keys: ${{ runner.os }}-cargo-v2-wasm- + + - name: Build docling-wasm (release) + run: cargo build -p docling-wasm --target wasm32-unknown-unknown --release --locked + + - name: Generate the JS bindings + run: | + # Pin wasm-bindgen-cli to the wasm-bindgen version in Cargo.lock: a + # mismatch produces glue the module rejects at load time. + version=$(cargo metadata --format-version 1 --locked \ + | python3 -c 'import json,sys; print(next(p["version"] for p in json.load(sys.stdin)["packages"] if p["name"]=="wasm-bindgen"))') + echo "wasm-bindgen $version" + cargo install wasm-bindgen-cli --version "$version" --locked + wasm-bindgen --target web --out-dir crates/docling-wasm/www/pkg \ + target/wasm32-unknown-unknown/release/docling_wasm.wasm + + - name: Assemble the site + run: | + mkdir -p site + cp -r crates/docling-wasm/www/. site/ + # Jekyll would swallow paths that start with an underscore. + touch site/.nojekyll + ls -la site site/pkg + + - uses: actions/upload-pages-artifact@v3 + with: + path: site + + deploy: + name: deploy + needs: build + runs-on: ubuntu-latest + environment: + name: github-pages + url: ${{ steps.deployment.outputs.page_url }} + steps: + - id: deployment + uses: actions/deploy-pages@v4 diff --git a/README.md b/README.md index 120efc42..f79ee34f 100644 --- a/README.md +++ b/README.md @@ -12,6 +12,13 @@ pipeline is supported, validated byte-for-byte against live docling. See [`docs/MIGRATION.md`](./docs/MIGRATION.md) for the full architecture, the Python → Rust mapping, and per-format conformance. +**▶ [Try it in your browser](https://docling-project.github.io/docling.rs/)** — +the whole converter compiled to wasm: drop a DOCX, PDF, XLSX, EPUB … and get +Markdown, docling JSON or DocLang XML back. Nothing is uploaded; the page runs +entirely on your device, phone included. Scanned pages can be OCR'd there too +(layout + PP-OCR + TableFormer via ONNX Runtime Web) once you point it at the +models. See [`crates/docling-wasm`](./crates/docling-wasm/README.md). + Developed with **Claude Code** and _[TENET](https://github.com/artiz/tenet/tree/master)_ (minimalistic spec driven design framework). ## Status diff --git a/crates/docling-pdf/src/assemble.rs b/crates/docling-pdf/src/assemble.rs index 8d39ce2c..14d18495 100644 --- a/crates/docling-pdf/src/assemble.rs +++ b/crates/docling-pdf/src/assemble.rs @@ -895,7 +895,7 @@ fn code_region_text(region: &Region, cells: &[TextCell]) -> String { /// left edges), then place each cell. A model-free stand-in for TableFormer that /// recovers grid-aligned tables from the precise PDF text layer (it does not /// resolve row/column spans). -fn reconstruct_table(region: &Region, cells: &[TextCell]) -> Vec> { +pub fn reconstruct_table(region: &Region, cells: &[TextCell]) -> Vec> { let mut inside: Vec<&TextCell> = cells .iter() .filter(|c| { @@ -965,6 +965,59 @@ fn reconstruct_table(region: &Region, cells: &[TextCell]) -> Vec> { grid } +/// Does the geometric reconstruction of a table look trustworthy enough to use +/// as-is, instead of paying for TableFormer? +/// +/// [`reconstruct_table`] derives columns by clustering cell **left edges**. On a +/// clean grid that is exact, but when a column's entries are not left-aligned +/// (or the OCR boxes wobble) the clustering splits one real column into several, +/// and the result is a wide, mostly-empty grid — the "spurious empty columns" +/// failure TableFormer exists to fix. +/// +/// Two symptoms separate the two cases, and both are properties of the grid +/// alone (no model needed): +/// * **density** — a real table is mostly full; a split-up one is mostly holes; +/// * **thin columns** — a column carrying at most one entry across several rows +/// is almost always a split artefact rather than a real column. +/// +/// Deliberately conservative: it answers `true` only for grids that are plainly +/// well-formed, so the expensive path stays the default whenever there is doubt. +/// A caller that skips TableFormer on `true` trades no quality for the time. +pub fn geometric_table_is_reliable(rows: &[Vec]) -> bool { + let ncols = rows.iter().map(Vec::len).max().unwrap_or(0); + // Fewer than two columns is not a grid this heuristic can vouch for: it is + // exactly the shape a collapsed table takes, and TableFormer may recover + // real structure from it. + if rows.len() < 2 || ncols < 2 { + return false; + } + let filled = |c: &String| !c.trim().is_empty(); + let total = rows.len() * ncols; + let full = rows.iter().flatten().filter(|c| filled(c)).count(); + if (full as f32) < MIN_TABLE_FILL * total as f32 { + return false; + } + // A column used by at most one row, when there are rows enough to tell. + if rows.len() >= 3 { + for ci in 0..ncols { + let used = rows + .iter() + .filter(|r| r.get(ci).is_some_and(filled)) + .count(); + if used <= 1 { + return false; + } + } + } + true +} + +/// Share of a geometric grid's cells that must carry text for it to be trusted +/// without TableFormer. Chosen well above the density a left-edge split +/// produces (those land nearer a third) and below what a genuine table with a +/// few blank cells reaches. +const MIN_TABLE_FILL: f32 = 0.6; + /// The union bbox of the text cells assigned to a region (same >50%-overlap /// rule as [`region_text`]), or `None` when no cell lands in it. docling's /// LayoutPostprocessor shrinks a regular cluster's bbox to its cells, and the @@ -1657,6 +1710,46 @@ mod tests { use crate::pdfium_backend::{LinkAnnot, PdfPage, TextCell}; use docling_core::Node; + /// The geometric-reliability gate, on the two shapes it has to tell apart. + #[test] + fn geometric_reliability_rejects_split_column_grids() { + let g = |rows: &[&[&str]]| -> Vec> { + rows.iter() + .map(|r| r.iter().map(|c| c.to_string()).collect()) + .collect() + }; + // A genuine grid: dense, every column carrying entries. Nothing for + // TableFormer to improve, so geometry is used as-is. + assert!(super::geometric_table_is_reliable(&g(&[ + &["Datum", "Leistung", "Anzahl", "Kosten"], + &["04.07", "Internet", "1", "40.30"], + &["04.07", "Telefon", "2", "8.06"], + ]))); + // The left-edge split artefact (the shape a scanned invoice produced): + // one real label column plus values scattered across three sparse ones. + assert!(!super::geometric_table_is_reliable(&g(&[ + &["www.magenta.at/faq", "", "", ""], + &["Serviceteam", "", "", ""], + &["Telefon", "0676/2000", "", ""], + &["Kundennummer", "", "", "1.21699482"], + &["Rechnungsnummer", "", "922769430725", ""], + &["Rechnungsdatum", "", "", "04.07.2025"], + ]))); + // A column only one row ever uses is a split artefact even when the + // grid is otherwise dense. + assert!(!super::geometric_table_is_reliable(&g(&[ + &["a", "b", ""], + &["c", "d", ""], + &["e", "f", "g"], + ]))); + // Degenerate shapes are never vouched for — TableFormer may recover + // structure a collapsed reconstruction lost. + assert!(!super::geometric_table_is_reliable(&g(&[&[ + "only one column" + ]]))); + assert!(!super::geometric_table_is_reliable(&[])); + } + /// A `picture` region is cropped out of the rendered page, whatever built /// that page. The browser pipeline (#157) has no pdfium but does hand over /// the rasterized bitmap through `from_cells_with_image`, so it must get diff --git a/crates/docling-pdf/src/lib.rs b/crates/docling-pdf/src/lib.rs index ddc4397b..85b71d53 100644 --- a/crates/docling-pdf/src/lib.rs +++ b/crates/docling-pdf/src/lib.rs @@ -14,6 +14,11 @@ // build still flags genuinely dead code). #![cfg_attr(not(feature = "ml"), allow(dead_code))] +// Reading-order assembly. Public under `ocr-prep` so the browser pipeline can +// reuse the geometric table reconstruction and its reliability gate (#157). +#[cfg(feature = "ocr-prep")] +pub mod assemble; +#[cfg(not(feature = "ocr-prep"))] mod assemble; mod dp_lines; #[cfg(feature = "ml")] diff --git a/crates/docling-wasm/README.md b/crates/docling-wasm/README.md index 8d75c22c..3b107b7b 100644 --- a/crates/docling-wasm/README.md +++ b/crates/docling-wasm/README.md @@ -100,9 +100,16 @@ wasm-bindgen --target web --out-dir crates/docling-wasm/www/pkg \ ## Demo +**Live: ** — deployed from +`www/` by [`.github/workflows/pages.yml`](../../.github/workflows/pages.yml) on +every push to `master`. No models are published with it, so the declarative +converters and text-layer PDFs work straight away and OCR starts once you give +the page models (device picker or Hugging Face — see below). + [`www/index.html`](./www/index.html) is the whole thing on one page: drop a file, pick the output (Markdown / JSON / DocLang), pick how images render, and -optionally turn on OCR for scanned pages. After the `wasm-bindgen` step above: +optionally turn on OCR for scanned pages. To run it locally, after the +`wasm-bindgen` step above: ```bash python3 -m http.server -d crates/docling-wasm/www 8901 @@ -136,7 +143,19 @@ wasm output matching native — drift can only come from the runtime kernels. |---|---|---| | **1** — `ocr_image` | OCR a single scanned **image** | PP-OCRv3 recognition (~10 MB) | | **2** — `ScannedConverter` / `convert_scanned_image` | Scanned **PDF/image** → Markdown: layout + OCR + reading order, tables via geometric reconstruction | + RT-DETR layout (`layout_heron_int8.onnx`, ~68 MB) | -| **3** — `ScannedConverter.addPageTf` | Real **TableFormer** table structure instead of geometric | + `tableformer/{encoder,decoder_kv,bbox}.onnx` (~380 MB) | +| **3** — `ScannedConverter.addPageTf` | Real **TableFormer** table structure, on the tables that need it | + `tableformer/{encoder,decoder_kv,bbox}.onnx` (~380 MB) | + +Stage 3 is *selective*. TableFormer's encoder runs once per table region and +costs seconds, so each table is first reconstructed geometrically (free, from +the OCR cell positions) and the model is invoked only when that grid looks +unreliable — `docling_pdf::assemble::geometric_table_is_reliable`. The tell is +how `reconstruct_table` derives columns: it clusters cell *left edges*, which is +exact on a clean grid but splits one real column into several when entries are +not left-aligned, leaving a wide, mostly-empty table. So a grid is trusted only +when it is dense (≥60% of cells carry text) and has no column that just one row +uses; anything else goes to TableFormer. Well-formed tables therefore cost +nothing extra, and the pages that used to show spurious empty columns still get +the model. Stage 1 recognition wrapper: diff --git a/crates/docling-wasm/src/scanned.rs b/crates/docling-wasm/src/scanned.rs index 20bf0838..a6bc0dff 100644 --- a/crates/docling-wasm/src/scanned.rs +++ b/crates/docling-wasm/src/scanned.rs @@ -22,6 +22,7 @@ //! ``` //! (`www/index.html` is the complete wiring.) +use docling_pdf::assemble::{geometric_table_is_reliable, reconstruct_table}; use docling_pdf::layout::{decode_layout, layout_input, SIDE}; use docling_pdf::ocr_prep::{ batch_input, decode_row, dict_chars, normalize_polarity, prep_region_lines, prep_table_words, @@ -215,19 +216,24 @@ impl ScannedConverter { let table_rows = if let Some(tf) = tf { let mut rows = Vec::with_capacity(regions.len()); for r in ®ions { - if r.label == "table" { - rows.push( - crate::tableformer::predict_table_rows( - tf, - &img, - [r.l, r.t, r.r, r.b], - &cells, - ) - .await, - ); - } else { + if r.label != "table" { + rows.push(None); + continue; + } + // TableFormer costs seconds per region (the fp32 encoder runs + // once per table), so spend it only where it buys something: + // when the free geometric reconstruction already yields a dense, + // well-formed grid it is what TableFormer would agree with, and + // `None` tells assemble to keep it. + let geometric = reconstruct_table(r, &cells); + if geometric_table_is_reliable(&geometric) { rows.push(None); + continue; } + rows.push( + crate::tableformer::predict_table_rows(tf, &img, [r.l, r.t, r.r, r.b], &cells) + .await, + ); } rows } else { diff --git a/crates/docling-wasm/www/index.html b/crates/docling-wasm/www/index.html index bd138e20..5d5f6599 100644 --- a/crates/docling-wasm/www/index.html +++ b/crates/docling-wasm/www/index.html @@ -89,7 +89,7 @@

docling.rs in your browser

-