paddleocr-vl-rs

A Python-free Rust pipeline that turns a PDF or image into structured markdown using PaddleOCR-VL.

Layout detection (PP-DocLayoutV3 via ONNX) finds the regions of a page; each region is cropped and recognized by the PaddleOCR-VL vision-language model (running on mistral.rs); the results are reassembled in reading order into markdown, with tables rendered from the model's OTSL output. No Python or Paddle runtime at inference time.

Architecture

PDF/image
   |  render pages (pdftoppm)
   v
PP-DocLayoutV3 (ONNX Runtime)         <- this repo: src/lib.rs
   |  Vec<Region> in reading order (25 layout classes)
   v
region crops + manifest.json          <- this repo: src/main.rs, src/assemble.rs
   |  one crop PNG + resolved task prompt per region
   v
PaddleOCR-VL VLM (via mistral.rs)     <- examples/recognize.rs (links mistral.rs)
   |  SigLIP/NaViT vision encoder -> Adaptive-MLP connector -> ERNIE-4.5-0.3B LM
   |  results.json = [{read_order, class, text}]
   v
reading-order assembly -> markdown    <- this repo: src/assemble.rs (incl. OTSL -> table)

The layout + assembly half (this crate) builds standalone with no GPU or engine dependency. The recognition VLM is a separate mistral.rs build; the two stages talk only through manifest.json and results.json.

results.json is the model's raw output, not the finished text. The repetition guard (below) runs in assemble::read_results, on ingest — so the contract file keeps a faithful record of what the VLM actually emitted, which is what makes a degenerate region diagnosable at all. The cost is that a consumer who reads results.json directly bypasses the guard and can get a region that loops to the token cap. Take the text through paddleocr-layout assemble (or call assemble::read_results); if you must parse results.json yourself, apply assemble::truncate_repetitive_content + assemble::truncate_repeating_lines to each block, with min_count 5000 for table and 50 for everything else — that is exactly what read_results does.

Upstream contributions

The recognition model and a general engine fix were contributed upstream to mistral.rs; this repo is the document pipeline built on top of them.

• mistral.rs #2320 -- feat(models): Support PaddleOCR-VL. The recognition VLM itself (SigLIP/NaViT vision encoder -> mlp_AR connector -> ERNIE-4.5-0.3B), loaded via --arch paddleocr_vl. Closes #2128.
• mistral.rs #2319 -- fix(llg): honor tokenizer special flag in toktrie detok. PaddleOCR-VL emits tables as OTSL tokens <fcel> / <nl> (flagged special=false); the completion detok was dropping them and collapsing tables to run-on text. A general fix, not model-specific.
• llguidance #361 -- the root cause: a <...>-name heuristic in toktrie_hf_tokenizers overrides an explicit special=false.

Status / correctness

OmniDocBench v1.5, full run — the primary result. The same weights on the benchmark's 1355 v1.5 pages, scored by the official scorer, across three stacks: the published PaddleOCR-VL-1.5 figures, the same model run through llama.cpp (an independent C++ implementation, used as a cross-stack control), and this Rust port.

metric	published PaddleOCR-VL-1.5	llama.cpp	this port
text Edit ↓	0.035	0.0309	0.0322
reading-order Edit ↓	0.042	0.0413	0.0414
table TEDS ↑	92.76	92.52	92.82
formula CDM ↑	94.21	92.20	93.25
Overall ↑	94.50	93.87	94.28

Accuracy is preserved against the reference on text, reading order and tables, and formula CDM is now within 0.96 of published (Overall −0.22). Getting there cost two fixes that were found by running the port's own output through its own guards, both of them parity gaps against upstream, not model behaviour: a repetition truncator (upstream's truncate_repetitive_content, which this port lacked) and crop_margin on formula crops (which upstream applies and this port did not).

The formula gap was ours far more than it was the model's, and this README used to say the opposite. The earlier reading — "−2.44 CDM, and it is the model's CJK difficulty, not a defect in this port, because llama.cpp reproduces it" — was wrong, and wrong for a structural reason worth stating: llama.cpp re-recognizes this pipeline's crop PNGs, so it is an independent control on the decode path and a common-mode one on the crop path. It could never have detected a missing crop step. crop_margin recovered 1.48 of the 2.44 (91.77 → 93.25), and it helped CJK formulas more than English ones (+2.46 vs +1.11), which is precisely the evidence the old argument claimed to rule out. Full A/B, the CDM environment trap, and what remains: docs/BENCHMARKS.md.

Every cell is the page reduction of the subset: v1.5 slice, from the official scorer. Overall is the benchmark's own ((1 − text_Edit) × 100 + table_TEDS + formula_CDM) / 3, applied identically to both measured columns. The re-scored baseline reproduces the previously published figures of this table to 4 dp (CDM 0.9177, english 0.9349, simplified_chinese 0.8730), which is what makes the deltas trustworthy rather than a scoring-setup artifact.

These headline numbers were assembled by splicing per-fix A/B deltas onto one scored run — the repetition guard re-assembled in place, the crop_margin formula crops re-recognized and spliced back. A single clean run of all four fixes from the start (full1651_allfix, config bench/omnidocbench/data/subsets/full1651_allfix.end2end.yaml) reproduces them end-to-end: on subset: v1.5, text-edit 0.0322 (identical to the splice), table TEDS 92.99, reading-order 0.0410, formula-edit 0.1692 — table and reading order land marginally better clean than spliced, none worse. (CDM is not in that run; it needs the separate CDM environment, so the 93.25 above stands from the crop_margin CDM A/B.) Full detail: docs/BENCHMARKS.md.

The shipped OmniDocBench.json is a 1651-page superset (it bundles 296 adversarial *_hard pages that are not on the leaderboard); scoring the superset instead gives the pessimistic text 0.0368 / TEDS 90.36 / RO 0.0434. Both columns, the evidence for the 1355-page reading, and the caveat that it is an inference (the leaderboard does not publish its page list) are in docs/BENCHMARKS.md.

• Token-for-token greedy parity vs the transformers-5.13 reference across a 9-item corpus (plain text, tables, formulas, spotting, seal, chart, CJK, low-quality scan, 2-column), on both CPU-f32 and GPU-bf16. 9/9 match golden token ids.
• Layout stage: Rust preprocess+run+decode matches an onnxruntime reference within ~0.05 px on the sample page (resampler drift only). See tests/parity_layout.rs.
• Load-once recognition is gated on byte-identical output vs the per-page path (24 pages / 189 crops covering 22 of the 25 layout classes, 24/24 identical) — a speed mode that changes a token is a bug.
• Degenerate regions (the model loops and never emits EOS) are truncated on ingest by assemble::truncate_repetitive_content, a port of upstream's own truncate_repetitive_content with its per-class floors. PaddleOCR-VL decodes greedily with no repetition penalty — upstream's predictor ignores the parameter outright — so this string guard, not the sampler, is where the original stack handles it too. Measured, not assumed: re-assembling and re-scoring the same 1649-page results.json with and without it moves every v1.5 metric the right way — text Edit 0.0327→0.0323, table TEDS 92.75→92.82, table Edit 0.0568→0.0556, formula Edit 0.1833→0.1817, reading-order 0.0415→0.0414. It alters 204 of 78,710 recognized blocks, and every one of them was degenerate. See docs/BENCHMARKS.md.
• Formula crops are trimmed to their ink before recognition (assemble::crop_margin), which is what upstream does for formula blocks and only formula blocks. Worth −6.6% formula edit on v1.5 (0.1817 → 0.1697) — and −5.6% on llama.cpp over the same crops, which is the cross-stack signature of a crop-path fix rather than a port bug.

4-bit ISQ quantization is near-lossless

mistral.rs can quantize the weights at load (ISQ). Scored on the 150-page stratified subset with the official scorer, bf16 vs ISQ-Q4K, both arms carrying the OTSL detok fix so tables render (subset: v1.5, page_avg):

metric	bf16	ISQ-Q4K	Δ
text Edit ↓	0.0321	0.0302	−0.0019
reading-order Edit ↓	0.0604	0.0604	0.0000
formula Edit ↓ (proxy, not CDM)	0.1796	0.1676	−0.0120
table TEDS ↑	0.9397	0.9306	−0.0091
table TEDS-S ↑	0.9754	0.9720	−0.0034

Text, reading order and formula move toward better or flat under 4-bit — the signature of run-to-run noise, not a real gain, since 4-bit rounding cannot out-read bf16. The quantization is nonetheless doing real work: 116 of 150 predictions differ from the bf16 arm, so Q4K changes the exact tokens on 77% of pages while leaving the aggregates intact. The one directionally-consistent cost is table structure (TEDS −0.0091 and table Edit +0.0080, both worse) — OTSL is a structured token sequence with span markers, more sensitive to weight rounding than free text.

A raw ISQ run's collapsed tables (TEDS 0.0) are not the quantization: the bf16 control run without the detok fix collapses identically, so that is mistral.rs #2319 missing, not 4-bit. Caveat: recognition for these arms was an out-of-tree run — the committed examples/recognize.rs carries no ISQ flag — so unlike the rest of this section it is a recorded measurement, not reproducible from this tree. The scoring is: see docs/BENCHMARKS.md.

Quick start

Prerequisites (the pipeline itself is Python-free at inference; you still need the artifacts):

1. ONNX Runtime shared library. Any recent 1.2x build works. Point the loader at it: export ORT_DYLIB_PATH=/path/to/libonnxruntime.so (e.g. from a pip install onnxruntime).
2. PP-DocLayoutV3 ONNX graph (the layout model). Export its path: export PADDLEOCR_LAYOUT_MODEL=/path/to/PP-DocLayoutV3.onnx.
3. PaddleOCR-VL-1.5 or -1.6 checkpoint (the recognition weights) from Hugging Face. Either works unchanged — see Checkpoint versions.
4. poppler-utils for pdftoppm if you start from PDFs.

Build the layout binary (standalone, no GPU/engine deps):

cargo build --release          # produces target/release/paddleocr-layout

Run the layout stage on one page (writes crops + manifest.json):

./target/release/paddleocr-layout page.png out/

Build the recognition step against mistral.rs. The recognition stage needs both upstream PRs: #2320 (the PaddleOCR-VL model) and #2319 (the OTSL detok fix — without it the table tokens <fcel>/<nl> are dropped and tables collapse to run-on text). Both are still open upstream, so until they land, build from the branch that carries the two of them on top of the upstream base:

git clone https://github.com/subin9/mistral.rs.git && cd mistral.rs
git checkout paddleocr-vl-pipeline    # = upstream + #2320 (model) + #2319 (detok fix)

The two patches touch disjoint files (the model adds a new arch; the detok fix is one file in mistralrs-core), so they compose cleanly — that branch is just the two applied in order, with nothing else added. Then drop examples/recognize.rs from this repo into a small binary crate that depends on mistralrs (or into mistralrs/examples/) — it is what provides the PADDLEOCR_VL_GPU toggle and the load-once --list mode used below:

# CPU/f32 (deterministic parity path):
PADDLEOCR_VL_WEIGHTS=/path/to/PaddleOCR-VL-1.5 recognize out/
# GPU/bf16 (needs a --features cuda,flash-attn mistral.rs build):
PADDLEOCR_VL_GPU=1 PADDLEOCR_VL_WEIGHTS=/path/to/PaddleOCR-VL-1.5 recognize out/
# Many pages: load the ~1.9GB checkpoint once, not once per page.
PADDLEOCR_VL_GPU=1 PADDLEOCR_VL_WEIGHTS=... recognize --list pages.txt   # one page dir per line

Checkpoint versions: 1.5 and 1.6 are interchangeable

PaddleOCR-VL-1.6 is a weights-only release and runs on this port with no code change and no rebuild — point PADDLEOCR_VL_WEIGHTS at a 1.6 checkout and it loads. Verified against the published 1.5, rather than assumed from the version number:

artifact	1.5 vs 1.6
config.json	byte-identical
tokenizer.json / tokenizer.model	byte-identical (sha256)
modeling_/configuration_/processing_/image_processing_*.py	byte-identical
preprocessor_config / generation_config / added_tokens / chat_template	byte-identical
model.safetensors tensor signature	620 tensors, identical names, shapes and dtypes
model.safetensors content	differs (this is the release)
inference.yml	one string: model_name: …-1.5-0.9B → …-1.6-0.9B

The tensor signature was read out of the safetensors header (an HTTP range request for the header bytes, not a 1.9 GB download), and the downloaded checkpoint was confirmed by content hash (85a479d5…), not by filename — the same rule this repo applies to the GGUF.

Smoke-tested end to end: the existing binary, unmodified, loaded 1.6 and recognized a 6-region page (CPU-f32, greedy). Against 1.5 on the identical crops, 4 of 6 regions are token-identical; one differs by a dash glyph, and one is a image-class crop where 1.5 runs away into a ☐ ☑ loop (the case the repetition guard exists for) while 1.6 terminates cleanly. That is a smoke test, not an accuracy claim — n=1 page, and image is a visual-only class assembly drops before scoring. No OmniDocBench run has been scored on 1.6, so every benchmark number in this repo remains a 1.5 number.

Reassemble the reading-order markdown:

./target/release/paddleocr-layout assemble out/results.json

Or run the whole PDF -> markdown flow end to end:

examples/pdf_to_markdown.sh input.pdf out/     # see the script header for the env vars it needs

Performance

Full cross-stack numbers (fair baseline, both axes, kernel-optimization work, batching, honest residual) are in docs/BENCHMARKS.md. Honest summary:

• The transformers reference is a correctness floor here, not a competitor; the port is token-faithful to it.
• GPU-bf16: ~parity end-to-end on short OCR output (a ~1.7x prefill loss offset by a ~1.4x decode win), VRAM neutral. CPU-f32: slower end-to-end on short output (prefill-bound), leaner on memory.
• Fused vision + LM attention (Sdpa/flash) closes most of the GPU prefill gap; the residual is candle's dense vision GEMM/MLP vs torch's oneDNN/cuBLAS -- a candle-maturity ceiling, not claimed closed.
• llama.cpp is faster per page, and we report that plainly: 2.7x. Same box, same 118-page sample, same crops, bf16 on both sides, layout cost charged to both. Median page 8.42s (this port, load-once) vs 3.1s (llama.cpp + layout). Deleting the per-page checkpoint reload took us 10.0s -> 8.42s and 3.2x -> 2.7x; the rest is 0.50s/crop vs 0.12s/crop of recognition. The candle vision-GEMM ceiling above is the best-supported explanation for that per-crop gap — inferred from where the time goes, not measured at the kernel level; the micro-benchmark that would settle it is specified in docs/FUTURE_WORK.md. Magnitude is workload-specific (compute-bound vision prefill, short OCR outputs) — the port's edge is a Python-free single binary that reproduces the model's accuracy, not throughput.
• Region batching buys nothing here (vision runs per-image regardless); recommended batch size is 1. Leakage-free, but no throughput win. Details and data in the benchmarks doc.

An earlier "1.44x / 1.88x faster" claim was withdrawn -- it measured an unfair, uncached baseline. An earlier 3.2x-vs-llama.cpp figure was superseded by 2.7x after the reload was deleted, and a 17s/page Rust median was retracted as a thrashing-box artifact.

Roadmap

See docs/FUTURE_WORK.md: the formula CDM gap (the one metric off parity), root-causing the runaway generation, the candle vision-GEMM lever (and a micro-benchmark to size it), assembler class-mapping expansion, and cu_seqlens packed-vision batching.

License and attribution

Apache-2.0 (see LICENSE).

• PaddleOCR-VL and PP-DocLayoutV3 are Apache-2.0 (PaddlePaddle). This pipeline follows their model architecture and preprocessing recipes.
• mistral.rs is MIT (Eric Buehler). The recognition stage runs on it.