docling-project · artiz · Jul 3, 2026 · Jul 2, 2026 · Jul 2, 2026 · Jul 2, 2026
diff --git a/.github/workflows/publish-models.yml b/.github/workflows/publish-models.yml
@@ -5,10 +5,12 @@
 #
 # The layout model and TableFormer are PyTorch→ONNX *format conversions* of
 # docling-project's own models (Apache-2.0 / CDLA-Permissive-2.0 — see
-# MODELS_NOTICE.md); no weights are retrained or modified here. pdfium and the
-# OCR model are re-hosted third-party binaries, unmodified, from their own
-# public releases — mirrored here only so `download_dependencies.sh` needs one
-# host instead of three.
+# MODELS_NOTICE.md); no weights are retrained here. The `*_int8` assets are
+# post-training quantizations of those same exports (scripts/quantize_models.py
+# — a numeric re-encoding of the weights, no retraining; conformance-validated
+# in PDF_PERFORMANCE.md). pdfium and the OCR model are re-hosted third-party
+# binaries, unmodified, from their own public releases — mirrored here only so
+# `download_dependencies.sh` needs one host instead of three.
 #
 # Trigger: manual only (workflow_dispatch) — it installs torch and exports two
 # models, several GB and 10-20 minutes, not something to run on every push.
@@ -65,6 +67,26 @@ jobs:
         id: tf-export
         run: python scripts/export_tableformer.py models/tableformer
 
+      # --- INT8 quantization (optional — fp32 assets publish regardless) ---
+      # Conv-only static INT8 of the layout model (calibrated on the repo's PDF
+      # corpus, checked out above) + dynamic INT8 of the TableFormer decoder.
+      # Validated conformance-neutral; see PDF_PERFORMANCE.md. Two separate
+      # steps so a TableFormer export failure doesn't block the layout int8.
+      - name: Install quantization deps
+        continue-on-error: true
+        id: quant-deps
+        run: pip install onnx onnxruntime sympy pypdfium2 pillow numpy
+
+      - name: Quantize layout model (static INT8, conv-only)
+        if: steps.quant-deps.outcome == 'success'
+        continue-on-error: true
+        run: python scripts/quantize_models.py layout
+
+      - name: Quantize TableFormer decoder (dynamic INT8)
+        if: steps.quant-deps.outcome == 'success' && steps.tf-export.outcome == 'success'
+        continue-on-error: true
+        run: python scripts/quantize_models.py tableformer-decoder
+
       # --- pdfium + OCR (re-hosted third-party binaries, not exports) -----
       - name: Fetch pdfium + OCR assets
         run: |
@@ -97,8 +119,10 @@ jobs:
           stage third-party/ocr_rec.onnx ocr_rec.onnx
           stage third-party/ppocr_keys_v1.txt ppocr_keys_v1.txt
           stage models/layout_heron.onnx layout_heron.onnx
+          stage models/layout_heron_int8.onnx layout_heron_int8.onnx
           stage models/tableformer/encoder.onnx encoder.onnx
           stage models/tableformer/decoder.onnx decoder.onnx
+          stage models/tableformer/decoder_int8.onnx decoder_int8.onnx
           stage models/tableformer/bbox.onnx bbox.onnx
           echo "staged:"
           ls -la release-assets/

diff --git a/PDF_PERFORMANCE.md b/PDF_PERFORMANCE.md
@@ -0,0 +1,207 @@
+# PDF pipeline — performance review & profiling notes
+
+Post-migration review of the PDF processing path: where the time actually goes,
+what was measured, which optimizations are validated, and a ranked backlog of
+further ideas that do **not** trade away output quality.
+
+Measured on a 4-core AVX-512(+VNNI/AMX) Xeon, release build (`lto = "thin"`),
+models from `scripts/download_dependencies.sh`, `FLEISCHWOLF_TIMING=1`.
+
+## Where the time goes
+
+Per-stage wall-clock share (summed across workers):
+
+| Stage | 1913-page text-heavy PDF¹ | 16-page table-heavy paper² | scanned page³ |
+|---|---:|---:|---:|
+| `layout.predict` (RT-DETR ONNX) | **80.3%** | 55.4% | 64.9% |
+| `image.resize` (3×→2× CatmullRom) | 14.9% | 7.9% | 18.5% |
+| `tableformer` | 2.8% | 32.1% | — |
+| `pdfium.render` | 1.8% | 3.7% | 16.5% |
+| `textparse` + assembly | ~0.2% | ~0.3% | ~0.1% |
+
+¹ `tests/data/pdf/large/dotnet-csharp-language-reference.pdf` — 936 s wall, ~0.49 s/page.
+² `tests/data/pdf/sources/2203.01017v2.pdf`.
+³ `tests/data/scanned/sources/ocr_test.pdf`.
+
+Two conclusions drive everything below:
+
+1. **ONNX inference is ~85–95% of PDF conversion time.** All the Rust-side text
+   extraction, parsing, and assembly work combined is under 1%. Rust-code
+   micro-optimizations are irrelevant to PDF throughput until the models get
+   faster; model-level and preprocessing-level changes are the only levers that
+   matter.
+2. Within TableFormer, the **autoregressive decode loop** dominates
+   (`tableformer.structure` ≈ 96% of the stage; the per-table page resample
+   `tableformer.inter_area` is ~1% of a conversion).
+
+The worker-pool topology heuristic in `lib.rs` (`workers × intra ≈ cores`,
+default 2×2 on 4 cores) was re-validated: 2×2 beat both 4×1 and 1×4 on the
+16-page document (11.6 s vs 12.2 s vs 15.6 s).
+
+## Validated win: INT8 quantization (quality-checked)
+
+`scripts/quantize_models.py` produces two quantized models. Point
+`DOCLING_LAYOUT_ONNX` / `DOCLING_TABLEFORMER_DECODER` at them to opt in.
+
+**These are now the default:** when the `*_int8` files sit next to the fp32
+models at the default paths, the pipeline loads them automatically.
+`FLEISCHWOLF_FP32=1` forces full precision, and an explicit
+`DOCLING_LAYOUT_ONNX` / `DOCLING_TABLEFORMER_DECODER` always wins (the
+conformance/groundtruth scripts pin fp32 explicitly, so snapshots stay
+deterministic).
+
+### Layout: static QDQ INT8, **Conv ops only** (~2.4× faster layout)
+
+Calibrated on 42 real corpus pages preprocessed exactly like
+`layout.rs::predict`. Only the HGNetv2 backbone convolutions are quantized;
+the transformer decoder and detection-head MatMuls stay fp32.
+
+| Configuration | layout.predict (16-page doc) | end-to-end wall | model size |
+|---|---:|---:|---:|
+| fp32 baseline | 17.2 s | 16.6 s | 172 MB |
+| **INT8 conv-only** | **7.2 s (2.4×)** | 11.5 s (1.45×) | 68 MB |
+| + INT8 TableFormer decoder | — | **12.3 s → see note** | — |
+
+On text-dominated documents (layout = 80% of time) the end-to-end gain
+approaches ~1.7–2×; on table-heavy ones it is ~1.4×.
+
+Full-scale run — the 1913-page `dotnet-csharp-language-reference.pdf`,
+INT8 layout + INT8 TableFormer decoder vs fp32, same machine and binary,
+back-to-back:
+
+| | fp32 | INT8 | ratio |
+|---|---:|---:|---:|
+| wall clock | 1406 s (0.74 s/page) | **770 s (0.40 s/page)** | **1.83×** |
+| `layout.predict` (summed) | 2667 s | 1350 s | 1.98× |
+| output difference | — | 1199 of 52,615 Markdown lines (2.3%) | |
+
+The 2.3% of differing lines are the same near-threshold classification flips
+seen on the corpus (where groundtruth conformance measured *equal or slightly
+better* under INT8 — 812 vs 833 summed diff-lines), not a systematic
+degradation. With layout halved, `image.resize` becomes the next stage
+(24.8% of the INT8 run), which is why backlog item 4 matters more after
+quantization.
+
+**Quality gate.** Markdown diffed across the full PDF+scanned corpus (23 files):
+
+- Conv-only INT8: 12/23 byte-identical to fp32; remaining diffs are small
+  region-classification flips. Against the committed groundtruth the summed
+  diff-line distance is **812 (INT8) vs 833 (fp32)** — i.e. conformance-neutral
+  (INT8 is marginally better on 3 fixtures, marginally worse on 2).
+- Full INT8 (convs + MatMuls) was **rejected**: 3/23 exact, with clear quality
+  loss (section headers demoted to plain text, page-footer text leaking into
+  the output) — the RT-DETR head's class scores sit near the 0.3 threshold and
+  cannot tolerate activation quantization.
+- Dynamic (weights-only) INT8 of the whole layout model was also rejected: it
+  is *slower* than fp32 (3.2 s vs 2.1 s per page-with-table) because inserted
+  per-activation quantize ops outweigh the MatMul savings while the conv
+  backbone stays fp32.
+
+### TableFormer decoder: dynamic INT8 (~10% faster tables, byte-identical)
+
+The autoregressive tag decoder is MatMul-only; weights-only dynamic INT8
+produced **byte-identical corpus output** and ~10% faster table decode
+(784 → 695 ms/table), 78 → 50 MB. Small but free.
+
+The decoder speed is *not* weight-bound — it is per-step overhead (see backlog
+item 2), which is why quantization helps so little there.
+
+## Ranked backlog of further ideas
+
+Ordered by expected impact ÷ risk. Items 1–3 attack the 85–95%.
+
+1. ~~**Ship/document the INT8 layout model as the default CPU
+   configuration**~~ **Done on this branch:** the pipeline prefers the int8
+   models when present (`FLEISCHWOLF_FP32=1` opts out),
+   `download_dependencies.sh` fetches them by default, and
+   `publish-models.yml` builds them. Biggest single validated win: ~1.4–2×
+   end-to-end.
+2. **TableFormer decode-loop overhead** (~800 ms/table, ~60–500 steps):
+   - ~~`decode_step` copies the whole KV cache out (`ocache.to_vec()`) and back
+     in every step — O(steps²·6·512) float traffic.~~ **Done on this branch:**
+     the cache and the encoder's cross-K/V + `enc_out` stay owned `ort` values
+     fed straight back into the next run (~9% faster structure decode,
+     byte-identical output).
+   - The exported graph still re-embeds the **full tag sequence** every step
+     (`tags` grows each iteration) even though attention is KV-cached. Re-export
+     the decoder to take only the last tag (the cache carries the history) —
+     this is the remaining per-step cost; see `scripts/export_tableformer.py`.
+3. **Layout batching for the parallel path**: the pool currently runs batch-1
+   inference per page. RT-DETR's 640×640 input is fixed-shape, so pages can be
+   batched (e.g. batch-4) per worker with one session — better core utilization
+   and less framework overhead on wide machines. Output is per-image, so
+   quality is unaffected. (Needs a re-export with a dynamic batch dim.)
+4. **Render → resize pipeline copies** (`pdfium_backend.rs:264-272`, ~15% on
+   text-heavy docs): pdfium's BGRA bitmap goes through `as_image()` (copy +
+   swizzle) then `.into_rgb8()` (second copy) before the 3×→2× CatmullRom
+   downsample (third buffer). A single BGRA→RGB pass into the resize source
+   removes one full-page copy + traversal per page. Keep the 1.5× supersample +
+   CatmullRom itself — it is deliberate PIL-BICUBIC parity for model input.
+   Also: when `layout.predict` is the only image consumer at 640×640, the
+   2×-page intermediate is only needed for TableFormer crops and OCR — a page
+   with no table regions and a text layer never needs it; rendering could be
+   deferred/skipped per page in a `no_ocr`-like fast path decided *after*
+   layout runs on a cheaper raster.
+5. **textparse font caching** (marginal for PDFs — textparse is ≤1% — but
+   real for `no_ocr` mode where it becomes the bottleneck):
+   - fonts are fully re-parsed (ToUnicode CMap decompression + tokenization,
+     Type1 program scan, width maps) for **every page** and every Form-XObject
+     invocation (`textparse.rs:794`); cache parsed `Font`s per document keyed
+     by the font dict's `ObjectId`, and cache decoded Form XObject content.
+   - ~~`line_cells` + `word_cells` run the identical build+contract twice per
+     page; one pass can emit both.~~ **Done on this branch**
+     (`dp_lines::line_and_word_cells`): ~1.25× faster `--no-ocr` conversion,
+     identical output.
+   - `decode_code`/`decompose_ligatures` allocate a `String` per glyph
+     (`textparse.rs:94-145`); decompose once at font-parse time and return
+     borrowed `&str`.
+   - RTL merge is O(n²) (string prepend + `Vec::remove(0)`,
+     `dp_lines.rs:87-155`); accumulate reversed and flip once per line.
+6. **OCR line batching** (`ocr.rs::recognize`): lines are recognized one at a
+   time on one thread (deliberately, for CTC determinism). Batching same-width
+   buckets keeps determinism per line and would speed scanned documents
+   several-fold; alternatively run multiple single-thread recognitions across
+   the existing worker pool.
+7. **ort session options**: checked — ONNX Runtime's C-API default is already
+   `ORT_ENABLE_ALL`, so an explicit optimization level gains nothing.
+   `with_optimized_model_path` (caching the optimized graph on disk) could
+   still shave per-worker model-load latency; only worth it if pool spin-up
+   shows up in a real deployment.
+
+## Correctness notes found during review (quality, not speed)
+
+- `textparse.rs` `"` operator: the `aw ac string "` form must set word/char
+  spacing (`tw`/`tc`) from its first two operands before showing the string;
+  they are currently ignored (`Tj | ' | "` share one arm), so documents using
+  `"` get wrong inter-word advances. **Fixed in this branch.**
+- `textparse.rs::page_size` ignores a non-zero MediaBox origin; a page with
+  e.g. `[9 9 621 801]` offsets all parser cells relative to pdfium's raster.
+  Rare, but cheap to guard: subtract the box origin when emitting glyph boxes.
+- OCR recognition ran un-instrumented; `ocr.page` is now a timed stage (this
+  branch), so scanned-corpus profiles attribute it correctly.
+
+## Reproducing
+
+```bash
+scripts/download_dependencies.sh
+cargo build --release
+
+# stage timing
+FLEISCHWOLF_TIMING=1 ./target/release/fleischwolf input.pdf > /dev/null
+
+# build the int8 models (used automatically once present)
+uv venv .venv-quant && uv pip install --python .venv-quant/bin/python \
+    onnx onnxruntime sympy pypdfium2 pillow numpy
+.venv-quant/bin/python scripts/quantize_models.py
+
+# force full precision for a run
+FLEISCHWOLF_FP32=1 ./target/release/fleischwolf input.pdf > /dev/null
+```
+
+Integration points: `scripts/download_dependencies.sh` fetches the
+pre-quantized assets by default (`--no-int8` skips; published by
+`.github/workflows/publish-models.yml`, which quantizes after export);
+`scripts/pdf_setup.sh` quantizes locally unless `FLEISCHWOLF_FP32=1`;
+`scripts/performance.sh` benchmarks whatever the pipeline default resolves to
+(int8 when present, `FLEISCHWOLF_FP32=1` for fp32); `examples/Dockerfile`
+bakes both precisions and defaults to int8 (`--build-arg INT8=0` for fp32).
diff --git a/README.md b/README.md
@@ -296,6 +296,7 @@ curl -fsSL https://raw.githubusercontent.com/artiz/fleischwolf/master/scripts/do
 | PP-OCRv3 rec + dictionary | `models/ocr_rec.onnx`, `models/ppocr_keys_v1.txt` |
 | TableFormer (optional) | `models/tableformer/{encoder,decoder,bbox}.onnx` (+ `.data` sidecars where the export needs them) |
 | Whisper tiny (audio/ASR; skip with `--no-asr`) | `models/asr/{encoder_model,decoder_model}.onnx`, `models/asr/vocab.json` (+ `added_tokens.json` for language selection) |
+| INT8 CPU models (optional; fetch with `--int8`) | `models/layout_heron_int8.onnx`, `models/tableformer/decoder_int8.onnx` |
 
 Idempotent — safe to re-run; it skips files already on disk. Pass `--force` to
 re-fetch everything, or set `$FLEISCHWOLF_MODELS_URL` to fetch from a
@@ -305,6 +306,31 @@ come from Hugging Face (`$FLEISCHWOLF_ASR_MODELS_URL` overrides, or point
 only for now — other platforms, or building the models from source, need
 [`scripts/pdf_setup.sh`](#testing) instead.
 
+### INT8 models (faster PDF conversion on CPU — the default)
+
+The `*_int8` assets are post-training quantizations of the same models:
+Conv-only static INT8 of the layout detector (calibrated on this repo's PDF
+corpus) and dynamic INT8 of the TableFormer decoder. On CPUs with AVX-512
+VNNI they make layout inference — the dominant PDF cost — **~2.4× faster**
+(~1.4–1.8× end-to-end) at conformance validated as unchanged against the
+corpus groundtruth; the TableFormer output is byte-identical. See
+[`PDF_PERFORMANCE.md`](./PDF_PERFORMANCE.md) for the measurements.
+
+**The pipeline uses them automatically** whenever they sit next to the fp32
+files at the default paths (`download_dependencies.sh` fetches them by
+default; `--no-int8` skips, or build them with `python
+scripts/quantize_models.py`). To force full precision:
+
+```bash
+FLEISCHWOLF_FP32=1 fleischwolf input.pdf          # keep the int8 files, use fp32
+# or pin a model explicitly — an explicit path always wins:
+export DOCLING_LAYOUT_ONNX=$PWD/models/layout_heron.onnx
+export DOCLING_TABLEFORMER_DECODER=$PWD/models/tableformer/decoder.onnx
+```
+
+(The [example Dockerfile](./examples/Dockerfile) bakes both precisions and
+defaults to INT8; build with `--build-arg INT8=0` for pure fp32.)
+
 Then either:
 
 ```bash
@@ -446,14 +472,17 @@ The image converts PDFs/images fully offline; the model export (torch +
 
 ## Performance
 
-`scripts/performance.sh` runs the **largest fixture of each supported type** through
-both engines (published Python `docling` vs the Rust release binary) and reports
-peak RSS, CPU utilization, and conversion time. Ratios below are docling ÷
-fleischwolf — bigger means Rust wins by more.
+`scripts/performance.sh` runs a representative fixture of each supported type
+through both engines (published Python `docling` vs the Rust release binary) and
+reports peak RSS, CPU utilization, and conversion time. Ratios below are
+docling ÷ fleischwolf — bigger means Rust wins by more. The PDF row is the
+**fp32** stack; the optional [INT8 models](#int8-models-faster-pdf-conversion-on-cpu)
+roughly double layout-inference speed on top of it (measured 1.83× end-to-end
+on a 1913-page document — see [`PDF_PERFORMANCE.md`](./PDF_PERFORMANCE.md)).
 
 | File | Size | Peak-memory ratio | CPU ratio | Warm-conversion speedup |
 |---|---:|---:|---:|---:|
-| `2203.01017v2.pdf` (PDF, 47 pp) | 6.9 MB | **2.2× less** | 1.3× | 1.2× |
+| `picture_classification.pdf` (PDF) | 208 KB | **2.3× less** | 1.0× | 2.3× |
 | `docx_rich_tables_01.docx` (DOCX) | 3.1 MB | **41× less** | 2.7× | 21× |
 | `wiki_duck.html` (HTML) | 240 KB | **57× less** | 3.2× | 46× |
 | `elife-56337.nxml` (JATS XML) | 180 KB | **61× less** | 2.9× | 10× |
@@ -465,10 +494,11 @@ fleischwolf — bigger means Rust wins by more.
 - **Peak memory** is where Rust wins decisively: a declarative conversion holds a
   few MB versus docling's ~750 MB (it imports torch even for non-ML formats). The
   PDF runs the full ML pipeline in both engines (torch vs ONNX), so the gap there
-  is 2.2× rather than 50×+, but Rust still peaks at 1.4 GB vs docling's 3.1 GB.
+  is 2.3× rather than 50×+, but Rust still peaks at 0.77 GB vs docling's 1.75 GB —
+  and the PDF converts **4.1× faster end-to-end** (docling re-pays its torch
+  import + model load on every invocation).
 - **CPU**: docling spreads across 2.7–3.2 cores for declarative work that Rust does
-  on a single core (~100%); on the PDF both go multi-core (Rust 525% vs docling
-  674%).
+  on a single core (~100%); on the PDF both go multi-core (~330% each here).
 - **Warm-conversion speedup** isolates the parse/convert work — it times docling
   *in-process* (excluding its ~3 s interpreter + import startup) against the Rust
   whole-process figure. Rust wins on substantial inputs (HTML 46×, DOCX 21×); the