feat(pdf): use the INT8 models by default, FLEISCHWOLF_FP32 opts out

When layout_heron_int8.onnx / tableformer/decoder_int8.onnx sit next to
the fp32 models at the default paths, the pipeline now loads them
automatically — the quantized stack is conformance-validated (see
PDF_PERFORMANCE.md) and ~2x faster on layout, so it is the right
default for CPU deployments. Resolution order per model: explicit
DOCLING_* env override > int8 default file (unless FLEISCHWOLF_FP32 is
set) > fp32 default file.

- download_dependencies.sh fetches the int8 assets by default
  (--no-int8 skips; --int8 kept as a no-op for compatibility).
- pdf_setup.sh quantizes locally unless FLEISCHWOLF_FP32=1.
- performance.sh mirrors the pipeline default (FLEISCHWOLF_FP32=1
  benchmarks fp32); the short-lived FLEISCHWOLF_INT8 toggle is gone.
- pdf_conformance.sh / pdf_groundtruth.sh already pin explicit fp32
  paths, so snapshot baselines stay byte-deterministic.

Verified: default resolves to int8 when present, FLEISCHWOLF_FP32=1
and explicit env paths both produce the fp32 reference output, and the
fp32 fallback engages when the int8 files are absent.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017fKFbatf55BK5j8vr3dZjj

Author: claude

PDF_PERFORMANCE.md

@@ -43,6 +43,13 @@ default 2×2 on 4 cores) was re-validated: 2×2 beat both 4×1 and 1×4 on the
 `scripts/quantize_models.py` produces two quantized models. Point
 `DOCLING_LAYOUT_ONNX` / `DOCLING_TABLEFORMER_DECODER` at them to opt in.
 
+**These are now the default:** when the `*_int8` files sit next to the fp32
+models at the default paths, the pipeline loads them automatically.
+`FLEISCHWOLF_FP32=1` forces full precision, and an explicit
+`DOCLING_LAYOUT_ONNX` / `DOCLING_TABLEFORMER_DECODER` always wins (the
+conformance/groundtruth scripts pin fp32 explicitly, so snapshots stay
+deterministic).
+
 ### Layout: static QDQ INT8, **Conv ops only** (~2.4× faster layout)
 
 Calibrated on 42 real corpus pages preprocessed exactly like
@@ -103,10 +110,12 @@ item 2), which is why quantization helps so little there.
 
 Ordered by expected impact ÷ risk. Items 1–3 attack the 85–95%.
 
-1. **Ship/document the INT8 layout model as the default CPU configuration**
-   (done here as an opt-in; consider making `download_dependencies.sh` fetch a
-   pre-quantized `layout_heron_int8.onnx` so users don't need the Python
-   tooling). Biggest single validated win: ~1.4–2× end-to-end.
+1. ~~**Ship/document the INT8 layout model as the default CPU
+   configuration**~~ **Done on this branch:** the pipeline prefers the int8
+   models when present (`FLEISCHWOLF_FP32=1` opts out),
+   `download_dependencies.sh` fetches them by default, and
+   `publish-models.yml` builds them. Biggest single validated win: ~1.4–2×
+   end-to-end.
 2. **TableFormer decode-loop overhead** (~800 ms/table, ~60–500 steps):
    - ~~`decode_step` copies the whole KV cache out (`ocache.to_vec()`) and back
      in every step — O(steps²·6·512) float traffic.~~ **Done on this branch:**
@@ -180,18 +189,19 @@ cargo build --release
 # stage timing
 FLEISCHWOLF_TIMING=1 ./target/release/fleischwolf input.pdf > /dev/null
 
-# quantize + opt in
+# build the int8 models (used automatically once present)
 uv venv .venv-quant && uv pip install --python .venv-quant/bin/python \
     onnx onnxruntime sympy pypdfium2 pillow numpy
 .venv-quant/bin/python scripts/quantize_models.py
-export DOCLING_LAYOUT_ONNX=$PWD/models/layout_heron_int8.onnx
-export DOCLING_TABLEFORMER_DECODER=$PWD/models/tableformer/decoder_int8.onnx
+
+# force full precision for a run
+FLEISCHWOLF_FP32=1 ./target/release/fleischwolf input.pdf > /dev/null
 ```
 
-Integration points: `scripts/download_dependencies.sh --int8` fetches
-pre-quantized assets from the models release (published by
+Integration points: `scripts/download_dependencies.sh` fetches the
+pre-quantized assets by default (`--no-int8` skips; published by
 `.github/workflows/publish-models.yml`, which quantizes after export);
-`scripts/pdf_setup.sh` quantizes locally with `FLEISCHWOLF_INT8=1`;
-`scripts/performance.sh` benchmarks the int8 stack with `FLEISCHWOLF_INT8=1`;
-`examples/Dockerfile` bakes both precisions and defaults to int8
-(`--build-arg INT8=0` for fp32).
+`scripts/pdf_setup.sh` quantizes locally unless `FLEISCHWOLF_FP32=1`;
+`scripts/performance.sh` benchmarks whatever the pipeline default resolves to
+(int8 when present, `FLEISCHWOLF_FP32=1` for fp32); `examples/Dockerfile`
+bakes both precisions and defaults to int8 (`--build-arg INT8=0` for fp32).

README.md

@@ -290,25 +290,30 @@ come from Hugging Face (`$FLEISCHWOLF_ASR_MODELS_URL` overrides, or point
 only for now — other platforms, or building the models from source, need
 [`scripts/pdf_setup.sh`](#testing) instead.
 
-### INT8 models (faster PDF conversion on CPU)
+### INT8 models (faster PDF conversion on CPU — the default)
 
-The optional `*_int8` assets are post-training quantizations of the same
-models: Conv-only static INT8 of the layout detector (calibrated on this
-repo's PDF corpus) and dynamic INT8 of the TableFormer decoder. On CPUs with
-AVX-512 VNNI they make layout inference — the dominant PDF cost — **~2.4×
-faster** (~1.4–1.7× end-to-end) at conformance validated as unchanged against
-the corpus groundtruth; the TableFormer output is byte-identical. See
-[`PDF_PERFORMANCE.md`](./PDF_PERFORMANCE.md) for the measurements. They are
-opt-in — point the pipeline at them explicitly:
+The `*_int8` assets are post-training quantizations of the same models:
+Conv-only static INT8 of the layout detector (calibrated on this repo's PDF
+corpus) and dynamic INT8 of the TableFormer decoder. On CPUs with AVX-512
+VNNI they make layout inference — the dominant PDF cost — **~2.4× faster**
+(~1.4–1.8× end-to-end) at conformance validated as unchanged against the
+corpus groundtruth; the TableFormer output is byte-identical. See
+[`PDF_PERFORMANCE.md`](./PDF_PERFORMANCE.md) for the measurements.
+
+**The pipeline uses them automatically** whenever they sit next to the fp32
+files at the default paths (`download_dependencies.sh` fetches them by
+default; `--no-int8` skips, or build them with `python
+scripts/quantize_models.py`). To force full precision:
 
 ```bash
-scripts/download_dependencies.sh --int8   # or: python scripts/quantize_models.py
-export DOCLING_LAYOUT_ONNX=$PWD/models/layout_heron_int8.onnx
-export DOCLING_TABLEFORMER_DECODER=$PWD/models/tableformer/decoder_int8.onnx
+FLEISCHWOLF_FP32=1 fleischwolf input.pdf          # keep the int8 files, use fp32
+# or pin a model explicitly — an explicit path always wins:
+export DOCLING_LAYOUT_ONNX=$PWD/models/layout_heron.onnx
+export DOCLING_TABLEFORMER_DECODER=$PWD/models/tableformer/decoder.onnx
 ```
 
-(The [example Dockerfile](./examples/Dockerfile) bakes and defaults to the
-INT8 models; build with `--build-arg INT8=0` for pure fp32.)
+(The [example Dockerfile](./examples/Dockerfile) bakes both precisions and
+defaults to INT8; build with `--build-arg INT8=0` for pure fp32.)
 
 Then either:
 

crates/fleischwolf-pdf/src/layout.rs

@@ -52,7 +52,9 @@ pub struct LayoutModel {
 }
 
 impl LayoutModel {
-    /// Load the ONNX model from `DOCLING_LAYOUT_ONNX` (or `models/layout_heron.onnx`).
+    /// Load the ONNX model from `DOCLING_LAYOUT_ONNX`. Without the override,
+    /// prefers `models/layout_heron_int8.onnx` when present (the quantized
+    /// default; `FLEISCHWOLF_FP32=1` opts out), else `models/layout_heron.onnx`.
     pub fn load() -> Result<Self, String> {
         Self::load_with(crate::intra_threads())
     }
@@ -61,8 +63,11 @@ impl LayoutModel {
     /// parallel page-worker pool loads its helper models on a single thread each
     /// and gets its speed-up from running pages concurrently instead.
     pub fn load_with(intra: usize) -> Result<Self, String> {
-        let path = std::env::var("DOCLING_LAYOUT_ONNX")
-            .unwrap_or_else(|_| "models/layout_heron.onnx".to_string());
+        let path = crate::model_path(
+            "DOCLING_LAYOUT_ONNX",
+            "models/layout_heron.onnx",
+            "models/layout_heron_int8.onnx",
+        );
         let session = Session::builder()
             .map_err(|e| format!("layout: builder: {e}"))?
             // Let inference use the available cores (ort otherwise defaults low);

crates/fleischwolf-pdf/src/lib.rs

@@ -74,6 +74,29 @@ pub(crate) fn intra_threads() -> usize {
         .unwrap_or(1)
 }
 
+/// True when `FLEISCHWOLF_FP32` (any value but `0`) forces the full-precision
+/// models even where an INT8 variant sits next to the fp32 default.
+pub(crate) fn fp32_forced() -> bool {
+    std::env::var("FLEISCHWOLF_FP32")
+        .map(|v| v != "0")
+        .unwrap_or(false)
+}
+
+/// Resolve a model path: an explicit env override always wins; otherwise the
+/// INT8 variant of the default path when it exists on disk (the quantized
+/// models are conformance-validated — see PDF_PERFORMANCE.md — and load/run
+/// markedly faster on CPU), unless `FLEISCHWOLF_FP32` opts back into full
+/// precision; else the fp32 default.
+pub(crate) fn model_path(env: &str, fp32_default: &str, int8_default: &str) -> String {
+    if let Ok(p) = std::env::var(env) {
+        return p;
+    }
+    if !fp32_forced() && std::path::Path::new(int8_default).exists() {
+        return int8_default.to_string();
+    }
+    fp32_default.to_string()
+}
+
 /// One page's assembled output: typed nodes plus the page's hyperlinks, kept
 /// separate so pages processed out of order can be stitched back in page order.
 type PageOut = (Vec<Node>, Vec<(String, String)>);

crates/fleischwolf-pdf/src/tableformer.rs

@@ -80,8 +80,13 @@ impl TableFormer {
     pub fn load_with(intra: usize) -> Option<Self> {
         let enc = std::env::var("DOCLING_TABLEFORMER_ENCODER")
             .unwrap_or_else(|_| "models/tableformer/encoder.onnx".to_string());
-        let dec = std::env::var("DOCLING_TABLEFORMER_DECODER")
-            .unwrap_or_else(|_| "models/tableformer/decoder.onnx".to_string());
+        // Prefer the INT8 decoder when present (byte-identical output, faster
+        // decode; FLEISCHWOLF_FP32=1 opts out) unless explicitly overridden.
+        let dec = crate::model_path(
+            "DOCLING_TABLEFORMER_DECODER",
+            "models/tableformer/decoder.onnx",
+            "models/tableformer/decoder_int8.onnx",
+        );
         let bbx = std::env::var("DOCLING_TABLEFORMER_BBOX")
             .unwrap_or_else(|_| "models/tableformer/bbox.onnx".to_string());
         if [&enc, &dec, &bbx]

scripts/download_dependencies.sh

@@ -29,13 +29,15 @@
 #   models/asr/{encoder_model,decoder_model}.onnx + vocab.json   (Whisper tiny,
 #     from Hugging Face; skip with --no-asr)
 #
-# With --int8, additionally fetches the INT8-quantized CPU models (see
+# Also fetches the INT8-quantized CPU models when the release hosts them (see
 # PDF_PERFORMANCE.md — ~2.4x faster layout inference at unchanged conformance):
 #   models/layout_heron_int8.onnx
 #   models/tableformer/decoder_int8.onnx
-# and prints the DOCLING_* exports that opt the pipeline into them. If the
-# release doesn't host the int8 assets (older tag), it falls back to telling
-# you how to produce them locally with scripts/quantize_models.py.
+# The pipeline picks these up automatically when they sit next to the fp32
+# files (no env vars needed); set FLEISCHWOLF_FP32=1 at runtime to force full
+# precision, or skip fetching them entirely with --no-int8. If the release
+# doesn't host the int8 assets (older tag), a note explains how to produce
+# them locally with scripts/quantize_models.py.
 #
 # pdfium is Linux x64 only for now, matching what's hosted in the release; for
 # other platforms (or to build the models from source) see scripts/pdf_setup.sh.
@@ -52,14 +54,15 @@ ASR_BASE_URL="${FLEISCHWOLF_ASR_MODELS_URL:-https://huggingface.co/onnx-communit
 
 FORCE=false
 WITH_ASR=true
-WITH_INT8=false
+WITH_INT8=true
 for arg in "$@"; do
   case "$arg" in
     --force) FORCE=true ;;
     --no-asr) WITH_ASR=false ;;
-    --int8) WITH_INT8=true ;;
+    --int8) WITH_INT8=true ;; # accepted for compatibility; int8 is the default
+    --no-int8) WITH_INT8=false ;;
     *)
-      echo "usage: download_dependencies.sh [--force] [--no-asr] [--int8]" >&2
+      echo "usage: download_dependencies.sh [--force] [--no-asr] [--no-int8]" >&2
       exit 2
       ;;
   esac
@@ -120,20 +123,18 @@ if [ "$WITH_ASR" = true ]; then
 fi
 
 if [ "$WITH_INT8" = true ]; then
-  # INT8-quantized CPU models (optional release assets). The pipeline stays on
-  # fp32 unless the DOCLING_* env vars below point at these files.
+  # INT8-quantized CPU models (optional release assets). The pipeline prefers
+  # them automatically when they sit at the default paths; FLEISCHWOLF_FP32=1
+  # forces the fp32 models at runtime.
   fetch_optional "$BASE_URL/layout_heron_int8.onnx" models/layout_heron_int8.onnx
   fetch_optional "$BASE_URL/decoder_int8.onnx" models/tableformer/decoder_int8.onnx
   if [ -f models/layout_heron_int8.onnx ]; then
-    echo "int8 models fetched — opt in with:"
-    echo "  export DOCLING_LAYOUT_ONNX=$(pwd)/models/layout_heron_int8.onnx"
-    if [ -f models/tableformer/decoder_int8.onnx ]; then
-      echo "  export DOCLING_TABLEFORMER_DECODER=$(pwd)/models/tableformer/decoder_int8.onnx"
-    fi
+    echo "int8 models present — used by default (FLEISCHWOLF_FP32=1 forces full precision)"
   else
-    echo "int8 assets not hosted at $BASE_URL — build them locally instead:"
+    echo "int8 assets not hosted at $BASE_URL — the fp32 models will be used."
+    echo "To build the int8 models locally (see PDF_PERFORMANCE.md):"
     echo "  pip install onnx onnxruntime sympy pypdfium2 pillow numpy"
-    echo "  python scripts/quantize_models.py    # see PDF_PERFORMANCE.md"
+    echo "  python scripts/quantize_models.py"
   fi
 fi
 

scripts/pdf_setup.sh

@@ -50,10 +50,12 @@ if [ ! -f models/tableformer/decoder.onnx ]; then
   fi
 fi
 
-# Optional: INT8-quantize the layout model + TableFormer decoder for faster CPU
-# inference (validated conformance-neutral — see PDF_PERFORMANCE.md). Opt in
-# with FLEISCHWOLF_INT8=1; needs onnx onnxruntime sympy pypdfium2 pillow numpy.
-if [ "${FLEISCHWOLF_INT8:-0}" = "1" ] && [ ! -f models/layout_heron_int8.onnx ]; then
+# INT8-quantize the layout model + TableFormer decoder for faster CPU
+# inference (validated conformance-neutral — see PDF_PERFORMANCE.md). The
+# pipeline prefers the int8 files automatically once they exist; skip building
+# them with FLEISCHWOLF_FP32=1. Needs onnx onnxruntime sympy pypdfium2 pillow
+# numpy — a missing-deps failure is non-fatal (fp32 keeps working).
+if [ "${FLEISCHWOLF_FP32:-0}" != "1" ] && [ ! -f models/layout_heron_int8.onnx ]; then
   echo "→ INT8-quantizing layout + TableFormer decoder"
   if ! "${PYTHON:-python3}" scripts/quantize_models.py; then
     echo "  ! quantization failed (missing deps?). The fp32 models still work;"
@@ -64,7 +66,7 @@ fi
 echo "done. export these before running the pipeline:"
 echo "  export PDFIUM_DYNAMIC_LIB_PATH=$(pwd)/.pdfium/lib"
 if [ -f models/layout_heron_int8.onnx ]; then
-  echo "  export DOCLING_LAYOUT_ONNX=$(pwd)/models/layout_heron_int8.onnx   # int8 (fp32: layout_heron.onnx)"
+  echo "  export DOCLING_LAYOUT_ONNX=$(pwd)/models/layout_heron_int8.onnx   # int8 default (fp32: layout_heron.onnx, or FLEISCHWOLF_FP32=1)"
 else
   echo "  export DOCLING_LAYOUT_ONNX=$(pwd)/models/layout_heron.onnx"
 fi
@@ -77,7 +79,7 @@ if [ -f models/tableformer/decoder.onnx ]; then
   # no matter where the binary/binding is actually invoked from.
   echo "  export DOCLING_TABLEFORMER_ENCODER=$(pwd)/models/tableformer/encoder.onnx"
   if [ -f models/tableformer/decoder_int8.onnx ]; then
-    echo "  export DOCLING_TABLEFORMER_DECODER=$(pwd)/models/tableformer/decoder_int8.onnx   # int8 (fp32: decoder.onnx)"
+    echo "  export DOCLING_TABLEFORMER_DECODER=$(pwd)/models/tableformer/decoder_int8.onnx   # int8 default (fp32: decoder.onnx, or FLEISCHWOLF_FP32=1)"
   else
     echo "  export DOCLING_TABLEFORMER_DECODER=$(pwd)/models/tableformer/decoder.onnx"
   fi

scripts/performance.sh

@@ -42,15 +42,11 @@ RUST_BIN="$(build_rust_release)"
 # Point the Rust PDF pipeline at the fetched libs/models (scripts/pdf_setup.sh)
 # using absolute paths, so it runs the full pipeline no matter the caller's CWD.
 # Harmless for non-PDF inputs (the binary only reads these for PDFs/images).
-# FLEISCHWOLF_INT8=1 benchmarks the INT8-quantized layout model + TableFormer
-# decoder instead (scripts/quantize_models.py; see PDF_PERFORMANCE.md).
+# Mirrors the pipeline's own default: the INT8 layout model + TableFormer
+# decoder when present (scripts/quantize_models.py; see PDF_PERFORMANCE.md),
+# fp32 with FLEISCHWOLF_FP32=1.
 [[ -e "$WORKSPACE_DIR/.pdfium/lib/libpdfium.so" ]] && export PDFIUM_DYNAMIC_LIB_PATH="${PDFIUM_DYNAMIC_LIB_PATH:-$WORKSPACE_DIR/.pdfium/lib}"
-if [[ "${FLEISCHWOLF_INT8:-0}" == "1" ]]; then
-  [[ -e "$WORKSPACE_DIR/models/layout_heron_int8.onnx" ]] || {
-    echo "error: FLEISCHWOLF_INT8=1 but models/layout_heron_int8.onnx is missing —" >&2
-    echo "run scripts/quantize_models.py (or download_dependencies.sh --int8) first" >&2
-    exit 1
-  }
+if [[ "${FLEISCHWOLF_FP32:-0}" != "1" && -e "$WORKSPACE_DIR/models/layout_heron_int8.onnx" ]]; then
   export DOCLING_LAYOUT_ONNX="${DOCLING_LAYOUT_ONNX:-$WORKSPACE_DIR/models/layout_heron_int8.onnx}"
   [[ -e "$WORKSPACE_DIR/models/tableformer/decoder_int8.onnx" ]] && export DOCLING_TABLEFORMER_DECODER="${DOCLING_TABLEFORMER_DECODER:-$WORKSPACE_DIR/models/tableformer/decoder_int8.onnx}"
 fi