Improve TableFormer model loading and add memory pattern optimization

.github/workflows/publish-models.yml

@@ -0,0 +1,115 @@
+# Export the RT-DETR layout model + TableFormer to ONNX, re-host pdfium + the
+# OCR model, and publish everything as GitHub Release assets, so
+# `scripts/download_dependencies.sh` can fetch the whole PDF/image ML stack
+# from a single host (this repo's releases) with zero configuration.
+#
+# The layout model and TableFormer are PyTorch→ONNX *format conversions* of
+# docling-project's own models (Apache-2.0 / CDLA-Permissive-2.0 — see
+# MODELS_NOTICE.md); no weights are retrained or modified here. pdfium and the
+# OCR model are re-hosted third-party binaries, unmodified, from their own
+# public releases — mirrored here only so `download_dependencies.sh` needs one
+# host instead of three.
+#
+# Trigger: manual only (workflow_dispatch) — it installs torch and exports two
+# models, several GB and 10-20 minutes, not something to run on every push.
+# Re-running with the same tag re-uploads (clobbers) matching asset names, so
+# it's safe to retry after fixing a failed export.
+#
+#   gh workflow run publish-models.yml                    # tag = models-v1
+#   gh workflow run publish-models.yml -f tag=models-v2    # a future re-export
+
+name: publish models
+
+on:
+  workflow_dispatch:
+    inputs:
+      tag:
+        description: "Release tag to publish/update (bump only when the export itself changes)."
+        required: false
+        default: "models-v1"
+
+permissions:
+  contents: write # create the release + upload assets
+
+jobs:
+  publish-models:
+    name: export + publish
+    runs-on: ubuntu-latest
+    timeout-minutes: 60
+    env:
+      GH_TOKEN: ${{ github.token }}
+      TAG: ${{ inputs.tag }}
+    steps:
+      - uses: actions/checkout@v5
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      # --- layout model (required) --------------------------------------
+      - name: Install layout export deps
+        run: pip install --upgrade pip && pip install torch transformers onnx
+
+      - name: Export layout model
+        run: python scripts/export_layout.py models/layout_heron.onnx
+
+      # --- TableFormer (optional — a failure here doesn't block layout) ---
+      - name: Install TableFormer export deps
+        continue-on-error: true
+        id: tf-deps
+        run: pip install docling_ibm_models onnxscript onnxruntime huggingface_hub
+
+      - name: Export TableFormer
+        if: steps.tf-deps.outcome == 'success'
+        continue-on-error: true
+        id: tf-export
+        run: python scripts/export_tableformer.py models/tableformer
+
+      # --- pdfium + OCR (re-hosted third-party binaries, not exports) -----
+      - name: Fetch pdfium + OCR assets
+        run: |
+          mkdir -p third-party
+          curl -fsSL -o /tmp/pdfium.tgz \
+            "https://github.com/bblanchon/pdfium-binaries/releases/latest/download/pdfium-linux-x64.tgz"
+          tar -xzf /tmp/pdfium.tgz -C third-party
+          curl -fsSL -o third-party/ocr_rec.onnx \
+            "https://huggingface.co/SWHL/RapidOCR/resolve/main/PP-OCRv3/ch_PP-OCRv3_rec_infer.onnx"
+          curl -fsSL -o third-party/ppocr_keys_v1.txt \
+            "https://raw.githubusercontent.com/PaddlePaddle/PaddleOCR/main/ppocr/utils/ppocr_keys_v1.txt"
+
+      # --- stage + publish -------------------------------------------------
+      # GitHub release assets can't contain "/", so tableformer/*.onnx is
+      # flattened to bare encoder.onnx / decoder.onnx / bbox.onnx (no naming
+      # collision with anything else in this release; download_dependencies.sh
+      # downloads these exact names). Each file is staged only if the export
+      # produced it, including the optional `.onnx.data` external-weights
+      # sidecar some exports need above ONNX's ~2GB protobuf limit (currently
+      # just the TableFormer decoder, but checked for every file since that's
+      # export-size dependent, not fixed).
+      - name: Stage release assets
+        run: |
+          mkdir -p release-assets
+          stage() { # <source-path> <asset-name>
+            [ -f "$1" ] && cp "$1" "release-assets/$2"
+            [ -f "$1.data" ] && cp "$1.data" "release-assets/$2.data"
+          }
+          stage third-party/lib/libpdfium.so libpdfium.so
+          stage third-party/ocr_rec.onnx ocr_rec.onnx
+          stage third-party/ppocr_keys_v1.txt ppocr_keys_v1.txt
+          stage models/layout_heron.onnx layout_heron.onnx
+          stage models/tableformer/encoder.onnx encoder.onnx
+          stage models/tableformer/decoder.onnx decoder.onnx
+          stage models/tableformer/bbox.onnx bbox.onnx
+          echo "staged:"
+          ls -la release-assets/
+
+      - name: Publish release
+        run: |
+          if gh release view "$TAG" --repo "${{ github.repository }}" >/dev/null 2>&1; then
+            gh release upload "$TAG" release-assets/* --repo "${{ github.repository }}" --clobber
+          else
+            gh release create "$TAG" release-assets/* \
+              --repo "${{ github.repository }}" \
+              --title "fleischwolf ML models ($TAG)" \
+              --notes-file MODELS_NOTICE.md
+          fi

MODELS_NOTICE.md

@@ -0,0 +1,25 @@
+# Third-party model notice
+
+`fleischwolf`'s PDF/image pipeline uses two ONNX graphs that are **format
+conversions of docling-project's own PyTorch models**, not weights fleischwolf
+trains or modifies. They're licensed separately from fleischwolf's own MIT
+code (see [`LICENSE`](./LICENSE)) under their upstream terms:
+
+| Model | Source | License |
+|---|---|---|
+| RT-DETR layout model (`layout_heron.onnx`) | [`docling-project/docling-layout-heron`](https://huggingface.co/docling-project/docling-layout-heron) | Apache-2.0 |
+| TableFormer (`tableformer/{encoder,decoder,bbox}.onnx`) | [`docling-project/docling-models`](https://huggingface.co/docling-project/docling-models) (`model_artifacts/tableformer/accurate`) | CDLA-Permissive-2.0 / Apache-2.0 |
+
+`scripts/export_layout.py` and `scripts/export_tableformer.py` do the
+conversion (PyTorch → ONNX via `torch.onnx.export`); no weights are retrained,
+fine-tuned, or otherwise altered. `.github/workflows/publish-models.yml` runs
+that conversion (and re-hosts pdfium + the OCR model alongside it) and
+publishes everything as GitHub Release assets on this repo (tag `models-v1`),
+fetched by `scripts/download_dependencies.sh` — see that script and
+`crates/fleischwolf-node/deps.js` — purely to spare downstream users the
+PyTorch/`transformers`/`docling_ibm_models` toolchain needed to export them
+locally.
+
+Both upstream licenses permit redistribution with attribution; this file is
+that attribution. See the linked model cards for the full license text and any
+additional upstream terms.

README.md

@@ -192,9 +192,18 @@ const json = await convertFileAsync('report.docx', { to: 'json' })
 
 Declarative formats (Markdown, HTML, DOCX, XLSX, …) work out of the box. The
 PDF/image pipeline needs pdfium + the ONNX models (not bundled), so it throws
-until you call `installDependencies()` — which auto-downloads pdfium/OCR and
-fetches the layout/TableFormer ONNX from a `modelsUrl` you host. A reusable
-`Pipeline` keeps those models warm across many PDFs.
+until you fetch them — a one-liner from your app's directory (pdfium and OCR
+from their own upstream releases; the layout model and TableFormer —
+PyTorch→ONNX exports of docling-project's own models,
+Apache-2.0/CDLA-Permissive-2.0, see [`MODELS_NOTICE.md`](./MODELS_NOTICE.md) —
+from fleischwolf's own hosted release), straight into `./models` and
+`./.pdfium`, which the package looks for by default — no env vars needed:
+
+```bash
+curl -fsSL https://raw.githubusercontent.com/artiz/fleischwolf/master/scripts/download_dependencies.sh | sh
+```
+
+A reusable `Pipeline` keeps those models warm across many PDFs.
 
 Runnable Node + Bun examples are in
 [`crates/fleischwolf-node/examples`](./crates/fleischwolf-node/examples)
@@ -237,6 +246,14 @@ export PDFIUM_DYNAMIC_LIB_PATH="$(pwd)/.pdfium/lib"
 export DOCLING_LAYOUT_ONNX="$(pwd)/models/layout_heron.onnx"
 export DOCLING_OCR_REC_ONNX="$(pwd)/models/ocr_rec.onnx"
 export DOCLING_OCR_DICT="$(pwd)/models/ppocr_keys_v1.txt"
+# Optional (falls back to geometric table reconstruction if unset/missing —
+# but the fallback is *silent*, so set these to be sure TableFormer is used,
+# especially if you invoke fleischwolf from anywhere but the repo root: the
+# defaults baked into the binary are relative paths, so a different working
+# directory makes them silently miss even when the files exist elsewhere).
+export DOCLING_TABLEFORMER_ENCODER="$(pwd)/models/tableformer/encoder.onnx"
+export DOCLING_TABLEFORMER_DECODER="$(pwd)/models/tableformer/decoder.onnx"
+export DOCLING_TABLEFORMER_BBOX="$(pwd)/models/tableformer/bbox.onnx"
 bash scripts/pdf_conformance.sh     # regenerate + diff the snapshot baseline (91/91)
 ```
 
@@ -251,6 +268,11 @@ cargo run -p fleischwolf-cli -- --strict crates/fleischwolf/sample.html
 cargo run -p fleischwolf-cli -- --to json crates/fleischwolf/sample.html
 cargo run -p fleischwolf-cli -- --to json crates/fleischwolf/sample.html > out.json
 
+# PDF/image conversion needs the ML models: scripts/download_dependencies.sh once,
+# then it just works — models/ and .pdfium/lib are picked up automatically.
+scripts/download_dependencies.sh
+cargo run -p fleischwolf-cli -- document.pdf
+
 # extract pictures (PDF/image inputs): embed as data URIs, or write ./artifacts/*.png
 cargo run -p fleischwolf-cli -- --images embedded   document.pdf
 cargo run -p fleischwolf-cli -- --images referenced document.pdf > out.md

crates/fleischwolf-node/README.md

@@ -93,50 +93,58 @@ for await (const chunk of streamFileMarkdown('paper.pdf')) {
 }
 ```
 
-### PDF / images: installing the ML models
+### PDF / images: getting the ML models
 
 Declarative formats (Markdown, HTML, DOCX, XLSX, …) are pure Rust and need
 nothing. The **PDF/image** path needs native assets that are *not* bundled in the
-addon — pdfium plus the ONNX models (layout, OCR, TableFormer) — the same way
-Python docling downloads its models on first use. Converting a PDF/image/METS
-input **throws** until they're installed:
+addon — pdfium plus the ONNX models (layout, OCR, TableFormer). Converting a
+PDF/image/METS input **throws** until they're on disk. Fetch them with a
+one-liner from your app's directory (where you'll `npm install fleischwolf`):
 
-```js
-import { installDependencies, checkDependencies, convertFileAsync } from 'fleischwolf'
+```bash
+curl -fsSL https://raw.githubusercontent.com/artiz/fleischwolf/master/scripts/download_dependencies.sh | sh
+```
 
-await convertFileAsync('paper.pdf') // ❌ throws: "requires the PDF/ML dependencies … call installDependencies()"
+```js
+import { convertFileAsync } from 'fleischwolf'
 
-await installDependencies()          // provisions everything, then:
-await convertFileAsync('paper.pdf')  // ✅ works
+const res = await convertFileAsync('paper.pdf', { to: 'markdown' }) // ✅ works
 ```
 
-What `installDependencies()` fetches, into `~/.cache/fleischwolf` (override with
-`dir` or `$FLEISCHWOLF_HOME`), wiring the matching `DOCLING_*` /
-`PDFIUM_DYNAMIC_LIB_PATH` env vars in-process:
-
-| Asset | Source | Required for |
-| --- | --- | --- |
-| **pdfium** | bblanchon prebuilt (auto, platform-detected) | PDF |
-| **OCR** rec model + dictionary | HuggingFace / GitHub (auto) | scanned pages |
-| **layout** (`layout_heron.onnx`) | your `modelsUrl` (see below) | PDF **and** image |
-| **TableFormer** (`tableformer/*.onnx`) | your `modelsUrl` | tables (else geometric fallback) |
-
-> **layout + TableFormer have no public prebuilt download.** They're PyTorch→ONNX
-> exports (`docling-project/docling-layout-heron`, `docling_ibm_models`). Host the
-> exported `.onnx` yourself and point `installDependencies` at the base URL via
-> `{ modelsUrl }` or `FLEISCHWOLF_MODELS_URL` — it fetches `layout_heron.onnx` and
-> `tableformer/{encoder,decoder,bbox}.onnx` from there. Or export them locally
-> (repo `scripts/export_layout.py`, `scripts/export_tableformer.py`) and set
-> `DOCLING_LAYOUT_ONNX` / `DOCLING_TABLEFORMER_*` — `installDependencies` detects
-> those as already installed. Without a layout source it installs pdfium/OCR and
-> throws, naming what's missing.
+`scripts/download_dependencies.sh` fetches everything from this repo's
+[GitHub Releases](https://github.com/artiz/fleischwolf/releases) straight into
+`./models` and `./.pdfium` — which this package (and the Rust CLI) look for by
+default, relative to the process's current directory, so no env vars or setup
+call are needed afterwards:
+
+| Asset | Destination |
+| --- | --- |
+| **pdfium** | `.pdfium/lib/libpdfium.so` |
+| **layout** (`layout_heron.onnx`) | `models/layout_heron.onnx` |
+| **OCR** rec model + dictionary | `models/ocr_rec.onnx`, `models/ppocr_keys_v1.txt` |
+| **TableFormer** | `models/tableformer/{encoder,decoder,bbox}.onnx` |
+
+> **layout + TableFormer are PyTorch→ONNX exports**
+> (`docling-project/docling-layout-heron`, Apache-2.0;
+> `docling-project/docling-models`, CDLA-Permissive-2.0/Apache-2.0 — see
+> [`MODELS_NOTICE.md`](../../MODELS_NOTICE.md) for full attribution), not
+> fleischwolf's own weights — fleischwolf hosts the converted `.onnx` as a
+> GitHub Release purely so you don't need a local Python/torch toolchain.
+> pdfium and the OCR model are re-hosted, unmodified, from their own public
+> releases, on the same host for convenience.
+>
+> Run it from wherever your app lives — the script only writes to `./models`
+> and `./.pdfium` under the current directory, e.g. in a container build step:
+> ```bash
+> cd /path/to/your/app && curl -fsSL https://raw.githubusercontent.com/artiz/fleischwolf/master/scripts/download_dependencies.sh | sh
+> ```
+>
+> To use your own export/host instead, point the env vars at it directly:
+> `DOCLING_LAYOUT_ONNX`, `DOCLING_OCR_REC_ONNX`, `DOCLING_OCR_DICT`,
+> `DOCLING_TABLEFORMER_{ENCODER,DECODER,BBOX}`, `PDFIUM_DYNAMIC_LIB_PATH` — an
+> env var always wins over the `./models` / `./.pdfium` default.
 
 ```js
-await installDependencies({
-  modelsUrl: 'https://you.example/fleischwolf-models', // serves layout_heron.onnx, tableformer/*.onnx
-  onProgress: (m) => console.log(m),
-})
-
 checkDependencies() // { home, pdfium, layout, ocr, tableformer, ready, missing }
 ```
 
@@ -188,7 +196,6 @@ JSON output always embeds extracted images as data URIs.
 | `streamFileMarkdown(path, options?)` | `AsyncGenerator<string>` | Markdown chunks in document order. |
 | `supportedFormats()` | `string[]` | Supported input format ids. |
 | `formatFromName(name)` | `string \| null` | Detect a format id from a filename/extension. |
-| `installDependencies(options?)` | `Promise<DependencyStatus>` | Download/validate the PDF/image models + pdfium. |
 | `checkDependencies(options?)` | `DependencyStatus` | Report which PDF/image deps are present. |
 
 `Pipeline` is the reusable warm PDF/image converter: `new Pipeline(converterOptions)`
@@ -235,12 +242,12 @@ cd examples
 npm install
 node node-basic.mjs        # ESM: file, bytes, JSON, reuse
 bun run bun-basic.ts       # Bun + TypeScript: async + streaming
-node pdf-pipeline.mjs       # installDependencies + warm Pipeline for PDFs
+node pdf-pipeline.mjs       # warm Pipeline for PDFs (run scripts/download_dependencies.sh first)
 ```
 
 - [`examples/node-basic.mjs`](examples/node-basic.mjs) — Node.js (ESM): file, bytes, JSON, reuse.
 - [`examples/bun-basic.ts`](examples/bun-basic.ts) — Bun + TypeScript, with async and streaming.
-- [`examples/pdf-pipeline.mjs`](examples/pdf-pipeline.mjs) — `installDependencies` + warm `Pipeline` for PDFs.
+- [`examples/pdf-pipeline.mjs`](examples/pdf-pipeline.mjs) — warm `Pipeline` for PDFs.
 
 The smoke test exercises the locally-built addon instead: `npm run build` once at
 the package root, then `node test/smoke.mjs` (or `bun test/smoke.mjs`).