feat(node,pdf): replace installDependencies() with scripts/download_dependencies.sh

installDependencies() duplicated what a plain bash script does better: fetch
pdfium + the ONNX models straight into ./models and ./.pdfium, which both the
Rust CLI and the Node bindings already default to relative to CWD. Removes the
JS download machinery entirely (deps.js keeps only path resolution + status
checks), deletes the now-pointless setup_nodejs_dependencies.sh wrapper, and
adds the CWD-relative .pdfium/lib fallback to pdfium_backend.rs so the Rust
side needs no env vars either.

publish-models.yml now also re-hosts pdfium + the OCR model (previously only
layout/TableFormer), so download_dependencies.sh talks to a single release.
TableFormer assets are published under their bare names (encoder.onnx,
decoder.onnx, bbox.onnx) instead of a tableformer- prefix, matching the
script's fixed target paths.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DofkqhMuAJbL9arnnuVisL

Author: claude

.github/workflows/publish-models.yml

@@ -1,11 +1,14 @@
-# Export the RT-DETR layout model + TableFormer to ONNX and publish them as
-# GitHub Release assets, so `installDependencies()` in the `fleischwolf` npm
-# package (crates/fleischwolf-node/deps.js) can fetch them with zero
-# configuration instead of every user needing a local torch toolchain.
+# Export the RT-DETR layout model + TableFormer to ONNX, re-host pdfium + the
+# OCR model, and publish everything as GitHub Release assets, so
+# `scripts/download_dependencies.sh` can fetch the whole PDF/image ML stack
+# from a single host (this repo's releases) with zero configuration.
 #
-# Both are PyTorch→ONNX *format conversions* of docling-project's own models
-# (Apache-2.0 / CDLA-Permissive-2.0 — see MODELS_NOTICE.md); no weights are
-# retrained or modified here.
+# The layout model and TableFormer are PyTorch→ONNX *format conversions* of
+# docling-project's own models (Apache-2.0 / CDLA-Permissive-2.0 — see
+# MODELS_NOTICE.md); no weights are retrained or modified here. pdfium and the
+# OCR model are re-hosted third-party binaries, unmodified, from their own
+# public releases — mirrored here only so `download_dependencies.sh` needs one
+# host instead of three.
 #
 # Trigger: manual only (workflow_dispatch) — it installs torch and exports two
 # models, several GB and 10-20 minutes, not something to run on every push.
@@ -62,13 +65,26 @@ jobs:
         id: tf-export
         run: python scripts/export_tableformer.py models/tableformer
 
+      # --- pdfium + OCR (re-hosted third-party binaries, not exports) -----
+      - name: Fetch pdfium + OCR assets
+        run: |
+          mkdir -p third-party
+          curl -fsSL -o /tmp/pdfium.tgz \
+            "https://github.com/bblanchon/pdfium-binaries/releases/latest/download/pdfium-linux-x64.tgz"
+          tar -xzf /tmp/pdfium.tgz -C third-party
+          curl -fsSL -o third-party/ocr_rec.onnx \
+            "https://huggingface.co/SWHL/RapidOCR/resolve/main/PP-OCRv3/ch_PP-OCRv3_rec_infer.onnx"
+          curl -fsSL -o third-party/ppocr_keys_v1.txt \
+            "https://raw.githubusercontent.com/PaddlePaddle/PaddleOCR/main/ppocr/utils/ppocr_keys_v1.txt"
+
       # --- stage + publish -------------------------------------------------
       # GitHub release assets can't contain "/", so tableformer/*.onnx is
-      # flattened to tableformer-*.onnx (deps.js downloads the matching flat
-      # names by default). Each file is staged only if the export produced it,
-      # including the optional `.onnx.data` external-weights sidecar some
-      # exports need above ONNX's ~2GB protobuf limit (currently just the
-      # TableFormer decoder, but checked for every file since that's
+      # flattened to bare encoder.onnx / decoder.onnx / bbox.onnx (no naming
+      # collision with anything else in this release; download_dependencies.sh
+      # downloads these exact names). Each file is staged only if the export
+      # produced it, including the optional `.onnx.data` external-weights
+      # sidecar some exports need above ONNX's ~2GB protobuf limit (currently
+      # just the TableFormer decoder, but checked for every file since that's
       # export-size dependent, not fixed).
       - name: Stage release assets
         run: |
@@ -77,10 +93,13 @@ jobs:
             [ -f "$1" ] && cp "$1" "release-assets/$2"
             [ -f "$1.data" ] && cp "$1.data" "release-assets/$2.data"
           }
+          stage third-party/lib/libpdfium.so libpdfium.so
+          stage third-party/ocr_rec.onnx ocr_rec.onnx
+          stage third-party/ppocr_keys_v1.txt ppocr_keys_v1.txt
           stage models/layout_heron.onnx layout_heron.onnx
-          stage models/tableformer/encoder.onnx tableformer-encoder.onnx
-          stage models/tableformer/decoder.onnx tableformer-decoder.onnx
-          stage models/tableformer/bbox.onnx tableformer-bbox.onnx
+          stage models/tableformer/encoder.onnx encoder.onnx
+          stage models/tableformer/decoder.onnx decoder.onnx
+          stage models/tableformer/bbox.onnx bbox.onnx
           echo "staged:"
           ls -la release-assets/
 

MODELS_NOTICE.md

@@ -13,10 +13,10 @@ code (see [`LICENSE`](./LICENSE)) under their upstream terms:
 `scripts/export_layout.py` and `scripts/export_tableformer.py` do the
 conversion (PyTorch → ONNX via `torch.onnx.export`); no weights are retrained,
 fine-tuned, or otherwise altered. `.github/workflows/publish-models.yml` runs
-that conversion and republishes the resulting `.onnx` files as GitHub Release
-assets on this repo (tag `models-v1`, downloaded automatically by
-`installDependencies()` in the `fleischwolf` npm package — see
-`crates/fleischwolf-node/deps.js`), purely to spare downstream users the
+that conversion (and re-hosts pdfium + the OCR model alongside it) and
+publishes everything as GitHub Release assets on this repo (tag `models-v1`),
+fetched by `scripts/download_dependencies.sh` — see that script and
+`crates/fleischwolf-node/deps.js` — purely to spare downstream users the
 PyTorch/`transformers`/`docling_ibm_models` toolchain needed to export them
 locally.
 

README.md

@@ -192,15 +192,15 @@ const json = await convertFileAsync('report.docx', { to: 'json' })
 
 Declarative formats (Markdown, HTML, DOCX, XLSX, …) work out of the box. The
 PDF/image pipeline needs pdfium + the ONNX models (not bundled), so it throws
-until you call `installDependencies()` — which downloads everything with zero
-configuration (pdfium and OCR from their own upstream releases; the layout
-model and TableFormer — PyTorch→ONNX exports of docling-project's own models,
+until you fetch them — a one-liner from your app's directory (pdfium and OCR
+from their own upstream releases; the layout model and TableFormer —
+PyTorch→ONNX exports of docling-project's own models,
 Apache-2.0/CDLA-Permissive-2.0, see [`MODELS_NOTICE.md`](./MODELS_NOTICE.md) —
-from fleischwolf's own hosted release). Pre-fetch everything ahead of time
-(e.g. in a container build step) with a one-liner from your app's directory:
+from fleischwolf's own hosted release), straight into `./models` and
+`./.pdfium`, which the package looks for by default — no env vars needed:
 
 ```bash
-curl -fsSL https://raw.githubusercontent.com/artiz/fleischwolf/master/scripts/setup_nodejs_dependencies.sh | bash
+curl -fsSL https://raw.githubusercontent.com/artiz/fleischwolf/master/scripts/download_dependencies.sh | sh
 ```
 
 A reusable `Pipeline` keeps those models warm across many PDFs.
@@ -268,6 +268,11 @@ cargo run -p fleischwolf-cli -- --strict crates/fleischwolf/sample.html
 cargo run -p fleischwolf-cli -- --to json crates/fleischwolf/sample.html
 cargo run -p fleischwolf-cli -- --to json crates/fleischwolf/sample.html > out.json
 
+# PDF/image conversion needs the ML models: scripts/download_dependencies.sh once,
+# then it just works — models/ and .pdfium/lib are picked up automatically.
+scripts/download_dependencies.sh
+cargo run -p fleischwolf-cli -- document.pdf
+
 # extract pictures (PDF/image inputs): embed as data URIs, or write ./artifacts/*.png
 cargo run -p fleischwolf-cli -- --images embedded   document.pdf
 cargo run -p fleischwolf-cli -- --images referenced document.pdf > out.md

crates/fleischwolf-node/README.md

@@ -93,61 +93,58 @@ for await (const chunk of streamFileMarkdown('paper.pdf')) {
 }
 ```
 
-### PDF / images: installing the ML models
+### PDF / images: getting the ML models
 
 Declarative formats (Markdown, HTML, DOCX, XLSX, …) are pure Rust and need
 nothing. The **PDF/image** path needs native assets that are *not* bundled in the
-addon — pdfium plus the ONNX models (layout, OCR, TableFormer) — the same way
-Python docling downloads its models on first use. Converting a PDF/image/METS
-input **throws** until they're installed:
+addon — pdfium plus the ONNX models (layout, OCR, TableFormer). Converting a
+PDF/image/METS input **throws** until they're on disk. Fetch them with a
+one-liner from your app's directory (where you'll `npm install fleischwolf`):
 
-```js
-import { installDependencies, checkDependencies, convertFileAsync } from 'fleischwolf'
+```bash
+curl -fsSL https://raw.githubusercontent.com/artiz/fleischwolf/master/scripts/download_dependencies.sh | sh
+```
 
-await convertFileAsync('paper.pdf') // ❌ throws: "requires the PDF/ML dependencies … call installDependencies()"
+```js
+import { convertFileAsync } from 'fleischwolf'
 
-await installDependencies()          // provisions everything, then:
-await convertFileAsync('paper.pdf')  // ✅ works
+const res = await convertFileAsync('paper.pdf', { to: 'markdown' }) // ✅ works
 ```
 
-What `installDependencies()` fetches, into `~/.cache/fleischwolf` (override with
-`dir` or `$FLEISCHWOLF_HOME`), wiring the matching `DOCLING_*` /
-`PDFIUM_DYNAMIC_LIB_PATH` env vars in-process:
+`scripts/download_dependencies.sh` fetches everything from this repo's
+[GitHub Releases](https://github.com/artiz/fleischwolf/releases) straight into
+`./models` and `./.pdfium` — which this package (and the Rust CLI) look for by
+default, relative to the process's current directory, so no env vars or setup
+call are needed afterwards:
 
-| Asset | Source | Required for |
-| --- | --- | --- |
-| **pdfium** | bblanchon prebuilt (auto, platform-detected) | PDF |
-| **OCR** rec model + dictionary | HuggingFace / GitHub (auto) | scanned pages |
-| **layout** (`layout_heron.onnx`) | fleischwolf's hosted release (auto) | PDF **and** image |
-| **TableFormer** (`tableformer-*.onnx`) | fleischwolf's hosted release (auto) | tables (else geometric fallback) |
+| Asset | Destination |
+| --- | --- |
+| **pdfium** | `.pdfium/lib/libpdfium.so` |
+| **layout** (`layout_heron.onnx`) | `models/layout_heron.onnx` |
+| **OCR** rec model + dictionary | `models/ocr_rec.onnx`, `models/ppocr_keys_v1.txt` |
+| **TableFormer** | `models/tableformer/{encoder,decoder,bbox}.onnx` |
 
 > **layout + TableFormer are PyTorch→ONNX exports**
 > (`docling-project/docling-layout-heron`, Apache-2.0;
 > `docling-project/docling-models`, CDLA-Permissive-2.0/Apache-2.0 — see
 > [`MODELS_NOTICE.md`](../../MODELS_NOTICE.md) for full attribution), not
-> fleischwolf's own weights. `installDependencies()` fetches the export
-> fleischwolf hosts as a GitHub Release by default — no configuration needed.
-> Override with `{ modelsUrl }` / `FLEISCHWOLF_MODELS_URL` to use your own
-> export/host instead (serving `layout_heron.onnx` and, optionally,
-> `tableformer-{encoder,decoder,bbox}.onnx`), or export locally (repo
-> `scripts/export_layout.py`, `scripts/export_tableformer.py`) and set
-> `DOCLING_LAYOUT_ONNX` / `DOCLING_TABLEFORMER_*` directly —
-> `installDependencies` detects those as already installed and skips fetching.
+> fleischwolf's own weights — fleischwolf hosts the converted `.onnx` as a
+> GitHub Release purely so you don't need a local Python/torch toolchain.
+> pdfium and the OCR model are re-hosted, unmodified, from their own public
+> releases, on the same host for convenience.
 >
-> Pre-fetch everything ahead of time (e.g. in a container build step), from
-> your app's directory:
+> Run it from wherever your app lives — the script only writes to `./models`
+> and `./.pdfium` under the current directory, e.g. in a container build step:
 > ```bash
-> curl -fsSL https://raw.githubusercontent.com/artiz/fleischwolf/master/scripts/setup_nodejs_dependencies.sh | bash
+> cd /path/to/your/app && curl -fsSL https://raw.githubusercontent.com/artiz/fleischwolf/master/scripts/download_dependencies.sh | sh
 > ```
+>
+> To use your own export/host instead, point the env vars at it directly:
+> `DOCLING_LAYOUT_ONNX`, `DOCLING_OCR_REC_ONNX`, `DOCLING_OCR_DICT`,
+> `DOCLING_TABLEFORMER_{ENCODER,DECODER,BBOX}`, `PDFIUM_DYNAMIC_LIB_PATH` — an
+> env var always wins over the `./models` / `./.pdfium` default.
 
 ```js
-await installDependencies({ onProgress: (m) => console.log(m) })
-
-// or, to use your own export/host instead of fleischwolf's default:
-await installDependencies({
-  modelsUrl: 'https://you.example/fleischwolf-models', // serves layout_heron.onnx, tableformer-*.onnx
-})
-
 checkDependencies() // { home, pdfium, layout, ocr, tableformer, ready, missing }
 ```
 
@@ -199,7 +196,6 @@ JSON output always embeds extracted images as data URIs.
 | `streamFileMarkdown(path, options?)` | `AsyncGenerator<string>` | Markdown chunks in document order. |
 | `supportedFormats()` | `string[]` | Supported input format ids. |
 | `formatFromName(name)` | `string \| null` | Detect a format id from a filename/extension. |
-| `installDependencies(options?)` | `Promise<DependencyStatus>` | Download/validate the PDF/image models + pdfium. |
 | `checkDependencies(options?)` | `DependencyStatus` | Report which PDF/image deps are present. |
 
 `Pipeline` is the reusable warm PDF/image converter: `new Pipeline(converterOptions)`
@@ -246,12 +242,12 @@ cd examples
 npm install
 node node-basic.mjs        # ESM: file, bytes, JSON, reuse
 bun run bun-basic.ts       # Bun + TypeScript: async + streaming
-node pdf-pipeline.mjs       # installDependencies + warm Pipeline for PDFs
+node pdf-pipeline.mjs       # warm Pipeline for PDFs (run scripts/download_dependencies.sh first)
 ```
 
 - [`examples/node-basic.mjs`](examples/node-basic.mjs) — Node.js (ESM): file, bytes, JSON, reuse.
 - [`examples/bun-basic.ts`](examples/bun-basic.ts) — Bun + TypeScript, with async and streaming.
-- [`examples/pdf-pipeline.mjs`](examples/pdf-pipeline.mjs) — `installDependencies` + warm `Pipeline` for PDFs.
+- [`examples/pdf-pipeline.mjs`](examples/pdf-pipeline.mjs) — warm `Pipeline` for PDFs.
 
 The smoke test exercises the locally-built addon instead: `npm run build` once at
 the package root, then `node test/smoke.mjs` (or `bun test/smoke.mjs`).