Skip to content

Commit 7725289

Browse files
artizartiz
authored
Merge pull request #18 from artiz/fix/tableformer-silent-fallback
Improve TableFormer model loading and add memory pattern optimization
2 parents 105c27a + c58b43d commit 7725289

16 files changed

Lines changed: 476 additions & 302 deletions

File tree

‎.github/workflows/publish-models.yml‎

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
# Export the RT-DETR layout model + TableFormer to ONNX, re-host pdfium + the
2+
# OCR model, and publish everything as GitHub Release assets, so
3+
# `scripts/download_dependencies.sh` can fetch the whole PDF/image ML stack
4+
# from a single host (this repo's releases) with zero configuration.
5+
#
6+
# The layout model and TableFormer are PyTorch→ONNX *format conversions* of
7+
# docling-project's own models (Apache-2.0 / CDLA-Permissive-2.0 — see
8+
# MODELS_NOTICE.md); no weights are retrained or modified here. pdfium and the
9+
# OCR model are re-hosted third-party binaries, unmodified, from their own
10+
# public releases — mirrored here only so `download_dependencies.sh` needs one
11+
# host instead of three.
12+
#
13+
# Trigger: manual only (workflow_dispatch) — it installs torch and exports two
14+
# models, several GB and 10-20 minutes, not something to run on every push.
15+
# Re-running with the same tag re-uploads (clobbers) matching asset names, so
16+
# it's safe to retry after fixing a failed export.
17+
#
18+
# gh workflow run publish-models.yml # tag = models-v1
19+
# gh workflow run publish-models.yml -f tag=models-v2 # a future re-export
20+
21+
name: publish models
22+
23+
on:
24+
workflow_dispatch:
25+
inputs:
26+
tag:
27+
description: "Release tag to publish/update (bump only when the export itself changes)."
28+
required: false
29+
default: "models-v1"
30+
31+
permissions:
32+
contents: write # create the release + upload assets
33+
34+
jobs:
35+
publish-models:
36+
name: export + publish
37+
runs-on: ubuntu-latest
38+
timeout-minutes: 60
39+
env:
40+
GH_TOKEN: ${{ github.token }}
41+
TAG: ${{ inputs.tag }}
42+
steps:
43+
- uses: actions/checkout@v5
44+
45+
- uses: actions/setup-python@v5
46+
with:
47+
python-version: "3.11"
48+
49+
# --- layout model (required) --------------------------------------
50+
- name: Install layout export deps
51+
run: pip install --upgrade pip && pip install torch transformers onnx
52+
53+
- name: Export layout model
54+
run: python scripts/export_layout.py models/layout_heron.onnx
55+
56+
# --- TableFormer (optional — a failure here doesn't block layout) ---
57+
- name: Install TableFormer export deps
58+
continue-on-error: true
59+
id: tf-deps
60+
run: pip install docling_ibm_models onnxscript onnxruntime huggingface_hub
61+
62+
- name: Export TableFormer
63+
if: steps.tf-deps.outcome == 'success'
64+
continue-on-error: true
65+
id: tf-export
66+
run: python scripts/export_tableformer.py models/tableformer
67+
68+
# --- pdfium + OCR (re-hosted third-party binaries, not exports) -----
69+
- name: Fetch pdfium + OCR assets
70+
run: |
71+
mkdir -p third-party
72+
curl -fsSL -o /tmp/pdfium.tgz \
73+
"https://github.com/bblanchon/pdfium-binaries/releases/latest/download/pdfium-linux-x64.tgz"
74+
tar -xzf /tmp/pdfium.tgz -C third-party
75+
curl -fsSL -o third-party/ocr_rec.onnx \
76+
"https://huggingface.co/SWHL/RapidOCR/resolve/main/PP-OCRv3/ch_PP-OCRv3_rec_infer.onnx"
77+
curl -fsSL -o third-party/ppocr_keys_v1.txt \
78+
"https://raw.githubusercontent.com/PaddlePaddle/PaddleOCR/main/ppocr/utils/ppocr_keys_v1.txt"
79+
80+
# --- stage + publish -------------------------------------------------
81+
# GitHub release assets can't contain "/", so tableformer/*.onnx is
82+
# flattened to bare encoder.onnx / decoder.onnx / bbox.onnx (no naming
83+
# collision with anything else in this release; download_dependencies.sh
84+
# downloads these exact names). Each file is staged only if the export
85+
# produced it, including the optional `.onnx.data` external-weights
86+
# sidecar some exports need above ONNX's ~2GB protobuf limit (currently
87+
# just the TableFormer decoder, but checked for every file since that's
88+
# export-size dependent, not fixed).
89+
- name: Stage release assets
90+
run: |
91+
mkdir -p release-assets
92+
stage() { # <source-path> <asset-name>
93+
[ -f "$1" ] && cp "$1" "release-assets/$2"
94+
[ -f "$1.data" ] && cp "$1.data" "release-assets/$2.data"
95+
}
96+
stage third-party/lib/libpdfium.so libpdfium.so
97+
stage third-party/ocr_rec.onnx ocr_rec.onnx
98+
stage third-party/ppocr_keys_v1.txt ppocr_keys_v1.txt
99+
stage models/layout_heron.onnx layout_heron.onnx
100+
stage models/tableformer/encoder.onnx encoder.onnx
101+
stage models/tableformer/decoder.onnx decoder.onnx
102+
stage models/tableformer/bbox.onnx bbox.onnx
103+
echo "staged:"
104+
ls -la release-assets/
105+
106+
- name: Publish release
107+
run: |
108+
if gh release view "$TAG" --repo "${{ github.repository }}" >/dev/null 2>&1; then
109+
gh release upload "$TAG" release-assets/* --repo "${{ github.repository }}" --clobber
110+
else
111+
gh release create "$TAG" release-assets/* \
112+
--repo "${{ github.repository }}" \
113+
--title "fleischwolf ML models ($TAG)" \
114+
--notes-file MODELS_NOTICE.md
115+
fi

‎MODELS_NOTICE.md‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# Third-party model notice
2+
3+
`fleischwolf`'s PDF/image pipeline uses two ONNX graphs that are **format
4+
conversions of docling-project's own PyTorch models**, not weights fleischwolf
5+
trains or modifies. They're licensed separately from fleischwolf's own MIT
6+
code (see [`LICENSE`](./LICENSE)) under their upstream terms:
7+
8+
| Model | Source | License |
9+
|---|---|---|
10+
| RT-DETR layout model (`layout_heron.onnx`) | [`docling-project/docling-layout-heron`](https://huggingface.co/docling-project/docling-layout-heron) | Apache-2.0 |
11+
| TableFormer (`tableformer/{encoder,decoder,bbox}.onnx`) | [`docling-project/docling-models`](https://huggingface.co/docling-project/docling-models) (`model_artifacts/tableformer/accurate`) | CDLA-Permissive-2.0 / Apache-2.0 |
12+
13+
`scripts/export_layout.py` and `scripts/export_tableformer.py` do the
14+
conversion (PyTorch → ONNX via `torch.onnx.export`); no weights are retrained,
15+
fine-tuned, or otherwise altered. `.github/workflows/publish-models.yml` runs
16+
that conversion (and re-hosts pdfium + the OCR model alongside it) and
17+
publishes everything as GitHub Release assets on this repo (tag `models-v1`),
18+
fetched by `scripts/download_dependencies.sh` — see that script and
19+
`crates/fleischwolf-node/deps.js` — purely to spare downstream users the
20+
PyTorch/`transformers`/`docling_ibm_models` toolchain needed to export them
21+
locally.
22+
23+
Both upstream licenses permit redistribution with attribution; this file is
24+
that attribution. See the linked model cards for the full license text and any
25+
additional upstream terms.

‎README.md‎

Lines changed: 25 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -192,9 +192,18 @@ const json = await convertFileAsync('report.docx', { to: 'json' })
192192

193193
Declarative formats (Markdown, HTML, DOCX, XLSX, …) work out of the box. The
194194
PDF/image pipeline needs pdfium + the ONNX models (not bundled), so it throws
195-
until you call `installDependencies()` — which auto-downloads pdfium/OCR and
196-
fetches the layout/TableFormer ONNX from a `modelsUrl` you host. A reusable
197-
`Pipeline` keeps those models warm across many PDFs.
195+
until you fetch them — a one-liner from your app's directory (pdfium and OCR
196+
from their own upstream releases; the layout model and TableFormer —
197+
PyTorch→ONNX exports of docling-project's own models,
198+
Apache-2.0/CDLA-Permissive-2.0, see [`MODELS_NOTICE.md`](./MODELS_NOTICE.md) —
199+
from fleischwolf's own hosted release), straight into `./models` and
200+
`./.pdfium`, which the package looks for by default — no env vars needed:
201+
202+
```bash
203+
curl -fsSL https://raw.githubusercontent.com/artiz/fleischwolf/master/scripts/download_dependencies.sh | sh
204+
```
205+
206+
A reusable `Pipeline` keeps those models warm across many PDFs.
198207

199208
Runnable Node + Bun examples are in
200209
[`crates/fleischwolf-node/examples`](./crates/fleischwolf-node/examples)
@@ -237,6 +246,14 @@ export PDFIUM_DYNAMIC_LIB_PATH="$(pwd)/.pdfium/lib"
237246
export DOCLING_LAYOUT_ONNX="$(pwd)/models/layout_heron.onnx"
238247
export DOCLING_OCR_REC_ONNX="$(pwd)/models/ocr_rec.onnx"
239248
export DOCLING_OCR_DICT="$(pwd)/models/ppocr_keys_v1.txt"
249+
# Optional (falls back to geometric table reconstruction if unset/missing —
250+
# but the fallback is *silent*, so set these to be sure TableFormer is used,
251+
# especially if you invoke fleischwolf from anywhere but the repo root: the
252+
# defaults baked into the binary are relative paths, so a different working
253+
# directory makes them silently miss even when the files exist elsewhere).
254+
export DOCLING_TABLEFORMER_ENCODER="$(pwd)/models/tableformer/encoder.onnx"
255+
export DOCLING_TABLEFORMER_DECODER="$(pwd)/models/tableformer/decoder.onnx"
256+
export DOCLING_TABLEFORMER_BBOX="$(pwd)/models/tableformer/bbox.onnx"
240257
bash scripts/pdf_conformance.sh # regenerate + diff the snapshot baseline (91/91)
241258
```
242259

@@ -251,6 +268,11 @@ cargo run -p fleischwolf-cli -- --strict crates/fleischwolf/sample.html
251268
cargo run -p fleischwolf-cli -- --to json crates/fleischwolf/sample.html
252269
cargo run -p fleischwolf-cli -- --to json crates/fleischwolf/sample.html > out.json
253270

271+
# PDF/image conversion needs the ML models: scripts/download_dependencies.sh once,
272+
# then it just works — models/ and .pdfium/lib are picked up automatically.
273+
scripts/download_dependencies.sh
274+
cargo run -p fleischwolf-cli -- document.pdf
275+
254276
# extract pictures (PDF/image inputs): embed as data URIs, or write ./artifacts/*.png
255277
cargo run -p fleischwolf-cli -- --images embedded document.pdf
256278
cargo run -p fleischwolf-cli -- --images referenced document.pdf > out.md

‎crates/fleischwolf-node/README.md‎

Lines changed: 44 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -93,50 +93,58 @@ for await (const chunk of streamFileMarkdown('paper.pdf')) {
9393
}
9494
```
9595

96-
### PDF / images: installing the ML models
96+
### PDF / images: getting the ML models
9797

9898
Declarative formats (Markdown, HTML, DOCX, XLSX, …) are pure Rust and need
9999
nothing. The **PDF/image** path needs native assets that are *not* bundled in the
100-
addon — pdfium plus the ONNX models (layout, OCR, TableFormer) — the same way
101-
Python docling downloads its models on first use. Converting a PDF/image/METS
102-
input **throws** until they're installed:
100+
addon — pdfium plus the ONNX models (layout, OCR, TableFormer). Converting a
101+
PDF/image/METS input **throws** until they're on disk. Fetch them with a
102+
one-liner from your app's directory (where you'll `npm install fleischwolf`):
103103

104-
```js
105-
import { installDependencies, checkDependencies, convertFileAsync } from 'fleischwolf'
104+
```bash
105+
curl -fsSL https://raw.githubusercontent.com/artiz/fleischwolf/master/scripts/download_dependencies.sh | sh
106+
```
106107

107-
await convertFileAsync('paper.pdf') // ❌ throws: "requires the PDF/ML dependencies … call installDependencies()"
108+
```js
109+
import { convertFileAsync } from 'fleischwolf'
108110

109-
await installDependencies() // provisions everything, then:
110-
await convertFileAsync('paper.pdf') // ✅ works
111+
const res = await convertFileAsync('paper.pdf', { to: 'markdown' }) // ✅ works
111112
```
112113

113-
What `installDependencies()` fetches, into `~/.cache/fleischwolf` (override with
114-
`dir` or `$FLEISCHWOLF_HOME`), wiring the matching `DOCLING_*` /
115-
`PDFIUM_DYNAMIC_LIB_PATH` env vars in-process:
116-
117-
| Asset | Source | Required for |
118-
| --- | --- | --- |
119-
| **pdfium** | bblanchon prebuilt (auto, platform-detected) | PDF |
120-
| **OCR** rec model + dictionary | HuggingFace / GitHub (auto) | scanned pages |
121-
| **layout** (`layout_heron.onnx`) | your `modelsUrl` (see below) | PDF **and** image |
122-
| **TableFormer** (`tableformer/*.onnx`) | your `modelsUrl` | tables (else geometric fallback) |
123-
124-
> **layout + TableFormer have no public prebuilt download.** They're PyTorch→ONNX
125-
> exports (`docling-project/docling-layout-heron`, `docling_ibm_models`). Host the
126-
> exported `.onnx` yourself and point `installDependencies` at the base URL via
127-
> `{ modelsUrl }` or `FLEISCHWOLF_MODELS_URL` — it fetches `layout_heron.onnx` and
128-
> `tableformer/{encoder,decoder,bbox}.onnx` from there. Or export them locally
129-
> (repo `scripts/export_layout.py`, `scripts/export_tableformer.py`) and set
130-
> `DOCLING_LAYOUT_ONNX` / `DOCLING_TABLEFORMER_*` — `installDependencies` detects
131-
> those as already installed. Without a layout source it installs pdfium/OCR and
132-
> throws, naming what's missing.
114+
`scripts/download_dependencies.sh` fetches everything from this repo's
115+
[GitHub Releases](https://github.com/artiz/fleischwolf/releases) straight into
116+
`./models` and `./.pdfium` — which this package (and the Rust CLI) look for by
117+
default, relative to the process's current directory, so no env vars or setup
118+
call are needed afterwards:
119+
120+
| Asset | Destination |
121+
| --- | --- |
122+
| **pdfium** | `.pdfium/lib/libpdfium.so` |
123+
| **layout** (`layout_heron.onnx`) | `models/layout_heron.onnx` |
124+
| **OCR** rec model + dictionary | `models/ocr_rec.onnx`, `models/ppocr_keys_v1.txt` |
125+
| **TableFormer** | `models/tableformer/{encoder,decoder,bbox}.onnx` |
126+
127+
> **layout + TableFormer are PyTorch→ONNX exports**
128+
> (`docling-project/docling-layout-heron`, Apache-2.0;
129+
> `docling-project/docling-models`, CDLA-Permissive-2.0/Apache-2.0 — see
130+
> [`MODELS_NOTICE.md`](../../MODELS_NOTICE.md) for full attribution), not
131+
> fleischwolf's own weights — fleischwolf hosts the converted `.onnx` as a
132+
> GitHub Release purely so you don't need a local Python/torch toolchain.
133+
> pdfium and the OCR model are re-hosted, unmodified, from their own public
134+
> releases, on the same host for convenience.
135+
>
136+
> Run it from wherever your app lives — the script only writes to `./models`
137+
> and `./.pdfium` under the current directory, e.g. in a container build step:
138+
> ```bash
139+
> cd /path/to/your/app && curl -fsSL https://raw.githubusercontent.com/artiz/fleischwolf/master/scripts/download_dependencies.sh | sh
140+
> ```
141+
>
142+
> To use your own export/host instead, point the env vars at it directly:
143+
> `DOCLING_LAYOUT_ONNX`, `DOCLING_OCR_REC_ONNX`, `DOCLING_OCR_DICT`,
144+
> `DOCLING_TABLEFORMER_{ENCODER,DECODER,BBOX}`, `PDFIUM_DYNAMIC_LIB_PATH` — an
145+
> env var always wins over the `./models` / `./.pdfium` default.
133146
134147
```js
135-
await installDependencies({
136-
modelsUrl: 'https://you.example/fleischwolf-models', // serves layout_heron.onnx, tableformer/*.onnx
137-
onProgress: (m) => console.log(m),
138-
})
139-
140148
checkDependencies() // { home, pdfium, layout, ocr, tableformer, ready, missing }
141149
```
142150
@@ -188,7 +196,6 @@ JSON output always embeds extracted images as data URIs.
188196
| `streamFileMarkdown(path, options?)` | `AsyncGenerator<string>` | Markdown chunks in document order. |
189197
| `supportedFormats()` | `string[]` | Supported input format ids. |
190198
| `formatFromName(name)` | `string \| null` | Detect a format id from a filename/extension. |
191-
| `installDependencies(options?)` | `Promise<DependencyStatus>` | Download/validate the PDF/image models + pdfium. |
192199
| `checkDependencies(options?)` | `DependencyStatus` | Report which PDF/image deps are present. |
193200
194201
`Pipeline` is the reusable warm PDF/image converter: `new Pipeline(converterOptions)`
@@ -235,12 +242,12 @@ cd examples
235242
npm install
236243
node node-basic.mjs # ESM: file, bytes, JSON, reuse
237244
bun run bun-basic.ts # Bun + TypeScript: async + streaming
238-
node pdf-pipeline.mjs # installDependencies + warm Pipeline for PDFs
245+
node pdf-pipeline.mjs # warm Pipeline for PDFs (run scripts/download_dependencies.sh first)
239246
```
240247
241248
- [`examples/node-basic.mjs`](examples/node-basic.mjs) — Node.js (ESM): file, bytes, JSON, reuse.
242249
- [`examples/bun-basic.ts`](examples/bun-basic.ts) — Bun + TypeScript, with async and streaming.
243-
- [`examples/pdf-pipeline.mjs`](examples/pdf-pipeline.mjs) — `installDependencies` + warm `Pipeline` for PDFs.
250+
- [`examples/pdf-pipeline.mjs`](examples/pdf-pipeline.mjs) — warm `Pipeline` for PDFs.
244251
245252
The smoke test exercises the locally-built addon instead: `npm run build` once at
246253
the package root, then `node test/smoke.mjs` (or `bun test/smoke.mjs`).

0 commit comments

Comments
 (0)