feat(asr): audio transcription — Whisper via ONNX, pure Rust (closes Phase 7)

Ports docling's AsrPipeline (native-Whisper path, its ASR defaults:
whisper tiny, temperature-0 greedy, timestamps) as a new fleischwolf-asr
crate, Rust all the way down:

- symphonia demuxes/decodes the audio in-process — wav, mp3, flac, ogg,
  aac, m4a, plus the audio track of mp4/mov; no ffmpeg. AVI is the one
  container symphonia cannot demux (clear error).
- A ported log-mel front-end (80 Slaney mel filters, centered STFT,
  Whisper's exact log/clamp/rescale), unit-tested against librosa
  reference values.
- Whisper tiny encoder/decoder ONNX (onnx-community export) on ort —
  the same runtime as layout/TableFormer/OCR. Greedy decoding implements
  OpenAI's rules: special-token + non-speech-symbol suppression (the
  default suppress_tokens="-1" set, derived by decode-scanning the
  vocabulary), suppress_blank, forced initial timestamp with
  max_initial_timestamp, timestamp pairing/monotonicity, the
  timestamp-probability-mass rule, and the no-speech gate
  (p_nospeech > 0.6 && avg_logprob < -1). 30-second windowing advances
  seek to the last emitted timestamp.
- GPT-2 byte-level BPE decoding from vocab.json alone (no merges needed
  for id→text). FLEISCHWOLF_ASR_LANG selects the language via
  added_tokens.json (default en).
- Each segment renders as a `[time: start-end] text` paragraph (docling's
  conversation form, Python float formatting).

Wiring: InputFormat::Audio now converts (the rejects-unimplemented test
moves to XmlDoclang, the one remaining backend-less format);
download_dependencies.sh fetches Whisper tiny from Hugging Face
(--no-asr to skip; DOCLING_ASR_{ENCODER,DECODER,VOCAB} override).

Verified end-to-end on tests/data/audio: all 12 audio fixtures + the
mp4/mov video fixtures transcribe the LibriVox sample correctly
("Shakespeare on scenery by Oscar Wilde…", [time: 0.0-10.0]); silence
yields whisper-tiny's documented model-faithful behavior. 8 new unit
tests (mel filterbank vs librosa, silence floor, timestamp rules, byte
decoder, resampler, wav decode, time formatting).

Docs: README (status, models table, try-it), MIGRATION (status:
migration complete; crate map; ML table; Phase 7 done), COMPARING (HTML
31/32 + refreshed gap table from the earlier commits).

Author: claude

COMPARING.md

@@ -181,7 +181,7 @@ case — see the divergence table below.
 | **XLSX** | **9 / 9** ✅ | 9 / 9 |
 | **PPTX** | **7 / 7** ✅ | 7 / 7 |
 | **DOCX** | **25 / 26** | 25 / 26 |
-| **HTML** | **28 / 33** | 28 / 33 |
+| **HTML** | **31 / 32** ‡ | 31 / 32 |
 | **PDF** | **6 / 14** † | 7 / 14 |
 
 > † The pure-parse backends above are scored against **live** docling. **PDF** is a
@@ -194,6 +194,16 @@ case — see the divergence table below.
 > `1⁄4` fraction, where the Rust output's single space is the more faithful
 > rendering.
 
+> ‡ **HTML** is scored against the committed groundtruth
+> (`tests/data/html/groundtruth`, regenerated from live docling — which now
+> emits *compact* markdown tables) with the **deliberate** padded-vs-compact
+> table divergence (§4 of `MIGRATION.md`) normalized; the cell *content* is
+> byte-identical, including docling's exact multi-space seams in deeply nested
+> table cells. The one remaining miss is `wiki_duck`: its menus are collapsed
+> by external, host-relative stylesheets, so suppressing them requires those
+> stylesheets to be fetchable at render time (`--use-web-browser` with network
+> access) — inherently out of reach for a fully-offline conversion.
+
 **PDF** (`*.pdf`) ports docling's *standard* (discriminative) PDF pipeline. A
 **pure-Rust text parser** (`textparse.rs`, on `lopdf`) reconstructs each glyph's
 box from the font's own advance widths and the PDF text/graphics matrices — the
@@ -261,10 +271,13 @@ the drawing parts via the shared `ooxml` zip/rels helper). The one miss
 all-empty table.
 
 CSV, Markdown, AsciiDoc, and the DeepSeek-OCR Markdown variant are fully
-one-to-one. HTML's 5 remaining misses are a tail of docling-internal behaviours —
-some requiring **headless-browser rendering**, others (a large Wikipedia page,
-key-value form extraction) needing substantial structural work — see below. 28/33
-is roughly the ceiling for a pure-parse port.
+one-to-one. HTML is at **31/32**: key-value form regions, docling-faithful
+inline-image handling, inline visibility suppression, and deep nested-table
+cell flattening (docling's exact spacing — BeautifulSoup whitespace semantics,
+not rendered geometry) are all reproduced statically, and CSS-cascade
+visibility suppression is available behind the optional `web-browser` feature.
+The single remaining miss (`wiki_duck`) needs the page's external stylesheets
+fetchable at render time — see below.
 
 **AsciiDoc** (`*.asciidoc`/`*.adoc`) ports docling's line-oriented
 `AsciiDocBackend`: titles/sections, nested bullet/numbered lists (all rendered
@@ -293,22 +306,17 @@ tail of docling-specific quirks (below), each typically 1–2 lines.
 
 ### HTML
 
-Against live docling: **10 / 32** exact, **12 / 32** whitespace-normalized. (The
-older committed groundtruth would report a lower 6/32 — it predates docling's
-padded-table serializer; see §A.) The remaining real differences trace to a
-small number of *systematic* behaviours below, not to parsing errors — closing
-each tends to fix several fixtures at once.
-
-### Known divergences (tracked conformance gaps)
-
-HTML — remaining gaps (4 of 32), all blocked or impractical:
+**31 / 32** content-exact against the committed groundtruth (regenerated from
+live docling); the table-padding divergence is deliberate (§4 of
+`MIGRATION.md`). The four gaps once thought browser-bound have all been closed,
+three of them **without any browser**:
 
-| # | Difference | Example | Why it's blocked |
+| # | Former gap | Example | Resolution |
 |---|---|---|---|
-| 1 | Browser-rendering visibility / nav suppression | `wiki_duck` | docling renders the page in a headless browser to drop nav/menu/sidebar cruft — not replicable from parsing alone |
-| 2 | Key-value-pair / form extraction | `kvp_data_example` | docling's `form_region` subsystem builds key/marker/value relations using rendered bounding boxes and synthesizes `<!-- missing-text -->` placeholders — browser-bbox-dependent |
-| 3 | Browser-hidden image (mobile nav) dropped | `hyperlink_02` | docling drops it via rendered visibility |
-| 4 | Deeply-nested-table padding when flattened into a cell | `table_06` | docling pads the rich-cell text using rendered table bounding boxes; shallow nesting matches, ≥3-level deep padding does not |
+| 1 | Key-value-pair / form extraction | `kvp_data_example` | detected statically from docling's `keyN` / `keyN_valueM` / `keyN_marker` `id`-convention → `field_region`/`field_item` nodes + `<!-- missing-text -->` markers; no bboxes involved |
+| 2 | Browser-hidden image (mobile nav) dropped | `hyperlink_02` | docling never emits inline image markers; restricting the wrapper→picture path to `<a>` and dropping inline images reproduces it statically |
+| 3 | Deeply-nested-table cell padding | `table_06` | not rendered geometry after all — BeautifulSoup collapses whitespace-only text nodes to `\n`/`' '`, docling's `get_text` adds a trailing space per cell, and the `\n` runs flatten to spaces at render; ported byte-exact |
+| 4 | Stylesheet-collapsed nav/menus | `wiki_duck` | CSS-cascade visibility genuinely needs a rendered page: available behind the `web-browser` feature / `--use-web-browser` (Rust-driven Chromium). The saved fixture's stylesheets are external + host-relative, so offline conversion cannot resolve them — the one remaining miss |
 
 Markdown — **10/10**. Both former gaps fixed: `signature`/`stamp` → image blocks,
 and the `2\.` escape via contiguity-aware text merging (pulldown splits both

Cargo.toml

@@ -1,9 +1,10 @@
 # Fleischwolf — a Rust port of docling.
 #
-# A Cargo workspace of four crates:
+# A Cargo workspace of five crates:
 #   fleischwolf-core  — the DoclingDocument data model + Markdown/JSON serializers
 #   fleischwolf       — the DocumentConverter and the per-format backends
 #   fleischwolf-pdf   — the PDF/image ML pipeline (pdfium text + ONNX layout/OCR)
+#   fleischwolf-asr   — the audio/ASR pipeline (symphonia decode + ONNX Whisper)
 #   fleischwolf-cli   — the `fleischwolf` command-line binary
 # See MIGRATION.md for the Python → Rust mapping and what is / isn't ported.
 
@@ -12,6 +13,7 @@ resolver = "2"
 members = [
     "crates/fleischwolf-core",
     "crates/fleischwolf",
+    "crates/fleischwolf-asr",
     "crates/fleischwolf-cli",
     "crates/fleischwolf-pdf",
     # Node.js / Bun N-API bindings (published to npm, not crates.io).

MIGRATION.md

@@ -5,11 +5,11 @@ Rust. This document is the **current status**: what is migrated, how it compares
 to upstream docling, and what is intentionally not done yet. (The original
 phased plan is kept at the end as history.)
 
-> **Status: the format migration is essentially complete.** Every document
-> format in docling's pipeline except **audio/ASR** is supported, plus Markdown
-> (legacy + a Rust-only *strict* mode), docling-native **JSON** output, **image
-> extraction**, and **MHTML** (a fleischwolf-only extension docling doesn't
-> have). The declarative formats are pure-Rust and checked byte-for-byte
+> **Status: the format migration is complete.** Every document format in
+> docling's pipeline is supported — including **audio/ASR** (Whisper via ONNX,
+> in `fleischwolf-asr`) — plus Markdown (legacy + a Rust-only *strict* mode),
+> docling-native **JSON** output, **image extraction**, and **MHTML** (a
+> fleischwolf-only extension docling doesn't have). The declarative formats are pure-Rust and checked byte-for-byte
 > against *live* docling; the PDF/image/METS ML path lives in `fleischwolf-pdf`
 > (a pure-Rust PDF text parser + pdfium rasterization + ONNX
 > layout/TableFormer/OCR + a port of docling-parse's line sanitizer) and is also
@@ -30,13 +30,15 @@ Four layers, mirroring docling's:
 | **Converter** | `docling/document_converter.py` | `fleischwolf` — `converter.rs` (format dispatch + XML content sniffing) |
 | **Backends** | `docling/backend/*` | `fleischwolf` — `backend/*` (one per format) |
 | **PDF/ML pipeline** | `docling/pipeline/*`, `docling/models/*` | `fleischwolf-pdf` — pdfium + ONNX layout/OCR + assembly |
+| **Audio/ASR pipeline** | `docling/pipeline/asr_pipeline.py` | `fleischwolf-asr` — symphonia decode + log-mel + ONNX Whisper |
 | **CLI** | `docling/cli` | `fleischwolf-cli` |
 
 ```text
 crates/
 ├── fleischwolf-core/   # DoclingDocument, Node model, markdown.rs, json.rs, base64.rs, labels.rs
 ├── fleischwolf/        # DocumentConverter, source/format detection, backend/*.rs, ooxml.rs
 ├── fleischwolf-pdf/    # pdfium_backend, layout (RT-DETR/ONNX), ocr (PP-OCRv3/ONNX), assemble, mets
+├── fleischwolf-asr/    # audio decode (symphonia), mel.rs, whisper.rs (ONNX), tokenizer.rs
 ├── fleischwolf-cli/    # `--strict`, `--to md|json`, `--images placeholder|embedded|referenced`
 └── fleischwolf-node/   # Node.js/Bun N-API bindings (napi-rs), published to npm as `fleischwolf`
 ```
@@ -99,6 +101,7 @@ close — see `PDF_CONFORMANCE.md`. A deterministic snapshot baseline
 | PDF | **pure-Rust text parser** (`textparse.rs`, font-advance glyph boxes) + pdfium page render → RT-DETR layout (ONNX) → **TableFormer** table structure (ONNX) → PP-OCRv3 OCR for scanned pages → **docling-parse line sanitizer** (`dp_lines.rs`) + reading-order assembly |
 | Images (tiff/webp/png/jpeg) | the same pipeline, image as a single page |
 | METS / Google Books | `.tar.gz` of per-page hOCR + TIFF → cells from hOCR → the same layout+assembly path (no OCR needed) |
+| Audio (wav/mp3/flac/ogg/aac/m4a + mp4/mov audio tracks) | `fleischwolf-asr`: **symphonia** decode (no ffmpeg) → 16 kHz mono → ported log-mel front-end → **Whisper tiny** encoder/decoder (ONNX, greedy with OpenAI's timestamp rules — docling's ASR defaults) → `[time: start-end] text` paragraphs. AVI is the one container symphonia can't demux. |
 
 ---
 
@@ -203,11 +206,10 @@ deliberate scope boundary or a cosmetic, single-fixture polish gap.
 
 **Out of scope by design:**
 
-- **Audio / ASR.** docling's Whisper-based speech path. A separate ML boundary
-  like PDF; deferred by design.
 - **VLM pipelines** (SmolDocling / remote VLM) and **enrichment models** (picture
   classification, formula understanding, code understanding). Model-bound; out of
-  scope for the discriminative port.
+  scope for the discriminative port. (**Audio/ASR is now done** — see §2; the
+  only container gap is AVI, which symphonia cannot demux.)
 - **XML DocLang / DocTags** input backend — no `.dclg` sources in the corpus to
   verify against, and not in the requested scope.
 - **Older patent schemas.** USPTO covers the modern `v4x` XML only; the
@@ -293,9 +295,9 @@ The port followed roughly: **Phase 0** skeleton & API → **Phase 2** text/marku
 (Markdown, CSV, HTML, AsciiDoc, DeepSeek) → **Phase 3** Office & e-book (DOCX,
 PPTX, XLSX, EPUB, ODF) → **Phase 4** long tail (XML families, LaTeX, Email,
 WebVTT, JSON) → **Phase 5–6** the PDF/image ML pipeline (pdfium + ONNX layout/OCR
-+ geometric tables) → output formats (strict Markdown, JSON, image extraction).
-Audio/ASR (the old "Phase 7" tail) and PyO3 interop bindings remain the main
-unbuilt pieces.
++ geometric tables) → output formats (strict Markdown, JSON, image extraction) →
+**Phase 7** audio/ASR (symphonia + ONNX Whisper). PyO3 interop bindings remain
+the one unbuilt piece.
 
 ## Why "Fleischwolf"? 🦀
 

README.md

@@ -28,7 +28,7 @@ the full architecture, the Python → Rust mapping, and the phased plan.
 
 The public API works end to end across **Markdown, CSV, HTML, AsciiDoc, DOCX,
 PPTX, XLSX, EPUB, ODF, WebVTT, Email, MHTML, JATS, USPTO, XBRL, LaTeX, JSON,
-PDF, images and METS** — plus Markdown / docling-JSON output and image
+PDF, images, METS and audio** — plus Markdown / docling-JSON output and image
 extraction. MHTML is a fleischwolf-only extension (docling has no MHTML
 backend): saved-webpage `.mhtml`/`.mht` archives are parsed as a MIME message
 with [`mail-parser`](https://crates.io/crates/mail-parser) (which conforms to
@@ -39,8 +39,16 @@ lives in `fleischwolf-pdf`: a pure-Rust PDF text parser, pdfium for page
 rasterization, and an ONNX layout/TableFormer/OCR stack. TableFormer is ported
 to ONNX and run on every detected table region to recover its structure;
 geometric reconstruction from cell positions remains only as the fallback when
-the TableFormer graphs aren't present (see `PDF_CONFORMANCE.md`). Audio/ASR is
-the main format still on the roadmap (see `MIGRATION.md`).
+the TableFormer graphs aren't present (see `PDF_CONFORMANCE.md`).
+
+**Audio/ASR** (docling's Whisper pipeline) lives in `fleischwolf-asr`, and it is
+Rust all the way down: [`symphonia`](https://crates.io/crates/symphonia)
+demuxes/decodes the container in-process (wav, mp3, flac, ogg, aac, m4a — plus
+the audio track of mp4/mov; no ffmpeg), a ported log-mel front-end feeds a
+**Whisper tiny** encoder/decoder exported to ONNX (run on `ort`, greedy with
+OpenAI's timestamp rules — docling's ASR defaults), and each segment becomes a
+`[time: start-end] text` paragraph. `FLEISCHWOLF_ASR_LANG` picks the language
+(default `en`). AVI is the one container symphonia cannot demux.
 
 Output is checked against upstream Python docling — declarative formats
 byte-for-byte against live docling, the ML pipeline against a deterministic
@@ -271,10 +279,13 @@ curl -fsSL https://raw.githubusercontent.com/artiz/fleischwolf/master/scripts/do
 | RT-DETR layout | `models/layout_heron.onnx` |
 | PP-OCRv3 rec + dictionary | `models/ocr_rec.onnx`, `models/ppocr_keys_v1.txt` |
 | TableFormer (optional) | `models/tableformer/{encoder,decoder,bbox}.onnx` (+ `.data` sidecars where the export needs them) |
+| Whisper tiny (audio/ASR; skip with `--no-asr`) | `models/asr/{encoder_model,decoder_model}.onnx`, `models/asr/vocab.json` (+ `added_tokens.json` for language selection) |
 
 Idempotent — safe to re-run; it skips files already on disk. Pass `--force` to
 re-fetch everything, or set `$FLEISCHWOLF_MODELS_URL` to fetch from a
-different host (your own export, an internal mirror, …). pdfium is Linux x64
+different host (your own export, an internal mirror, …); the Whisper assets
+come from Hugging Face (`$FLEISCHWOLF_ASR_MODELS_URL` overrides, or point
+`DOCLING_ASR_{ENCODER,DECODER,VOCAB}` at explicit files). pdfium is Linux x64
 only for now — other platforms, or building the models from source, need
 [`scripts/pdf_setup.sh`](#testing) instead.
 
@@ -368,6 +379,10 @@ cargo run -p fleischwolf-cli -- --to json crates/fleischwolf/sample.html > out.j
 scripts/download_dependencies.sh
 cargo run -p fleischwolf-cli -- document.pdf
 
+# transcribe audio (wav/mp3/flac/ogg/aac/m4a, or an mp4/mov audio track) — the
+# Whisper models come from the same download script
+cargo run -p fleischwolf-cli -- recording.mp3
+
 # extract pictures (PDF/image inputs): embed as data URIs, or write ./artifacts/*.png
 cargo run -p fleischwolf-cli -- --images embedded   document.pdf
 cargo run -p fleischwolf-cli -- --images referenced document.pdf > out.md

crates/fleischwolf-asr/Cargo.toml

@@ -0,0 +1,22 @@
+[package]
+name = "fleischwolf-asr"
+description = "Audio/ASR pipeline for Fleischwolf (Whisper via ONNX; a Rust port of docling's ASR pipeline)."
+version.workspace = true
+edition.workspace = true
+license.workspace = true
+repository.workspace = true
+homepage.workspace = true
+readme.workspace = true
+keywords.workspace = true
+categories.workspace = true
+# ort needs a newer compiler than the std-only crates' MSRV (same as fleischwolf-pdf).
+rust-version = "1.85"
+
+[dependencies]
+fleischwolf-core = { path = "../fleischwolf-core", version = "0.11.0" }
+# Same ort as fleischwolf-pdf so the workspace keeps a single ONNX runtime.
+ort = { version = "2.0.0-rc.12", default-features = false, features = ["std", "download-binaries", "tls-rustls"] }
+serde_json = "1"
+# Pure-Rust audio demuxing/decoding for every container docling's ASR accepts:
+# wav (pcm), mp3, flac, ogg/vorbis, aac (adts), m4a/mp4 (isomp4+aac).
+symphonia = { version = "0.5", default-features = false, features = ["wav", "pcm", "mp3", "flac", "ogg", "vorbis", "aac", "isomp4"] }