Complete the migration: nested-table flattening (HTML 31/32) + Audio/ASR (Whisper via ONNX) - #24
Merged
Merged
Conversation
The last HTML content gap, previously believed to need rendered bounding boxes. It doesn't: docling's odd spacing in deeply nested-table cells (`I II III IV V E1 …`) is pure text processing — - BeautifulSoup collapses each whitespace-only text node to a single newline (when it spans a source line break) or a single space; - docling's `get_text` keeps every other text node verbatim and appends one trailing space per `p`/`li`/`th`/`td`; - the markdown serializer flattens a nested table to its own grid cells joined with single spaces (`_collect_subtree_text`), so a still-deeper table's `\n` runs survive into the cell text; - table cells finally render `\n` → space, yielding exactly `1 + #source-line-breaks` spaces at each seam. Reproduce that in Rust: new `subtree_text` (docling `get_text` + BeautifulSoup whitespace semantics + `_clean_unicode` replacements, ASCII whitespace only since bs4 leaves ` ` alone), `parse_table` refactored to take the cell-text function, and the nested-in-cell table arm now flattens one grid level with raw subtree text below. Spanning cells repeat into every grid slot they cover, as docling's grid walk does. table_06's data row is now byte-identical to docling's groundtruth; the offline sweep puts HTML at 31/32 content-exact, the only remainder being wiki_duck's external-stylesheet menus (needs CSS fetchable at render time — inherently not an offline behaviour). MIGRATION §4/§5 rewritten: the browser-render subsystem is closed out, and §5 now separates deliberate scope boundaries from cosmetic per-fixture polish gaps.
…Phase 7)
Ports docling's AsrPipeline (native-Whisper path, its ASR defaults:
whisper tiny, temperature-0 greedy, timestamps) as a new fleischwolf-asr
crate, Rust all the way down:
- symphonia demuxes/decodes the audio in-process — wav, mp3, flac, ogg,
aac, m4a, plus the audio track of mp4/mov; no ffmpeg. AVI is the one
container symphonia cannot demux (clear error).
- A ported log-mel front-end (80 Slaney mel filters, centered STFT,
Whisper's exact log/clamp/rescale), unit-tested against librosa
reference values.
- Whisper tiny encoder/decoder ONNX (onnx-community export) on ort —
the same runtime as layout/TableFormer/OCR. Greedy decoding implements
OpenAI's rules: special-token + non-speech-symbol suppression (the
default suppress_tokens="-1" set, derived by decode-scanning the
vocabulary), suppress_blank, forced initial timestamp with
max_initial_timestamp, timestamp pairing/monotonicity, the
timestamp-probability-mass rule, and the no-speech gate
(p_nospeech > 0.6 && avg_logprob < -1). 30-second windowing advances
seek to the last emitted timestamp.
- GPT-2 byte-level BPE decoding from vocab.json alone (no merges needed
for id→text). FLEISCHWOLF_ASR_LANG selects the language via
added_tokens.json (default en).
- Each segment renders as a `[time: start-end] text` paragraph (docling's
conversation form, Python float formatting).
Wiring: InputFormat::Audio now converts (the rejects-unimplemented test
moves to XmlDoclang, the one remaining backend-less format);
download_dependencies.sh fetches Whisper tiny from Hugging Face
(--no-asr to skip; DOCLING_ASR_{ENCODER,DECODER,VOCAB} override).
Verified end-to-end on tests/data/audio: all 12 audio fixtures + the
mp4/mov video fixtures transcribe the LibriVox sample correctly
("Shakespeare on scenery by Oscar Wilde…", [time: 0.0-10.0]); silence
yields whisper-tiny's documented model-faithful behavior. 8 new unit
tests (mel filterbank vs librosa, silence floor, timestamp rules, byte
decoder, resampler, wav decode, time formatting).
Docs: README (status, models table, try-it), MIGRATION (status:
migration complete; crate map; ML table; Phase 7 done), COMPARING (HTML
31/32 + refreshed gap table from the earlier commits).
artiz
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two features that close out MIGRATION's remaining format work, plus the doc refresh:
1. Docling-exact nested-table cell flattening (HTML → 31/32)
The last HTML content gap (
table_06) turned out not to need rendered bounding boxes. Reverse-engineered against docling's source: the odd spacing (I II III IV V E1 …) is BeautifulSoup whitespace semantics (whitespace-only text node →\n/' ') + docling'sget_texttrailing spaces +_collect_subtree_textgrid-join +\n→space at render. Ported byte-exact (newsubtree_text+flatten_nested_table;parse_tablerefactored to take the cell-text fn). Data row byte-identical to groundtruth; offline sweep 31/32 content-exact (onlywiki_duck's offline external CSS remains).2. Audio/ASR — docling's
AsrPipeline, pure Rust (newfleischwolf-asrcrate)ort(same runtime as layout/TableFormer/OCR), greedy with OpenAI's full rule set: non-speech-symbol suppression (derived by decode-scanning the vocab),suppress_blank, forced initial timestamp, pairing/monotonicity, timestamp-probability-mass, the no-speech gate, and 30-s windowing with timestamp-based seek.vocab.json;FLEISCHWOLF_ASR_LANGfor language.[time: start-end] textparagraphs (docling's conversation form).InputFormat::Audiowired into the converter;download_dependencies.shfetches Whisper tiny from HF (--no-asrto skip;DOCLING_ASR_*overrides).Verified end-to-end: all 12 audio fixtures + mp4/mov transcribe the LibriVox sample correctly (
[time: 0.0-10.0] Shakespeare on scenery by Oscar Wilde…) across every container; silence behavior is model-faithful. 16 new unit tests across both features.3. Docs
Testing
cargo fmt --check,cargo clippy --workspace --all-targets -- -D warnings(default +web-browserfeature), fullcargo test --workspace(11/11 suites incl. the all-format regression harness) all pass. ASR model files are runtime-optional with a clear error when absent — CI needs no models.🤖 Generated with Claude Code