[feat] Native Whisper: greedy decode default hangs on long-form audio - expose beam_size / condition_on_previous_text / temperature

[BBC-Esq]

Summary

docling's native (openai-whisper) ASR backend forwards almost none of whisper's decoding parameters to transcribe(). In particular beam_size, condition_on_previous_text, and temperature are not fields on InlineAsrNativeWhisperOptions and silently fall back to whisper's defaults. On long-form audio those defaults trigger whisper's well-known runaway-repetition failure mode — the model gets stuck re-decoding and transcription slows to roughly real-time or never finishes. Because docling exposes no override, there is no workaround through the docling API.

Root cause (isolated empirically)

I tested large-v3 on an 18.7-min clean, single-speaker TED talk (TED-LIUM, Elizabeth Gilbert — deliberately not "hard" audio), driving raw whisper.transcribe() directly while holding everything at docling's settings (auto-detected language, condition_on_previous_text=True, word_timestamps=False) and varying only beam_size:

beam_size	spirals (4 runs each)	completion time
None — greedy decode (whisper's default, i.e. what docling uses)	2 / 4 hung	~150 s (slow)
1 — beam search	0 / 4	~90–150 s (faster)

The dominant, isolated trigger is whisper's default greedy decoding (beam_size=None). Beam search (even width 1) terminates sequences differently, avoids the runaway repetition, and sidesteps the costly temperature-retry fallback — so it is both more reliable and faster. The failure is non-deterministic (whisper's temperature fallback samples without a fixed seed), which is why it doesn't reproduce on every run — but at the greedy default the rate is high enough to make the native backend unreliable for long-form.

condition_on_previous_text=False is whisper's documented mitigation for this repetition mode (it stops one bad window from seeding the next); I didn't isolate it independently here because beam_size >= 1 already prevented the spiral, but it's worth exposing alongside.

Additional data:

• Reproduces on hard audio too: native tiny on a ~19.8-min earnings22 clip couldn't finish a single pass in 17+ minutes (RTF ≈ 1×).
• For contrast, the WhisperS2T (CTranslate2) backend transcribes the same clips in seconds and never exhibits this, because it doesn't use this decode path. (Transcription quality not assessed here.)

What docling currently does

In docling/pipeline/asr_pipeline.py (native path):

result = self.model.transcribe(str(fpath), verbose=self.verbose, word_timestamps=self.word_timestamps)

Only verbose and word_timestamps are forwarded. beam_size, condition_on_previous_text, and temperature are not fields on InlineAsrNativeWhisperOptions, so they take whisper's defaults with no way to override.

Requested change

Expose whisper's decoding parameters on InlineAsrNativeWhisperOptions and forward them in the native transcribe() call — at minimum:

• beam_size — the most impactful; consider defaulting to 1+ rather than greedy.
• condition_on_previous_text
• temperature

(language would be a useful nice-to-have too — docling currently auto-detects on every call.)

Secondary

docling's native option also defaults word_timestamps=True, overriding whisper's own default of False and adding DTW word-alignment cost out of the box. Suggest exposing it but defaulting to whisper's False.

Discovered while working on #2938.

[sriharan0804]

Hi @BBC-Esq , I’d like to work on this.

My plan is to expose beam_size, condition_on_previous_text, temperature, and optionally language in InlineAsrNativeWhisperOptions, then forward them to the native Whisper transcribe() call.

I’ll keep the defaults backward-compatible unless maintainers prefer changing the default behavior.

[BBC-Esq]

Hey @sriharan0804 I did a quick PR before I saw your message even though I really didn't want to because I'm still trying to finalize number 2938 that's taking months...So feel free to use my PR as a basis or just redo it, and then I'll delete the branch in my docling fork to simplify things on my end. Either way, let me know what you decide!

[BBC-Esq]

BTW @sriharan0804 IBM has strict signing off on commits and PR stuff and formatting (e.g. "feat:") etc. that supposedly helps keep them organized...it's a pain but make sure you understand all of that too...

[sriharan0804]

Thanks @BBC-Esq, no problem at all.

I’ll take a look at your PR and use it as a basis if that is okay. I’ll make sure to follow Docling’s commit sign-off and PR formatting requirements before opening my PR.

Thanks for the heads-up!

[BBC-Esq]

Yep, just let me know if you do decide to take it on so I can delete mine...thanks.

[sriharan0804]

Thanks! I appreciate it.
I'll take this one on and implement it in my own branch, so you can go ahead and delete your PR/branch when it's convenient.
Thanks again for letting me know, and I appreciate the heads-up about the signing and formatting requirements.

[sriharan0804]

@BBC-Esq , I've updated the PR to address the requested native Whisper decoding options.

The current implementation:

• Exposes beam_size and condition_on_previous_text through InlineAsrNativeWhisperOptions.
• Forwards beam_size, condition_on_previous_text, temperature, and language to the native whisper.transcribe() call.
• Includes a unit test covering the forwarding of the decoding options.
• Was validated locally with a smoke test using Whisper on a sample audio file, and the existing formatting/lint checks pass.

Please let me know if you'd like me to make any additional changes or adjust the implementation.

[zeweihan]

Excellent writeup and empirical isolation — the beam_size=None greedy default being both the dominant hang trigger and slower than beam search is a clean finding, and the non-determinism (temperature fallback sampling without a fixed seed) explains exactly why this slips past smoke tests. Adding a regulated-document angle on why this lands harder than "ASR is slow," since long-form audio is a first-class input for legal/compliance corpora.

1. Long-form is the norm in legal audio, not the edge case — and it's much longer than 18 minutes. Depositions run 2–8 hours, court recordings and board/committee meetings run multi-hour, and recorded client calls / evidence voicemails make up a continuous ingestion stream. An 18.7-minute clean single-speaker TED talk hanging 2/4 runs is a red flag precisely because it's the easy case: real legal audio is hard (crosstalk, overlapping speakers, whispered testimony, courtroom acoustics), and ASR reliability degrades fast on exactly that input. "It hung on the easy clip" means the production reliability ceiling on the actual corpus is materially worse — and long-form is the corpus, not a stress test.

2. Non-deterministic hang is the dangerous failure class for evidence work, because the partial output looks authoritative. The worst part of a runaway-repetition stall isn't the wait — it's that whisper is emitting text while it spirals. If the run is killed mid-stall (timeout, OOM, operator gives up), you're left with a transcript that's syntactically clean and timestamped up to the stall point, with the tail silently missing. For a deposition transcript that's a material omission: the last 40 minutes of testimony — frequently where the key admission sits — is absent, and nothing in the artifact signals where transcription stopped. A reviewer comparing "transcript" to "audio length" might catch it; a reviewer running retrieval over the transcript never will. That's the same plausible-but-wrong shape as the export-layer asymmetries elsewhere here: values present, structurally incomplete, no degradation signal.

3. The non-determinism is itself an audit problem. In a legal/privileged-audio pipeline the operator often needs to assert "this transcript was produced from this recording under these settings" — a reproducibility requirement that adversarial review (and some evidence rules) effectively demand. A backend that "sometimes finishes, sometimes hangs, with a different tokenization each run because temperature-fallback samples without a seed" can't make that assertion. Exposing beam_size (which makes the decode path deterministic) fixes both the hang and the reproducibility property in one move — which is a strong argument for making a non-greedy default rather than just exposing the knob.

4. condition_on_previous_text is semantically load-bearing for legal speech, which is why exposing it (not just defaulting it off) matters. Whisper's documented mitigation is condition_on_previous_text=False, and turning it off does kill the repetition cascade. But for depositions the previous-window conditioning carries real meaning — "as I stated earlier", "to follow up on counsel's question", "the figure I gave you" — and disabling it globally trades the hang risk for a coherence/context-loss risk in exactly the corpora where cross-reference resolution matters. The right fix is exposing the flag so operators can tune per-corpus (off for long monologues, on for examination), rather than a single baked default.

One concrete suggestion alongside the param-exposure PR: when the native backend detects the runaway-repetition signature (rapidly increasing decode time per window / repeated token sequences), emit a decode_stall_detected annotation on the segment rather than only blocking — converts a silent partial into an informed one, and gives the operator a deterministic signal that "this transcript may be incomplete past segment N" instead of discovering the gap during review. That's the same pattern as surfacing unsupported-image / dropped-spans elsewhere in the project: the hang itself can be mitigated with beam_size, but the partial-looks-complete failure mode deserves its own surface even after the decode-path fix.

Happy to contribute a long-form multi-speaker fixture (or a deposition-style clip with cross-references) if that'd help validate the stall-detection path.

[sriharan0804]

@zeweihan Thanks for taking the time to write this up. Those are some really good points, especially around long-form legal audio and the risks of partial transcripts looking complete.

I also like the idea of surfacing something like a decode_stall_detected annotation. That feels like a useful follow-up enhancement, as it would make these kinds of failures much more visible to users instead of silently producing incomplete output.

And thanks for offering to contribute long-form test fixtures as well—that would definitely be valuable for validating improvements in this area.

[zeweihan]

Legal-domain perspective on why the non-determinism here is the part that matters most, beyond the perf story.

Long-form audio is the default legal shape, not an edge case. Depositions, court proceedings, arbitration hearings, and recorded witness interviews routinely run 1–8 hours of single-speaker or two-speaker audio. So "hangs on 18.7 min" maps directly onto "the second half of a 3-hour deposition silently never transcribes." That's an evidentiary-completeness gap, not a slow batch job.

Non-deterministic completion is the real hazard. The temperature-fallback retry samples without a fixed seed, so the same .wav produces a different transcript run-to-run. In an evidentiary context "the transcript" needs to be a reproducible artifact — opposing counsel re-running the same file should get the same words. A decoder whose output depends on which greedy window spiraled is a transcript you cannot re-verify, which quietly undermines the whole point of transcribing the record.

The runaway-repetition failure mode is silent corruption, not a crash. This is the part @BBC-Esq's beam_size isolation surfaces most clearly: when greedy gets stuck, it doesn't error — it emits high-confidence repeated tokens that look fluent. So a downstream reader (or RAG chunker) gets a plausible-looking passage that is hallucinated repetition, with no signal that decoding failed. For legal audio that maps onto a fabricated-looking stretch of testimony: the worst class of plausible-but-wrong.

condition_on_previous_text=True is actively dangerous for testimony. One misheard phrase seeds the next window, and the error propagates forward as if it were established context. A misheard objection or a wrong number becomes "established" for everything that follows. The False default isn't just the documented mitigation for repetition — it's the one that limits the blast radius of a single decoding error to one window instead of the rest of the file.

Concrete ask that would make transcripts auditable for regulated use: alongside exposing the params, emit a small decoding_manifest per run — which segments used greedy vs beam, whether repetition was detected, any segments that hit the temperature fallback. That turns "the decoder hung / drifted" from a thing you discover by eyeballing the transcript into a thing you can assert against. It also lets a reviewer weight low-confidence / repetition-flagged segments rather than treating the whole transcript as uniformly authoritative.