feat: opt-in coalescing of timed ASR calls

[Eoin-Houstoun]

Adds an opt-in gate that coalesces timed ASR calls. Off by default, so nothing
changes unless you ask for it.

--asr-coalesce-min-s   defer ASR until this much new audio has accrued (0 = off)

Why

transcription_processor() runs process_iter() once per arriving chunk, and
each call costs a full-length encoder pass regardless of how much new audio it
contains. When chunks are short, most of that work re-encodes audio the previous
pass already saw.

There is currently no way to trade update cadence for that work. --min-chunk-size
looks like the knob but never reaches this path when VAD/VAC is on, which is the
default:

# audio_processor.py
chunk_seconds = self.args.vac_chunk_size if self.args.vac else self.args.min_chunk_size

Measured rather than assumed: --min-chunk-size at 0.1 (default), 0.5 and 1.0
gives 40 ASR calls in all three cases.

Numbers

base + faster-whisper, CPU, 3 samples fed at real time, 3 reps per setting.
n_asr_calls counts process_iter() invocations and reproduces exactly across
reps, so it is the figure to trust here rather than the milliseconds.

--asr-coalesce-min-s	ASR calls	compute (ms)	WER	first word (ms)	mean gap (ms)
0 (default, off)	40	11484	0.0833	1167	466
0.50	37	11339	0.0833	1167	466
0.75	20	6330	0.0667	1500	851
1.00	19	6730	0.0833	1667	872
1.50	13	4806	0.0667	2167	1111
2.00	10	3994	0.0667	2532	1529

Three things you would want to know before merging:

• The response is a step, not a curve. VAC emits chunks near 0.5 s on this
  audio, so 0.5 barely fires while 0.75 coalesces nearly every pair. The useful
  value is tied to a deployment's chunk cadence, which is why this ships with no
  tuned default.
• WER is unchanged within this fixture's resolution. It reads 0.0833 /
  0.0833 / 0.0667 / 0.0833 / 0.0667 across those rows. That oscillation is noise
  on a small word count, and I am not claiming an accuracy effect either way.
  Only 3 samples, so treat the magnitudes as indicative, not general.
• Draining at boundaries costs 2 to 3 passes per setting versus deferring
  through them, which is the difference between these numbers and an earlier
  revision of this PR. That is the price of not losing audio, and it is worth it.

Review notes

Three source files plus tests and a docs/troubleshooting.md entry.

Deferred audio is drained with a real counted process_iter() before every
boundary handler (Silence, ChangeSpeaker, SENTINEL). This matters on
LocalAgreement, whose finish() runs no inference and whose HypothesisBuffer
needs two agreeing passes to commit: without the drain, a deferred tail is either
never transcribed or discarded by the init() that follows a boundary. The drain
supplies the first pass so the handler's own pass is the second, matching the
behaviour when every chunk is inferred on arrival. Tokens the SENTINEL drain
commits are passed to _finish_transcription() explicitly, since finish() only
reports the hypothesis tail.

The threshold logic is two pure functions, resolve_coalesce_min_s and
should_defer_inference, testable without loading a model, following
resolve_retention_seconds in test_retention.py. Deferral is bounded by the
threshold plus one chunk.

Tests: tests/test_asr_coalescing.py covers the threshold logic including that
the shipped default resolves to disabled, and tests/test_asr_coalescing_pipeline.py
drives the real pipeline for the two boundary cases. Both pipeline tests fail on
the pre-drain code with a diverged transcript tail. They feed at speed=1.0 and
cut mid-speech deliberately: at speed=0 the whole file arrives as one chunk,
nothing is deferred, and they pass vacuously.

Full suite 117 passed, ruff check . clean, rebased on v0.2.25.

Provenance

A machine was in the loop. This began as output from an automated optimisation
run (Artemis Discovery, the tool I work on at TurinTech) pointed at the streaming
policy, with ASR compute as the objective and the existing WER gate as the
constraint.

What it produced is not what is here. I reproduced every number against a clean
checkout and in doing so cut half its patch (a dedup at silence and speaker
boundaries that measured 40 calls, i.e. contributed nothing), dropped its
headline accuracy claim as noise, flipped the default from on to off, and
rewrote the logic to be testable. Worth naming the failure modes: it defaulted a
behaviour change to on, and reported an accuracy win a wider sweep does not
support. The boundary data-loss bugs found in review were in its output too, and
survived my own review of it.

CONTRIBUTING asks for significant changes to be discussed first, so happy to move
this to an issue if you would rather start there, or to reshape the flag.

[QuentinFuxa]

Thanks for measuring before proposing, and the premise checks out on my side too: with VAC on (the default), pacing comes from vac_chunk_size at audio_processor.py:79 and --min-chunk-size never throttles process_iter (its only other consumer, AlignAttConfig.segment_length, is never read). The gate being opt-in and the pure-function tests following the test_retention.py precedent are both right. But the deferral is not safe for the LocalAgreement policy, and both problems are silent data loss, so this needs changes before merge.

Blockers:

1. Trailing deferred audio is never transcribed at end of stream. The final chunks accrue less than min_s, get inserted via insert_audio_chunk, and skip process_iter; then SENTINEL arrives and _finish_transcription() runs with no deferral flush. OnlineASRProcessor.finish() (local_agreement/online_asr.py:399-408) runs no inference at all: it returns transcript_buffer.buffer, which by construction excludes the deferred audio, and the buffer_transcription fallback in _finish_transcription reads the same stale text. This fires on nearly every ended stream, because _flush_remaining_pcm (audio_processor.py:984-1000) always enqueues a final sub-second chunk right before SENTINEL, and with min_s at 0.75-2.0 that chunk is deferred. So with localagreement (any backend, including REST /v1/audio/transcriptions), the last words of every session are silently dropped. The other processors are safe because their finish/start_silence run real inference over pending audio; LocalAgreement's do not.

2. Words spoken just before a long silence can no longer commit. The reset at Silence.is_starting assumes "boundary flushes; nothing is stranded", but for LocalAgreement that is only half true: start_silence (online_asr.py:161-164) runs process_iter once over the deferred audio, and HypothesisBuffer.flush (online_asr.py:59-86) only commits a token seen in two consecutive agreeing passes (confidence_validation is off by default). Deferred audio gets its first-ever pass inside start_silence, so its tokens stay uncommitted, and when the silence ends at >= 5 s, end_silence calls init(offset=...) (online_asr.py:166-177), which discards them permanently. On main the tail is processed on arrival and again at start_silence, so the two-pass agreement commits it; with the gate enabled that second pass no longer exists.

Both have the same fix, and it keeps your design: when deferred_audio_s > 0, drain it with a real (counted) process_iter before every boundary handler (Silence, ChangeSpeaker, SENTINEL) instead of just zeroing the counter. That restores main's behavior exactly at boundaries and only coalesces in steady state, which is where your savings come from anyway.

Asks:

3. --asr-coalesce-max-s is currently inert: resolve_coalesce_window enforces max > min, so the prospective < max_s conjunct in should_defer_inference is unreachable, and deferral already terminates once prospective >= min_s, so held-back audio is inherently bounded by min_s plus one chunk. As shipped it is only an obscure on/off validator, with a trap: the default max=1.0 silently disables any --asr-coalesce-min-s >= 1.0, which includes three of the six rows in your own benchmark table. Either drop the flag (my preference: one knob, and the time-to-first-word bound genuinely is min_s) or give it real semantics, and in any case fail or warn loudly when the pair resolves to disabled.

4. Please add one TestHarness test of the actual deferral path (the helpers tests are fine but cannot catch either blocker): localagreement + coalescing on, feed real audio that ends mid-speech, assert the final words are present after finish(); ideally a second one with a > 5 s silence asserting the pre-silence tail commits. Either would have caught both blockers.

5. Scope the docs to the whisper-family backends: the qwen3 processors already coalesce internally (_MIN_NEW_SECONDS = 1.0 in the vllm processors, pending_sec/due_after in qwen3-streaming), so the flag does approximately nothing there. Worth one sentence in docs/troubleshooting.md so people do not tune it against the wrong backend. For the longer term I keep in mind that a processor-level gate honoring is_last cannot strand audio by construction, but I am fine landing this at the pipeline level once the boundaries drain.

Nits: fetch get_buffer() outside the lock and assign under it, matching the timeout branch at audio_processor.py:508-513; and the "boundary flushes; nothing is stranded" comment needs to go or be corrected either way.

[Eoin-Houstoun]

Thanks for the review, and for the line numbers. They meant I could go straight to both of these instead of hunting for them.

Both blockers reproduce. I instrumented the deferral state rather than reasoning about it, and the end-of-stream one shows up immediately: cut the feed mid-speech and deferred_audio_s=0.200 is still outstanding when SENTINEL arrives, and nothing ever infers it. Fixed.

Boundaries now drain. Deferred audio gets a real counted process_iter() before each of Silence, ChangeSpeaker and SENTINEL, instead of the counter just being zeroed. That hands LocalAgreement the first of its two agreeing passes, so the handler's own pass is the second, matching main where every chunk is inferred on arrival.

Two things I ran into:

• ChangeSpeaker used to continue past the token emission block. new_speaker() calls init() and wipes the hypothesis buffer, so anything the drain committed has to get out first. That branch now falls through when the drain produced tokens and continues otherwise, so nothing changes when there's nothing to emit.
• finish() only reports the hypothesis tail, so whatever the SENTINEL drain commits can't come back through it. _finish_transcription() takes those as pending_tokens and prepends them.

--asr-coalesce-max-s is gone. The conjunct was unreachable, and the default quietly disabled three rows of my own benchmark table, which I should have caught before posting it. One knob now. The bound is min_s plus a chunk and the docs say so. A negative value warns and disables rather than failing silently.

Tests. Added, and they do fail on the pre-fix code with a diverged transcript tail.

They nearly didn't, though. My first attempt fed at speed=0 and passed against the broken code, because at speed=0 the whole file arrives as a single chunk (n_transcription_calls=1), nothing is ever deferred, and the test asserts nothing at all. They now feed at speed=1.0 and cut mid-speech at a point that leaves a sub-threshold remainder outstanding. Both constraints are in the module docstring, since either one quietly makes the tests meaningless again.

Docs, though vaguer than you asked for. I couldn't find _MIN_NEW_SECONDS in either vllm processor on current main, and qwen3_streaming/online.py looks like a re-export shim over qwen3_asr_causal, which isn't vendored here. Point me at where those live and I'll name them properly. For now it says the flag is for the whisper-family backends, and that anything already batching internally won't gain much from it.

Nits done: get_buffer() is fetched outside the lock and assigned under it, matching the timeout branch, and the "boundary flushes" comment is gone, since it was simply wrong. While I was in there I also dropped the buffer re-publish from the deferral path. get_buffer() can't change while no inference is running, and re-publishing it raw would have undone the committed-prefix strip.

Rebased on main (v0.2.25). Full suite 117 passed, ruff clean.

On the processor-level gate honouring is_last: agreed it's the better shape, and unlike this one it can't strand audio by construction. Happy to do that as a follow-up if you'd rather it ended up there.