Add optional language-aware VAD segmentation for multilingual inference - #1468
Open
Frenzie wants to merge 4 commits into
Open
Add optional language-aware VAD segmentation for multilingual inference#1468Frenzie wants to merge 4 commits into
Frenzie wants to merge 4 commits into
Conversation
This is SYSTRAN#1274 updated to current master (minus the README formatting changes).
Batched multilingual inference assigns one language token per batch window. A window that spans a language switch produces one language's transcription over the other language's audio, which the model might truncate, resulting in "missing" audio. The non-batched seek loop hides this; batched mode does not. Add an opt-in `language_aware_vad_segments` flag (default False) to both WhisperModel.transcribe and BatchedInferencePipeline.transcribe. When enabled with multilingual=True and vad_filter=True, VAD speech chunks are split at detected language change points before decoding: - WhisperModel._split_vad_chunks_by_language runs a cheap two-point screen (first/last 3 s) per chunk to flag mixed chunks, then a 3 s sliding refine (1.5 s hop) only on those, batch-encoded. - BatchedInferencePipeline._group_chunks_by_language groups same-language sub-chunks into <= chunk_length windows in original-audio coordinates (no packing, so no onset stripping); unconfident sub-chunks fold into a neighbour rather than fragmenting. - On the non-batched path the split regions are routed as clip_timestamps seek clips so the seek loop processes each separately.
Frenzie
force-pushed
the
multilingual-language-aware-vad-segments
branch
from
July 13, 2026 09:38
1e9803f to
55d53e7
Compare
Frenzie
force-pushed
the
multilingual-language-aware-vad-segments
branch
from
July 17, 2026 11:10
611b9b2 to
979c8d0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes on top of #1466 to prevent conflicts, but it's otherwise unrelated.
The motivation for this PR is that multilingual mode regularly drops up to 20 seconds of transcription in situations where the language switches, especially in batch mode. In regular mode the problem seems to be less severe. I believe it might be due to some kind of truncation by the model when it's operating in the wrong language.
Improving the accuracy of the detected language in the multilingual output is a nice side effect, at a cost of merely being about three times slower. But I want to emphasize that the main concern is transcriptions that are completely missing.