Skip to content

feat: add fast ASR backend #2938

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cau-git merged 28 commits into from
Jul 1, 2026
Merged

feat: add fast ASR backend #2938

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
d1fe22d
feat: add WhisperS2T-reborn ASR backend
BBC-Esq Mar 26, 2026
0eafbd4
feat: add CLI support for WhisperS2T model variants
BBC-Esq Mar 26, 2026
5123259
test: add WhisperS2T backend tests and rename test file
BBC-Esq Mar 26, 2026
3ce53c6
chore: add whisper-s2t-reborn dependency to ASR extras
BBC-Esq Mar 26, 2026
66b3b2a
fix: add missing CLI mapping for WHISPER_DISTIL_LARGE_V3_5_S2T
BBC-Esq Apr 17, 2026
36ae11f
feat: validate WhisperS2T model capabilities at config time
BBC-Esq Apr 17, 2026
f32d989
fix(asr): route WhisperS2T run() through _process_conversation()
BBC-Esq May 23, 2026
68010ce
fix(asr): coerce WhisperS2T compute_type to float32 on CPU
BBC-Esq May 23, 2026
0044683
test: pin WhisperS2T CPU compute_type coercion and zero-duration hand…
BBC-Esq May 23, 2026
38736e4
ci: exempt tests/test_asr.py from max-lines lint (warn list)
BBC-Esq May 23, 2026
ea6ec92
refactor(tests): split tests/test_asr.py into per-backend files
BBC-Esq May 23, 2026
8b2df79
style: drop trailing blank line in test_asr_mlx_whisper.py
BBC-Esq May 23, 2026
84ec372
test(asr): use CPU device in WhisperS2T init test for upstream device…
BBC-Esq Jun 20, 2026
fe5167f
fix(asr): exclude whisper-s2t-reborn on Apple Silicon in format-audio…
BBC-Esq Jun 20, 2026
971dde2
chore: require whisper-s2t-reborn>=1.6.2
BBC-Esq Jun 21, 2026
461e997
Merge remote-tracking branch 'upstream/main' into add-fast-asr-backend
BBC-Esq Jun 24, 2026
245ce7a
refactor(asr): rename WhisperS2T compute_type option to torch_dtype
BBC-Esq Jun 25, 2026
aff52ae
refactor(asr): rename WhisperS2T cpu_threads option to num_threads
BBC-Esq Jun 25, 2026
d632064
refactor(asr): drop unused WhisperS2T num_workers option
BBC-Esq Jun 25, 2026
6c1394b
chore(deps): add <2 upper bound to whisper-s2t-reborn
BBC-Esq Jun 25, 2026
3dc2823
chore(ci): restore .github/max-lines-ignore to match upstream
BBC-Esq Jun 25, 2026
ca6aee5
refactor(asr): keep PR diff scoped to ASR-only changes
BBC-Esq Jun 25, 2026
ef212e6
refactor(asr): keep WhisperS2T opt-in, not the auto-selected default
BBC-Esq Jun 25, 2026
49eb9a6
docs(asr): add module docstring for pipeline_options_asr_model
BBC-Esq Jun 25, 2026
06f28c3
docs(asr): note LD_LIBRARY_PATH setup for WhisperS2T on Linux/CUDA
BBC-Esq Jun 25, 2026
0afcac9
docs(asr): reference only cuBLAS in the WhisperS2T LD_LIBRARY_PATH note
BBC-Esq Jun 25, 2026
4613c5c
chore(deps): require whisper-s2t-reborn>=1.6.3 for cuDNN-free ctransl…
BBC-Esq Jun 25, 2026
ef4cb80
Merge remote-tracking branch 'upstream/main' into add-fast-asr-backend
BBC-Esq Jun 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 44 additions & 0 deletions docling/cli/main.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,20 +50,34 @@
from docling.datamodel.accelerator_options import AcceleratorDevice, AcceleratorOptions
from docling.datamodel.asr_model_specs import (
WHISPER_BASE,
WHISPER_BASE_EN_S2T,
WHISPER_BASE_MLX,
WHISPER_BASE_NATIVE,
WHISPER_BASE_S2T,
WHISPER_DISTIL_LARGE_V3_5_S2T,
WHISPER_DISTIL_LARGE_V3_S2T,
WHISPER_DISTIL_MEDIUM_EN_S2T,
WHISPER_DISTIL_SMALL_EN_S2T,
WHISPER_LARGE,
WHISPER_LARGE_MLX,
WHISPER_LARGE_NATIVE,
WHISPER_LARGE_V3_S2T,
WHISPER_LARGE_V3_TURBO_S2T,
WHISPER_MEDIUM,
WHISPER_MEDIUM_EN_S2T,
WHISPER_MEDIUM_MLX,
WHISPER_MEDIUM_NATIVE,
WHISPER_MEDIUM_S2T,
WHISPER_SMALL,
WHISPER_SMALL_EN_S2T,
WHISPER_SMALL_MLX,
WHISPER_SMALL_NATIVE,
WHISPER_SMALL_S2T,
WHISPER_TINY,
WHISPER_TINY_EN_S2T,
WHISPER_TINY_MLX,
WHISPER_TINY_NATIVE,
WHISPER_TINY_S2T,
WHISPER_TURBO,
WHISPER_TURBO_MLX,
WHISPER_TURBO_NATIVE,
Expand Down Expand Up @@ -1156,6 +1170,36 @@ def convert( # noqa: C901
elif asr_model == AsrModelType.WHISPER_TURBO_NATIVE:
asr_pipeline_options.asr_options = WHISPER_TURBO_NATIVE

# Explicit WhisperS2T models (CTranslate2 backend - fastest)
elif asr_model == AsrModelType.WHISPER_TINY_S2T:
asr_pipeline_options.asr_options = WHISPER_TINY_S2T
elif asr_model == AsrModelType.WHISPER_TINY_EN_S2T:
asr_pipeline_options.asr_options = WHISPER_TINY_EN_S2T
elif asr_model == AsrModelType.WHISPER_BASE_S2T:
asr_pipeline_options.asr_options = WHISPER_BASE_S2T
elif asr_model == AsrModelType.WHISPER_BASE_EN_S2T:
asr_pipeline_options.asr_options = WHISPER_BASE_EN_S2T
elif asr_model == AsrModelType.WHISPER_SMALL_S2T:
asr_pipeline_options.asr_options = WHISPER_SMALL_S2T
elif asr_model == AsrModelType.WHISPER_SMALL_EN_S2T:
asr_pipeline_options.asr_options = WHISPER_SMALL_EN_S2T
elif asr_model == AsrModelType.WHISPER_DISTIL_SMALL_EN_S2T:
asr_pipeline_options.asr_options = WHISPER_DISTIL_SMALL_EN_S2T
elif asr_model == AsrModelType.WHISPER_MEDIUM_S2T:
asr_pipeline_options.asr_options = WHISPER_MEDIUM_S2T
elif asr_model == AsrModelType.WHISPER_MEDIUM_EN_S2T:
asr_pipeline_options.asr_options = WHISPER_MEDIUM_EN_S2T
elif asr_model == AsrModelType.WHISPER_DISTIL_MEDIUM_EN_S2T:
asr_pipeline_options.asr_options = WHISPER_DISTIL_MEDIUM_EN_S2T
elif asr_model == AsrModelType.WHISPER_LARGE_V3_S2T:
asr_pipeline_options.asr_options = WHISPER_LARGE_V3_S2T
elif asr_model == AsrModelType.WHISPER_DISTIL_LARGE_V3_S2T:
asr_pipeline_options.asr_options = WHISPER_DISTIL_LARGE_V3_S2T
elif asr_model == AsrModelType.WHISPER_LARGE_V3_TURBO_S2T:
asr_pipeline_options.asr_options = WHISPER_LARGE_V3_TURBO_S2T
elif asr_model == AsrModelType.WHISPER_DISTIL_LARGE_V3_5_S2T:
asr_pipeline_options.asr_options = WHISPER_DISTIL_LARGE_V3_5_S2T

else:
_log.error(f"{asr_model} is not known")
raise ValueError(f"{asr_model} is not known")
Expand Down
Loading
Loading