feat: add fast ASR backend

[BBC-Esq]

Add Fast WhisperS2T-Reborn (based on ctranslate2) ASR Backend

This PR adds support for the whisper-s2t-reborn library as a high-performance ASR backend option, providing significantly faster transcription speeds. Within the "pipeline," it automatically chooses this backend if the criteria are met - e.g. cuda, etc. Previously, it was eithe MLX with a simple fallback to vanilla openai whisper. Vanilla whisper is now the final fallback if neither MLX nor WhisperS2T are available.

New Files/Classes:

• InlineAsrWhisperS2TOptions - Configuration class for WhisperS2T with options for compute type, batch size, beam size, and more
• _WhisperS2TModel - Model wrapper class implementing the WhisperS2T transcription pipeline

Modified Files:

• pipeline_options_asr_model.py - Added WHISPER_S2T to InferenceAsrFramework enum; Added InlineAsrWhisperS2TOptions configuration class
• asr_model_specs.py - Added 12 pre-configured model specs (tiny, base, small, medium, large-v3 + distilled variants); Extended AsrModelType enum
• asr_pipeline.py - Added _WhisperS2TModel class with device parsing fix for CTranslate2 compatibility; Updated AsrPipeline to handle new backend
• cli/main.py - Added CLI support for all WhisperS2T model variants

Notes

• Includes _parse_device() helper to handle device strings like "cuda:0" → ("cuda", 0) for CTranslate2 compatibility
• Supported devices: CPU, CUDA (MPS not supported by CTranslate2)

  Ctranslate2 is currently working on RocM support as well...

I will give you the script I used to test this below. However, PLEASE note that I always use a "set_cuda_paths" that relies on pip-installed versions of CUDA/cuDNN. This function sets the relevant cuda-related paths to the pip-installed locations so I don't have to install/reinstall various versions of CUDA. If you want to test with the stereotypical systemwide installation don't use "set_cuda_paths."

That said, to run the test script: (1) create a venv, (2) activate it, (3) run the following command, and (4) run the script pasted below:

pip install whisper-s2t-reborn nvidia-cuda-runtime-cu12==12.8.90 nvidia-cublas-cu12==12.8.4.1 nvidia-cudnn-cu12==9.10.2.21 https://download.pytorch.org/whl/cu128/torch-2.9.1%2Bcu128-cp312-cp312-win_amd64.whl

HERE IS THE SCRIPT:

import os
import sys
import platform
import json
import traceback
import time
from pathlib import Path
from typing import Optional
from dataclasses import dataclass, field
from enum import Enum, auto

def set_cuda_paths():
    if platform.system() != "Windows":
        return
    venv_base = Path(sys.executable).parent.parent
    nvidia_base = venv_base / 'Lib' / 'site-packages' / 'nvidia'
    if not nvidia_base.exists():
        return
    cuda_path_runtime = nvidia_base / 'cuda_runtime' / 'bin'
    cuda_path_runtime_lib = nvidia_base / 'cuda_runtime' / 'lib' / 'x64'
    cuda_path_runtime_include = nvidia_base / 'cuda_runtime' / 'include'
    cublas_path = nvidia_base / 'cublas' / 'bin'
    cudnn_path = nvidia_base / 'cudnn' / 'bin'
    nvrtc_path = nvidia_base / 'cuda_nvrtc' / 'bin'
    nvcc_path = nvidia_base / 'cuda_nvcc' / 'bin'
    paths_to_add = [
        cuda_path_runtime,
        cuda_path_runtime_lib,
        cuda_path_runtime_include,
        cublas_path,
        cudnn_path,
        nvrtc_path,
        nvcc_path,
    ]
    current_value = os.environ.get('PATH', '')
    new_value = os.pathsep.join([str(p) for p in paths_to_add] + ([current_value] if current_value else []))
    os.environ['PATH'] = new_value
    triton_cuda_path = nvidia_base / 'cuda_runtime'
    current_cuda_path = os.environ.get('CUDA_PATH', '')
    new_cuda_path = os.pathsep.join([str(triton_cuda_path)] + ([current_cuda_path] if current_cuda_path else []))
    os.environ['CUDA_PATH'] = new_cuda_path
    if hasattr(os, 'add_dll_directory'):
        for path in paths_to_add:
            if path.exists():
                try:
                    os.add_dll_directory(str(path))
                except OSError:
                    pass

set_cuda_paths()

from PySide6.QtCore import Qt, QThread, Signal, QSize, QUrl
from PySide6.QtWidgets import (
    QApplication, QMainWindow, QWidget, QVBoxLayout, QHBoxLayout,
    QPushButton, QLabel, QFileDialog, QTextEdit, QTabWidget,
    QTreeWidget, QTreeWidgetItem, QSplitter, QProgressBar,
    QMessageBox, QStatusBar, QFrame, QGroupBox, QComboBox,
    QScrollArea, QSizePolicy, QSpinBox
)
from PySide6.QtGui import QFont, QColor, QPalette, QIcon

HAS_WEBENGINE = False
try:
    from PySide6.QtWebEngineWidgets import QWebEngineView
    HAS_WEBENGINE = True
except ImportError:
    pass


class ExportFormat(Enum):
    MARKDOWN = auto()
    TEXT = auto()
    HTML = auto()
    JSON = auto()
    SRT = auto()
    VTT = auto()


@dataclass
class TranscriptionSegment:
    start_time: float
    end_time: float
    text: str
    words: list = field(default_factory=list)


@dataclass
class TranscriptionResult:
    success: bool
    filename: str
    markdown: str = ""
    text: str = ""
    html: str = ""
    json_data: str = ""
    srt: str = ""
    vtt: str = ""
    segments: list = None
    processing_time: float = 0.0
    model_name: str = ""
    device: str = ""
    error_message: str = ""

    def __post_init__(self):
        if self.segments is None:
            self.segments = []


class TranscriptionWorker(QThread):
    finished = Signal(TranscriptionResult)
    progress = Signal(str)

    def __init__(self, file_path: str, model_name: str, device: str, batch_size: int):
        super().__init__()
        self.file_path = file_path
        self.model_name = model_name
        self.device = device
        self.batch_size = batch_size

    def run(self):
        try:
            self.progress.emit("Starting transcription...")
            result = self._transcribe_audio()
            self.finished.emit(result)
        except Exception as e:
            error_result = TranscriptionResult(
                success=False,
                filename=Path(self.file_path).name,
                error_message=f"{str(e)}\n\n{traceback.format_exc()}"
            )
            self.finished.emit(error_result)

    def _transcribe_audio(self) -> TranscriptionResult:
        file_path = Path(self.file_path)

        self.progress.emit("Importing Docling...")

        from docling.document_converter import DocumentConverter, AudioFormatOption
        from docling.datamodel.base_models import InputFormat
        from docling.datamodel.pipeline_options import AsrPipelineOptions
        from docling.datamodel.accelerator_options import AcceleratorOptions
        from docling.datamodel import asr_model_specs
        from docling.pipeline.asr_pipeline import AsrPipeline

        model_map = {
            "tiny": asr_model_specs.WHISPER_TINY_S2T,
            "tiny.en": asr_model_specs.WHISPER_TINY_EN_S2T,
            "base": asr_model_specs.WHISPER_BASE_S2T,
            "base.en": asr_model_specs.WHISPER_BASE_EN_S2T,
            "small": asr_model_specs.WHISPER_SMALL_S2T,
            "small.en": asr_model_specs.WHISPER_SMALL_EN_S2T,
            "distil-small.en": asr_model_specs.WHISPER_DISTIL_SMALL_EN_S2T,
            "medium": asr_model_specs.WHISPER_MEDIUM_S2T,
            "medium.en": asr_model_specs.WHISPER_MEDIUM_EN_S2T,
            "distil-medium.en": asr_model_specs.WHISPER_DISTIL_MEDIUM_EN_S2T,
            "large-v3": asr_model_specs.WHISPER_LARGE_V3_S2T,
            "distil-large-v3": asr_model_specs.WHISPER_DISTIL_LARGE_V3_S2T,
        }

        self.progress.emit(f"Loading model: {self.model_name}...")

        model_spec = model_map.get(self.model_name, asr_model_specs.WHISPER_LARGE_V3_S2T)
        model_spec = model_spec.model_copy(update={"batch_size": self.batch_size})

        device = self.device.split(":")[0] if ":" in self.device else self.device

        pipeline_options = AsrPipelineOptions(
            accelerator_options=AcceleratorOptions(device=device),
            asr_options=model_spec,
        )

        converter = DocumentConverter(
            format_options={
                InputFormat.AUDIO: AudioFormatOption(
                    pipeline_cls=AsrPipeline,
                    pipeline_options=pipeline_options,
                )
            }
        )

        self.progress.emit("Transcribing audio...")

        start_time = time.perf_counter()
        conv_result = converter.convert(str(file_path))
        end_time = time.perf_counter()

        processing_time = end_time - start_time

        self.progress.emit("Extracting content...")

        doc = conv_result.document

        segments = []
        if doc.texts:
            for text_item in doc.texts:
                start = 0.0
                end = 0.0

                if hasattr(text_item, 'source') and text_item.source:
                    track = text_item.source[0]
                    start = float(getattr(track, 'start_time', 0.0) or 0.0)
                    end = float(getattr(track, 'end_time', 0.0) or 0.0)

                segments.append(TranscriptionSegment(
                    start_time=start,
                    end_time=end,
                    text=text_item.text.strip()
                ))

        markdown_content = doc.export_to_markdown()
        text_content = "\n".join([seg.text for seg in segments])
        html_content = self._generate_html(segments, file_path.name)

        json_content = json.dumps({
            "filename": file_path.name,
            "model": self.model_name,
            "device": self.device,
            "batch_size": self.batch_size,
            "processing_time_seconds": processing_time,
            "segments": [
                {
                    "start": seg.start_time,
                    "end": seg.end_time,
                    "text": seg.text
                } for seg in segments
            ],
            "full_text": text_content
        }, indent=2)

        srt_content = self._generate_srt(segments)
        vtt_content = self._generate_vtt(segments)

        self.progress.emit("Done!")

        return TranscriptionResult(
            success=True,
            filename=file_path.name,
            markdown=markdown_content,
            text=text_content,
            html=html_content,
            json_data=json_content,
            srt=srt_content,
            vtt=vtt_content,
            segments=segments,
            processing_time=processing_time,
            model_name=self.model_name,
            device=self.device
        )

    def _format_timestamp(self, seconds: float, use_comma: bool = False) -> str:
        if seconds is None:
            seconds = 0.0
        hours = int(seconds // 3600)
        minutes = int((seconds % 3600) // 60)
        secs = int(seconds % 60)
        millis = int((seconds % 1) * 1000)
        sep = "," if use_comma else "."
        return f"{hours:02d}:{minutes:02d}:{secs:02d}{sep}{millis:03d}"

    def _generate_srt(self, segments: list) -> str:
        lines = []
        for i, seg in enumerate(segments, 1):
            lines.append(str(i))
            start = self._format_timestamp(seg.start_time, use_comma=True)
            end = self._format_timestamp(seg.end_time, use_comma=True)
            lines.append(f"{start} --> {end}")
            lines.append(seg.text)
            lines.append("")
        return "\n".join(lines)

    def _generate_vtt(self, segments: list) -> str:
        lines = ["WEBVTT", ""]
        for seg in segments:
            start = self._format_timestamp(seg.start_time, use_comma=False)
            end = self._format_timestamp(seg.end_time, use_comma=False)
            lines.append(f"{start} --> {end}")
            lines.append(seg.text)
            lines.append("")
        return "\n".join(lines)

    def _generate_html(self, segments: list, filename: str) -> str:
        segment_html = ""
        for seg in segments:
            start = self._format_timestamp(seg.start_time)
            end = self._format_timestamp(seg.end_time)
            segment_html += f"""
            <div class="segment">
                <span class="timestamp">[{start} → {end}]</span>
                <span class="text">{seg.text}</span>
            </div>"""

        return f"""<!DOCTYPE html>
<html>
<head>
    <meta charset="utf-8">
    <title>Transcription: {filename}</title>
    <style>
        body {{
            font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
            padding: 20px;
            line-height: 1.8;
            background: #1e1e1e;
            color: #d4d4d4;
            max-width: 900px;
            margin: 0 auto;
        }}
        h1 {{
            color: #569cd6;
            border-bottom: 2px solid #3a3a3a;
            padding-bottom: 10px;
        }}
        .segment {{
            margin: 15px 0;
            padding: 10px;
            background: #252525;
            border-radius: 5px;
            border-left: 3px solid #569cd6;
        }}
        .timestamp {{
            color: #4ec9b0;
            font-family: 'Consolas', monospace;
            font-size: 0.85em;
            margin-right: 10px;
        }}
        .text {{
            color: #d4d4d4;
        }}
    </style>
</head>
<body>
    <h1>Transcription: {filename}</h1>
    {segment_html}
</body>
</html>"""


class TranscriptionStructureTree(QTreeWidget):
    def __init__(self):
        super().__init__()
        self.setHeaderLabels(["Element", "Details"])
        self.setColumnWidth(0, 200)
        self.setAlternatingRowColors(True)

    def populate(self, result: TranscriptionResult):
        self.clear()

        root = QTreeWidgetItem(self, [result.filename, ""])
        root.setExpanded(True)

        info_item = QTreeWidgetItem(root, ["Transcription Info", ""])
        info_item.setExpanded(True)
        QTreeWidgetItem(info_item, ["Model", result.model_name])
        QTreeWidgetItem(info_item, ["Device", result.device])
        QTreeWidgetItem(info_item, ["Processing Time", f"{result.processing_time:.2f}s"])

        if result.segments:
            segments_item = QTreeWidgetItem(root, ["Segments", f"({len(result.segments)})"])
            segments_item.setExpanded(True)

            for i, seg in enumerate(result.segments[:50], 1):
                start = self._format_time(seg.start_time)
                end = self._format_time(seg.end_time)
                preview = seg.text[:40] + "..." if len(seg.text) > 40 else seg.text
                seg_item = QTreeWidgetItem(segments_item, [f"Segment {i}", f"{start} → {end}"])
                QTreeWidgetItem(seg_item, ["Text", preview])

            if len(result.segments) > 50:
                QTreeWidgetItem(segments_item, ["...", f"({len(result.segments) - 50} more segments)"])

        stats_item = QTreeWidgetItem(root, ["Statistics", ""])
        stats_item.setExpanded(True)

        total_duration = 0.0
        if result.segments:
            total_duration = max((seg.end_time for seg in result.segments), default=0.0)

        text_len = len(result.text) if result.text else 0
        word_count = len(result.text.split()) if result.text else 0

        QTreeWidgetItem(stats_item, ["Duration", self._format_time(total_duration)])
        QTreeWidgetItem(stats_item, ["Segments", str(len(result.segments))])
        QTreeWidgetItem(stats_item, ["Characters", f"{text_len:,}"])
        QTreeWidgetItem(stats_item, ["Words", f"{word_count:,}"])

        if total_duration > 0:
            realtime_factor = result.processing_time / total_duration
            QTreeWidgetItem(stats_item, ["Realtime Factor", f"{realtime_factor:.2f}x"])

    def _format_time(self, seconds: float) -> str:
        if seconds is None:
            return "N/A"
        minutes = int(seconds // 60)
        secs = seconds % 60
        if minutes > 0:
            return f"{minutes}m {secs:.2f}s"
        return f"{secs:.2f}s"


class ContentViewer(QTabWidget):
    def __init__(self):
        super().__init__()

        self.markdown_view = QTextEdit()
        self.markdown_view.setReadOnly(True)
        self.markdown_view.setFont(QFont("Consolas", 10))
        self.addTab(self.markdown_view, "Markdown")

        self.text_view = QTextEdit()
        self.text_view.setReadOnly(True)
        self.text_view.setFont(QFont("Consolas", 10))
        self.addTab(self.text_view, "Plain Text")

        self.html_view = None
        if HAS_WEBENGINE:
            try:
                self.html_view = QWebEngineView()
                self.addTab(self.html_view, "HTML Preview")
            except Exception:
                self.html_view = None

        if self.html_view is None:
            self.html_preview_fallback = QTextEdit()
            self.html_preview_fallback.setReadOnly(True)
            self.html_preview_fallback.setFont(QFont("Consolas", 9))
            self.addTab(self.html_preview_fallback, "HTML Preview (Source)")

        self.srt_view = QTextEdit()
        self.srt_view.setReadOnly(True)
        self.srt_view.setFont(QFont("Consolas", 10))
        self.addTab(self.srt_view, "SRT Subtitles")

        self.vtt_view = QTextEdit()
        self.vtt_view.setReadOnly(True)
        self.vtt_view.setFont(QFont("Consolas", 10))
        self.addTab(self.vtt_view, "VTT Subtitles")

        self.json_view = QTextEdit()
        self.json_view.setReadOnly(True)
        self.json_view.setFont(QFont("Consolas", 9))
        self.addTab(self.json_view, "JSON Structure")

        self.segments_view = QTextEdit()
        self.segments_view.setReadOnly(True)
        self.segments_view.setFont(QFont("Consolas", 10))
        self.addTab(self.segments_view, "Segments")

    def display(self, result: TranscriptionResult):
        self.markdown_view.setPlainText(result.markdown)
        self.text_view.setPlainText(result.text)

        if self.html_view is not None:
            self.html_view.setHtml(result.html)
        else:
            self.html_preview_fallback.setPlainText(result.html)

        self.srt_view.setPlainText(result.srt)
        self.vtt_view.setPlainText(result.vtt)
        self.json_view.setPlainText(result.json_data)

        if result.segments:
            segments_text = ""
            for i, seg in enumerate(result.segments, 1):
                segments_text += f"{'='*60}\n"
                segments_text += f"SEGMENT {i}\n"
                segments_text += f"{'='*60}\n"
                segments_text += f"Start: {seg.start_time:.3f}s\n"
                segments_text += f"End:   {seg.end_time:.3f}s\n"
                segments_text += f"Duration: {seg.end_time - seg.start_time:.3f}s\n"
                segments_text += f"\n{seg.text}\n\n"
            self.segments_view.setPlainText(segments_text)
        else:
            self.segments_view.setPlainText("No segments found.")

    def clear_all(self):
        self.markdown_view.clear()
        self.text_view.clear()
        if self.html_view is not None:
            self.html_view.setHtml("")
        else:
            self.html_preview_fallback.clear()
        self.srt_view.clear()
        self.vtt_view.clear()
        self.json_view.clear()
        self.segments_view.clear()


class MainWindow(QMainWindow):
    SUPPORTED_FORMATS = (
        "Audio Files (*.mp3 *.wav *.flac *.m4a *.aac *.ogg *.wma *.opus *.webm *.mp4 *.mkv *.avi);;"
        "MP3 Files (*.mp3);;"
        "WAV Files (*.wav);;"
        "FLAC Files (*.flac);;"
        "M4A Files (*.m4a);;"
        "AAC Files (*.aac);;"
        "OGG Files (*.ogg);;"
        "WMA Files (*.wma);;"
        "Video Files (*.mp4 *.mkv *.avi *.webm);;"
        "All Files (*.*)"
    )

    MODEL_OPTIONS = [
        "large-v3",
        "distil-large-v3",
        "medium",
        "medium.en",
        "distil-medium.en",
        "small",
        "small.en",
        "distil-small.en",
        "base",
        "base.en",
        "tiny",
        "tiny.en",
    ]

    def __init__(self):
        super().__init__()
        self.setWindowTitle("Docling WhisperS2T Audio Transcription")
        self.setMinimumSize(1200, 800)
        self.worker: Optional[TranscriptionWorker] = None
        self.current_result: Optional[TranscriptionResult] = None
        self._setup_ui()

    def _setup_ui(self):
        central_widget = QWidget()
        self.setCentralWidget(central_widget)
        main_layout = QVBoxLayout(central_widget)
        main_layout.setSpacing(10)
        main_layout.setContentsMargins(10, 10, 10, 10)

        file_group = QGroupBox("File Selection")
        file_layout = QHBoxLayout(file_group)

        self.file_label = QLabel("No file selected")
        self.file_label.setStyleSheet("padding: 8px; background-color: palette(base); border-radius: 4px;")
        self.file_label.setSizePolicy(QSizePolicy.Expanding, QSizePolicy.Preferred)
        file_layout.addWidget(self.file_label)

        self.browse_button = QPushButton("Browse...")
        self.browse_button.setMinimumWidth(100)
        self.browse_button.clicked.connect(self._browse_file)
        file_layout.addWidget(self.browse_button)

        main_layout.addWidget(file_group)

        settings_group = QGroupBox("Transcription Settings")
        settings_layout = QHBoxLayout(settings_group)

        settings_layout.addWidget(QLabel("Model:"))
        self.model_combo = QComboBox()
        self.model_combo.addItems(self.MODEL_OPTIONS)
        self.model_combo.setCurrentText("large-v3")
        self.model_combo.setMinimumWidth(150)
        settings_layout.addWidget(self.model_combo)

        settings_layout.addSpacing(20)

        settings_layout.addWidget(QLabel("Device:"))
        self.device_combo = QComboBox()
        self.device_combo.addItems(["cuda", "cpu"])
        self.device_combo.setCurrentText("cuda")
        self.device_combo.setMinimumWidth(80)
        settings_layout.addWidget(self.device_combo)

        settings_layout.addSpacing(20)

        settings_layout.addWidget(QLabel("Batch Size:"))
        self.batch_spinbox = QSpinBox()
        self.batch_spinbox.setRange(1, 64)
        self.batch_spinbox.setValue(8)
        self.batch_spinbox.setMinimumWidth(80)
        settings_layout.addWidget(self.batch_spinbox)

        settings_layout.addStretch()

        self.transcribe_button = QPushButton("Transcribe")
        self.transcribe_button.setMinimumWidth(120)
        self.transcribe_button.setEnabled(False)
        self.transcribe_button.clicked.connect(self._start_transcription)
        settings_layout.addWidget(self.transcribe_button)

        main_layout.addWidget(settings_group)

        self.progress_bar = QProgressBar()
        self.progress_bar.setTextVisible(True)
        self.progress_bar.setFormat("")
        self.progress_bar.setMaximum(0)
        self.progress_bar.setMinimum(0)
        self.progress_bar.setVisible(False)
        main_layout.addWidget(self.progress_bar)

        splitter = QSplitter(Qt.Horizontal)

        left_panel = QWidget()
        left_layout = QVBoxLayout(left_panel)
        left_layout.setContentsMargins(0, 0, 0, 0)

        structure_label = QLabel("Transcription Structure")
        structure_label.setStyleSheet("font-weight: bold; padding: 5px;")
        left_layout.addWidget(structure_label)

        self.structure_tree = TranscriptionStructureTree()
        left_layout.addWidget(self.structure_tree)

        splitter.addWidget(left_panel)

        right_panel = QWidget()
        right_layout = QVBoxLayout(right_panel)
        right_layout.setContentsMargins(0, 0, 0, 0)

        content_label = QLabel("Transcription Content")
        content_label.setStyleSheet("font-weight: bold; padding: 5px;")
        right_layout.addWidget(content_label)

        self.content_viewer = ContentViewer()
        right_layout.addWidget(self.content_viewer)

        splitter.addWidget(right_panel)
        splitter.setSizes([300, 900])

        main_layout.addWidget(splitter, 1)

        button_layout = QHBoxLayout()

        self.export_combo = QComboBox()
        self.export_combo.addItems(["Markdown", "Plain Text", "HTML", "JSON", "SRT", "VTT"])
        self.export_combo.setMinimumWidth(120)
        button_layout.addWidget(QLabel("Export as:"))
        button_layout.addWidget(self.export_combo)

        self.export_button = QPushButton("Export...")
        self.export_button.setEnabled(False)
        self.export_button.clicked.connect(self._export_content)
        button_layout.addWidget(self.export_button)

        button_layout.addStretch()

        self.clear_button = QPushButton("Clear")
        self.clear_button.clicked.connect(self._clear_all)
        button_layout.addWidget(self.clear_button)

        main_layout.addLayout(button_layout)

        self.status_bar = QStatusBar()
        self.setStatusBar(self.status_bar)
        self.status_bar.showMessage("Ready - Select an audio file to begin")

    def _browse_file(self):
        file_path, _ = QFileDialog.getOpenFileName(
            self,
            "Select Audio File",
            "",
            self.SUPPORTED_FORMATS
        )
        if file_path:
            self.file_label.setText(file_path)
            self.transcribe_button.setEnabled(True)
            self.status_bar.showMessage(f"Selected: {Path(file_path).name}")

    def _start_transcription(self):
        file_path = self.file_label.text()
        if file_path == "No file selected":
            return

        self.transcribe_button.setEnabled(False)
        self.browse_button.setEnabled(False)
        self.export_button.setEnabled(False)
        self.model_combo.setEnabled(False)
        self.device_combo.setEnabled(False)
        self.batch_spinbox.setEnabled(False)
        self.progress_bar.setVisible(True)
        self.status_bar.showMessage("Transcribing...")

        self.content_viewer.clear_all()
        self.structure_tree.clear()

        self.worker = TranscriptionWorker(
            file_path,
            self.model_combo.currentText(),
            self.device_combo.currentText(),
            self.batch_spinbox.value()
        )
        self.worker.progress.connect(self._on_progress)
        self.worker.finished.connect(self._on_transcription_finished)
        self.worker.start()

    def _on_progress(self, message: str):
        self.progress_bar.setFormat(message)
        self.status_bar.showMessage(message)

    def _on_transcription_finished(self, result: TranscriptionResult):
        self.progress_bar.setVisible(False)
        self.transcribe_button.setEnabled(True)
        self.browse_button.setEnabled(True)
        self.model_combo.setEnabled(True)
        self.device_combo.setEnabled(True)
        self.batch_spinbox.setEnabled(True)

        self.current_result = result

        if result.success:
            self.content_viewer.display(result)
            self.structure_tree.populate(result)
            self.export_button.setEnabled(True)
            self.status_bar.showMessage(
                f"Successfully transcribed: {result.filename} in {result.processing_time:.2f}s"
            )
        else:
            QMessageBox.warning(
                self,
                "Transcription Failed",
                f"Failed to transcribe {result.filename}:\n\n{result.error_message}"
            )
            self.status_bar.showMessage(f"Transcription failed: {result.filename}")

    def _export_content(self):
        if not self.current_result or not self.current_result.success:
            return

        format_map = {
            "Markdown": ("md", self.current_result.markdown),
            "Plain Text": ("txt", self.current_result.text),
            "HTML": ("html", self.current_result.html),
            "JSON": ("json", self.current_result.json_data),
            "SRT": ("srt", self.current_result.srt),
            "VTT": ("vtt", self.current_result.vtt),
        }

        selected_format = self.export_combo.currentText()
        ext, content = format_map[selected_format]

        default_name = Path(self.current_result.filename).stem + f".{ext}"

        file_path, _ = QFileDialog.getSaveFileName(
            self,
            f"Export as {selected_format}",
            default_name,
            f"{selected_format} Files (*.{ext})"
        )

        if file_path:
            try:
                with open(file_path, "w", encoding="utf-8") as f:
                    f.write(content)
                self.status_bar.showMessage(f"Exported to: {file_path}")
                QMessageBox.information(self, "Export Successful", f"Content exported to:\n{file_path}")
            except Exception as e:
                QMessageBox.critical(self, "Export Failed", f"Failed to export:\n{str(e)}")

    def _clear_all(self):
        self.file_label.setText("No file selected")
        self.transcribe_button.setEnabled(False)
        self.export_button.setEnabled(False)
        self.content_viewer.clear_all()
        self.structure_tree.clear()
        self.current_result = None
        self.status_bar.showMessage("Ready - Select an audio file to begin")


def main():
    try:
        app = QApplication(sys.argv)
        app.setStyle("Fusion")

        palette = QPalette()
        palette.setColor(QPalette.Window, QColor(53, 53, 53))
        palette.setColor(QPalette.WindowText, QColor(255, 255, 255))
        palette.setColor(QPalette.Base, QColor(35, 35, 35))
        palette.setColor(QPalette.AlternateBase, QColor(53, 53, 53))
        palette.setColor(QPalette.ToolTipBase, QColor(25, 25, 25))
        palette.setColor(QPalette.ToolTipText, QColor(255, 255, 255))
        palette.setColor(QPalette.Text, QColor(255, 255, 255))
        palette.setColor(QPalette.Button, QColor(53, 53, 53))
        palette.setColor(QPalette.ButtonText, QColor(255, 255, 255))
        palette.setColor(QPalette.BrightText, QColor(255, 0, 0))
        palette.setColor(QPalette.Link, QColor(42, 130, 218))
        palette.setColor(QPalette.Highlight, QColor(42, 130, 218))
        palette.setColor(QPalette.HighlightedText, QColor(35, 35, 35))
        palette.setColor(QPalette.Disabled, QPalette.WindowText, QColor(127, 127, 127))
        palette.setColor(QPalette.Disabled, QPalette.Text, QColor(127, 127, 127))
        palette.setColor(QPalette.Disabled, QPalette.ButtonText, QColor(127, 127, 127))
        app.setPalette(palette)

        app.setStyleSheet("""
            QGroupBox {
                font-weight: bold;
                border: 1px solid #3a3a3a;
                border-radius: 5px;
                margin-top: 10px;
                padding-top: 10px;
            }
            QGroupBox::title {
                subcontrol-origin: margin;
                left: 10px;
                padding: 0 5px 0 5px;
            }
            QPushButton {
                padding: 8px 16px;
                border-radius: 4px;
                background-color: #2a82da;
                color: white;
                border: none;
            }
            QPushButton:hover {
                background-color: #3a92ea;
            }
            QPushButton:pressed {
                background-color: #1a72ca;
            }
            QPushButton:disabled {
                background-color: #555555;
                color: #888888;
            }
            QTabWidget::pane {
                border: 1px solid #3a3a3a;
                border-radius: 4px;
            }
            QTabBar::tab {
                padding: 8px 16px;
                margin-right: 2px;
                background-color: #3a3a3a;
                border-top-left-radius: 4px;
                border-top-right-radius: 4px;
            }
            QTabBar::tab:selected {
                background-color: #2a82da;
            }
            QTreeWidget {
                border: 1px solid #3a3a3a;
                border-radius: 4px;
            }
            QTextEdit {
                border: 1px solid #3a3a3a;
                border-radius: 4px;
            }
            QProgressBar {
                border: 1px solid #3a3a3a;
                border-radius: 4px;
                text-align: center;
            }
            QProgressBar::chunk {
                background-color: #2a82da;
            }
            QComboBox {
                padding: 5px 10px;
                border: 1px solid #3a3a3a;
                border-radius: 4px;
                background-color: #3a3a3a;
            }
            QComboBox::drop-down {
                border: none;
            }
            QComboBox QAbstractItemView {
                background-color: #3a3a3a;
                selection-background-color: #2a82da;
            }
            QSpinBox {
                padding: 5px 10px;
                border: 1px solid #3a3a3a;
                border-radius: 4px;
                background-color: #3a3a3a;
            }
        """)

        window = MainWindow()
        window.show()

        sys.exit(app.exec())

    except Exception as e:
        print(f"Fatal error: {e}")
        print(traceback.format_exc())
        input("Press Enter to exit...")
        sys.exit(1)


if __name__ == "__main__":
    main()

[github-actions]

✅ DCO Check Passed

Thanks @BBC-Esq, all your commits are properly signed off. 🎉

[dosubot]

Related Documentation

Checked 7 published document(s) in 1 knowledge base(s). No updates required.

^{How did I do? Any feedback?}  

[mergify]

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🟢 Enforce conventional commit

Wonderful, this rule succeeded.

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

[BBC-Esq]

Here goes nothing. I put a lot of time and effort into this PR and I jumped through all the freakin hoops as far as certifying the PR, correcting the syntax...Hopefully big company IBM does not equal shitty customer experience and/or online culture when simply trying to contribute to a cool idea. I don't even see a freakin button to request a review by whomever maintains this repo...Oh well, here goes nothing!

[codecov]

Codecov Report

❌ Patch coverage is 81.35593% with 33 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
docling/cli/main.py	0.00%	28 Missing ⚠️
docling/pipeline/asr_pipeline.py	96.38%	3 Missing ⚠️
docling/datamodel/pipeline_options_asr_model.py	91.30%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[BBC-Esq]

I'm trying my best to adhere to the multiple bots and their instructions...any help would be much appreciated. Thanks. lol.

[ceberam]

Hi @BBC-Esq
Thanks for taking the time to put together this PR! It’s clear you’ve invested a lot of effort, and we appreciate that.

We’ll review it shortly and let you know if anything needs clarification or further work.
For detailed guidance on contributions, please refer to the Docling contribution guidelines, the Docling developer guideline, and also the Code of Conduct.
In particular, you may want to check the section Coding Style Guildelines to learn how pre-commit hooks can help when creating new commits.
For now, I would suggest that you keep the focus of this PR on the feature you are proposing (whisper-s2t-reborn support) and leave out optional styling changes on non-related files.

We take all contributions seriously and will keep you posted on the status of this PR. If you have any questions in the meantime, just drop a comment here or open an issue.

Thanks again for your work!

[BBC-Esq]

Hi @BBC-Esq Thanks for taking the time to put together this PR! It’s clear you’ve invested a lot of effort, and we appreciate that.

We’ll review it shortly and let you know if anything needs clarification or further work. For detailed guidance on contributions, please refer to the Docling contribution guidelines, the Docling developer guideline, and also the Code of Conduct. In particular, you may want to check the section Coding Style Guildelines to learn how pre-commit hooks can help when creating new commits. For now, I would suggest that you keep the focus of this PR on the feature you are proposing (whisper-s2t-reborn support) and leave out optional styling changes on non-related files.

We take all contributions seriously and will keep you posted on the status of this PR. If you have any questions in the meantime, just drop a comment here or open an issue.

Thanks again for your work!

Thanks for your response. I can try to change the PR to remove the unrelated scripts if need be. They were included by accident when I ran ruff on my entire fork's codebase so...or feel free to modify the PR to the discrete topic at hand. Was taken agack by the number of bots all with different criteria but I'll review the links you sent.

[BBC-Esq]

Any chance this could be reviewed?

[BBC-Esq]

Hi @BBC-Esq

Thanks for taking the time to put together this PR! It’s clear you’ve invested a lot of effort, and we appreciate that.

We’ll review it shortly and let you know if anything needs clarification or further work.

For detailed guidance on contributions, please refer to the Docling contribution guidelines, the Docling developer guideline, and also the Code of Conduct.

In particular, you may want to check the section Coding Style Guildelines to learn how pre-commit hooks can help when creating new commits.

For now, I would suggest that you keep the focus of this PR on the feature you are proposing (whisper-s2t-reborn support) and leave out optional styling changes on non-related files.

We take all contributions seriously and will keep you posted on the status of this PR. If you have any questions in the meantime, just drop a comment here or open an issue.

Thanks again for your work!

Hello, you said you guys would be following up shortly? Is there anything preventing this from being merged? @ceberam

[PeterStaar-IBM]

@BBC-Esq let me follow up. We were right in the middle of a big refactoring to clean up the pipelines/stages/models/etc (see the merge conflicts). This is now getting finalized.

[ceberam]

Thanks again @BBC-Esq for the contribution and for exploring improvements in the ASR backend.
I’ve reviewed the PR and I’m not comfortable with the current form for several reasons:

1. Licensing

The repository currently does not contain a LICENSE file. Without an explicit open-source license, there is no clear legal grant of rights to use, modify, or redistribute the code.
Since this project is part of the LF AI & Data, we need all dependencies to have a clear, identifiable open-source license. Even though this repository appears to be a fork of WhisperS2T (which is MIT-licensed), that license must be explicitly included in the forked repository for it to be valid for redistribution.
Until this is clarified and properly addressed upstream, we cannot safely introduce this dependency.

2. Project health

The repository currently has:

• No visible community adoption (0 stars/forks)
• Limited signals of review
• No documented benchmark results supporting the performance claims

If performance is the main motivation, it would help to include reproducible benchmarks or comparative results demonstrating the advantage over existing ASR backend models. Without that, it is difficult to justify introducing an additional dependency with unclear maintenance status.

3. PR scope and conflicts

This PR also includes styling and formatting changes across multiple files that are unrelated to the new dependency. While some of those changes may be acceptable in isolation (e.g., I also prefer str | None to Optional[str]), they are out of scope for this feature and make the review more difficult. Our CONTRIBUTING.md guidelines already describe how to run the automatic style checks locally, which should prevent unrelated formatting diffs. In addition, as pointed out by @PeterStaar-IBM , recent changes have introduced some merge conflicts that should be addressed once the refactoring has finalized.

I would be happy to re-review again once:

• The upstream repository clearly includes an appropriate open-source license.
• There is objective evidence supporting the claimed performance benefits.
• The PR is cleaned up to only include changes strictly required for the ASR feature.
• The merge conflicts are resolved.

Thanks again for the contribution and for understanding the need to keep compliance, scope, and maintainability in mind.

[BBC-Esq]

@ceberam I appreciate such a thorough and professional response. I, too, care about the code base and your message makes sense.

LONG-WINDED RESPONSE HERE FOR LESS SCROLLING

1) Regarding the license issue, I'll fix that this week. Thanks for the link to LFAI, didn't know about that. I'll check out actually joining the org...

2. Regarding issue 2, I maintain the WhisperS2T-Reborn project and it's actively maintained. Other than TensorRT, it is the fastest library to run Whisper models currently available...and still just as good quality. It's been approximately two years, but I'll update my benchmarking suite, update the repo, and post a link here to review.

Regarding "unclear maintenance" status, I totally get that. You don't want to modify your guys' code base, add a dependency (even an optional one), and then have a part of your code base outdated because someone abandons WhisperS2T-Reborn. At a minimum, it would just be a hassle to unwind the "integration" so to speak.

Since I maintain WhisperS2T-Reborn, it logically makes sense to explain about my background a little to hopefully assuage your concern.

First off, I'm a lawyer by trade but I program as hobby. I originally became fascinated by LLMs and machine learning upon learning about "vector databases" and how they enable the search-ability of massive amounts of data; for example, data obtained during discovery in a lawsuit. In my profession, there's frequently large amounts of data in the form of text messages, emails, audio recordings, video clips, etc...

As an offshoot of that, transcribing audio and then being able to search it "semantically is a huge benefit for litigation purposes. It means that I no longer have to listen to hours of audio to pinpoint the minute/second where something was said that is important for litigation purposes, which ultimately saves the client billables. This is essentially what lead me to learn about Whisper models. The "vanilla" Whisper code, while revolutionary at the time, is painstakingly slow, which leads me to faster back-ends like Ctranslate2 upon which WhisperS2T-Reborn is based. That was 3-4 years ago now and I've been working with Whisper, vector databases, and other related technologies ever since. I was frequently in contact with the maintainer of the original "WhisperS2T" repository until he abandoned it in 2024 because he was offered a high-paying job, presumably using WhisperS2T to pad his resume. As you know, this frequently happens - i.e. people use their Github repositories to pad their resumes to land a high-paying job...nothing wrong with that, but often as part of a new job you can't work on competing open source projects and/or just don't have the time so...

At any rate, I decided to maintain it since it was slowly becoming incompatible with other APIs such as transformers as their code bases improved/changed.

I also maintain the models that WhisperS2T-Reborn relies upon by default, which you can see here on huggingface. I'd like to highlight that all of the models are converted from the original "float32" Whisper models, NOT the "float16" versions that OpenAI has decided to upload for better or worse. See the config here to see what I'm referring to.

ORIGINALLY, OpenAI uploaded float32 versions of all models but then replaced them with float16 versions. I disagreed with this because, yes, float16 is compatible with more GPUs and the file size is less, but if you're running on CPU it'll just have to be upcast to float32 anyways and you cannot regain the quality loss from the initial conversion from float32 to float16. Moreover, more recent GPUs support bfloat16...People wishing for the additional stability that bfloat16 offers won't get it...after all, when converting a float16 model to bfloat16 you cannot regain the lost precision.

In short, all of my models are converted from the original float32 versions so there is no quality loss. The WhisperS2T-Reborn library can choose the best model based on a user's hardware. For GPUs that support bfloat16, it'll choose bfloat16...and for older GPUs it'll fallback to float16. CPUs use float32 or...a user can still choose to use float32 on GPU for maximum quality.

This required downloading the original float32 weights of all whisper models from the history of the various repositories, adding any updated configuration "JSON" files (e.g. if OpenAI found a bug in something), then converting them to the Ctranslate2 format.

My repository provides float32, float16, and bfloatg16 versions of all Whisper models and the Ctranslate2 backend also supports "int8" if that's desirable.

Moreover, regarding Ctranslate2, it has an excellent feature whereby it'll convert at runtime if need be. For example, if you accidentally downloaded a bfloat16 model but your GPU only supports float16, it'll automatically convert and run in float16 without a hitch. Granted, there's a very small quality loss but at least it will not throw an error.

Ctranslate2 also recently adopted ROCm support in addition to CUDA, and already supports fast execution on both Intel and AMD cpus.

Incidentally, there's a fascinating video with a guy named Ben Fletcher who used to work at IBM regarding document parsing and RAG versus traditional similarity searches like tf/idf and bm25 and so on... https://www.youtube.com/watch?v=MdF--Fz0k4I&t=214s Maybe you guys know him, he used to work at IBM?

Basically, I've been working with Whisper models since approximately 2023 and don't plan to quite anytime soon, if that's any assurance regarding the "health" of the WhisperS2T-Reborn library...

Regarding item 3, I'll redo the PR to only pertain to the subject matter...it's just that without those additional changes your review is throwing errors...would be willing to do a separate PR to resolve those errors, however. I'll also do the stylistic checks locally like you suggest.

Cheers!

[BBC-Esq]

@ceberam Should be ready to review now. I updated the whisper-s2t-reborn library to support the whisper large v3 model because I noticed that docling uses that one. To address your specific questions:

1. MIT License has been added.

2. Benchmarks are located here https://github.com/BBC-Esq/WhisperS2T-reborn?tab=readme-ov-file#benchmarks and you can test yourself here if you'd like https://github.com/BBC-Esq/WhisperS2T-reborn/tree/main/benchmarks

3. The git history has been cleaned up in order to remove the extraneous changes not directly relevant to this PR.

A couple of questions, however...

1. I pinned whisper-s2t-reborn to torch 2.8+ but less than 3. Is that sufficient or are there other components of the [asr] option you think might need a different one? My only concern was that 2.8 wasn't low enough, but let me know.

2. The ctranslate2 backend to whisper-s2t-reborn actually supports int8 for whisper models. I intentionally didn't include that option. Thus, currently, when CPU is the compute device it'll default to the float32 version of whatever size of whisper model is chosen. The main difference is vram usage...not a speedup compared to float32 surprisingly. I'd recommend adding int8 later. It enables the ruybackend for certain CPUs and so on...but for simplicity's sake it's just float32, bfloat16, and float16 models for now.,

3. I changed the name of test for "asr" since the name was getting long with the current naming convention including the name of each backend...Hope you guys didn't mind.

Lastly, I'm attaching to this message some benchmarking scripts that actually utilize the docling pipeline as well. Should be pretty straighforward except notice how I like to pip install the cuda libraries and use the "set_cuda_paths" function instead of relying on a systemwide installation. Please let me know if you guys have any questions:

test_native_whisper_backend.py
test_whispers2t_backend.py

[BBC-Esq]

Everything should be good now I believe. I fixed the few remaining workflow issues I believe and have requested a re-review.

[BBC-Esq]

@ceberam or @PeterStaar-IBM I believe this is ready for review again. Is there anything I need to do? I've been updating my main branch as well as the development fork to incorporate the recent changes to the library as a whole.

[BBC-Esq]

I rebased and fixed the lingering ruff issue, so I hope, so should be good...

[mergify]

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🟢 Enforce conventional commit

Wonderful, this rule succeeded.

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

[cau-git]

@BBC-Esq Thanks for maintaining this branch and keeping up with the changes. Time to get it across the finish line!

I have only small remarks below for alignment purposes and potential config simplification, see below.

A few things going beyond the scope of this PR (so, after we merge this), that would be appreciated:

• It would be good if whisper S2T reborn could actually pin the revision of the HF models it pulls and expose that, so we can have a fully reproducible setup on the side of docling. Meaning, any change in model weights would also reflect as a code/config update of the revision hash for clarity and to meet good security compliance practices.
• We should build out the download facade for local offline usage like we do in other model families (used with artifacts_path and docling-tools models download utility). This would also need to be done for the original openai Whisper.

[mergify]

Merge Protections

🟢 Merge protection satisfied — ready to merge.

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

[BBC-Esq]

@BBC-Esq Thanks for maintaining this branch and keeping up with the changes. Time to get it across the finish line!

I have only small remarks below for alignment purposes and potential config simplification, see below.

A few things going beyond the scope of this PR (so, after we merge this), that would be appreciated:

* It would be good if whisper S2T reborn could actually pin the `revision` of the HF models it pulls and expose that, so we can have a fully reproducible setup on the side of docling. Meaning, any change in model weights would also reflect as a code/config update of the revision hash for clarity and to meet good security compliance practices.

* We should build out the download facade for local offline usage like we do in other model families (used with `artifacts_path` and `docling-tools models download` utility). This would also need to be done for the original openai Whisper.

Hey @cau-git Thanks for your message. I hadn't yet had a chance to address the rest of the issues that @ceberam had mentioned and then got your messages...I fully plan to so we can get this across the finish line! However, IRL stuff has some up (i.e. job stuff that makes money for my family and I) that's urgent...so give me a week or 1.5 weeks and I'll address everything. Didn't want you guys to think I forgot about this. Still want to get it done ASAP. ;-)

[BBC-Esq]

@cau-git and/or @ceberam. Quick question. Currently, my whispers2t-reborn repository pins the ctranslate2 dependency to 4.6.2. Originally, I intentionally did this because beginning with 4.6.3 ctranslate2 turns the cudnn reliance "off" by default in their published wheels....you must build manually to use cudnn because they implemented their own Conv1d CUDA operations, which were the only thing that cudnn was used for (and only specific to whisper models, no others).

See here for the release notes:

https://github.com/OpenNMT/CTranslate2/releases

According to Jordimas, who I trust and have dealt with for awhile now, the speed difference is negligible...and with my own testing I can't tell the difference.

See here:

OpenNMT/CTranslate2#1949

Question: Do you guys have a preference of (A) pinning to 4.6.2 so as to keep the minimal benefit of cudnn or (B) pin to higher release number so avoid the 300 MB+ cudnn dependency for cuda users. My thought is the latter, but I have no strong preference so wanted your feedback since I'm working on finalize in the PR. Thanks.

[EDIT] I decided to pin Whisper-S2T-reborn to ctranslate2 4.6.2 or higher, but less than 5.0. This will allow users to specifically pip install the last version of ctranslate2 that uses cudnn (4.6.2) if they really wanted to without pip check throwing a conflict notice...while still keeping it under a 5+ likely breaking change. Otherwise, if they install like most users it'll pull in the non-cudnn-reliant version of ctranslate2.

[BBC-Esq]

@cau-git @ceberam and whoever else. See these benchmarks. Let me know the best way to get the scripts to you. I can package them in a Claude skill ZIP if you'd like or paste the scripts in a piecemeal fashion. I used a new audio that's 18 minutes long to speed up the benchmarks instead of the 2 hour Sam Altman podcast, which I can send if need be.

PLEASE NOTE:

1. The whisper-s2t-reborn backend was run through docling as if the PR had been accepted. Therefore, it includes the minimal overhead from the docling itself. I ran these tests using both float32 and float16 "weights" of the models and the computations/math was also done in the same precision.

2. In contrast, the Vanilla whisper backend was tested using the openai-whisper library directly, NOT through docling. Docling's native path uses openai-whisper's greedy default (beam_size=None) + condition_on_previous_text=True, which triggers runaway repetition and/or hangs in my testing...see ([feat] Native Whisper: greedy decode default hangs on long-form audio - expose beam_size / condition_on_previous_text / temperature #3703). Therefore, I ran openai-whisper directly with beam_size=1 and condition_on_previous_text=False — what docling-native would do if/when feat(asr): expose beam_size and condition_on_previous_text (native) #3704 lands.

3. Additionally, the "vanilla" weights are only in float32, so you'll notice increased vram usage. The openai-whisper library allows me to specify float32 or float16 for the computations/math, so I tested both...but, again, the weights will always be in float32...Basically, I tested float32 and float16 computations. Note, docling currently defaults to float16 computations when a person has a GPU and float32 when using CPU...this is proper.

All testing done on an RTX 4090. I'm just showing the results for the whisper models that docling CURRENTLY supports for simplicity. However, whisper-s2t-reborn additionally supports these models that docling's vanilla whisper implementation does not:

tiny.en, base.en, small.en, distil-small.en, medium.en, distil-medium.en, distil-large-v3, distil-large-v3.5...all run with batch sizes 1, 2, 4, 8, 16, and 24.

Without further ado...

[BBC-Esq]

Closing out my own question above re: pinning ctranslate2 4.6.2 vs 4.6.3 — resolved. Moved to requiring whisper-s2t-reborn>=1.6.3 (4613c5c), which pulls the cuDNN-free ctranslate2 build. That drops the cuDNN dependency entirely (only cuBLAS remains, per the updated LD_LIBRARY_PATH note), so no manual cuDNN build/setup is needed.

[mergify]

Merge Protections

🟢 Merge protection satisfied — ready to merge.

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

[BBC-Esq]

@ceberam @cau-git — I believe the conditions from the February review are now addressed, and I've taken the path you proposed in June. Quick summary:

1. Licensing — whisper-s2t-reborn now ships an explicit MIT LICENSE (dual copyright: original author Shashi Kant Gupta © 2023 + me © 2025-2026), and the PyPI package declares License: MIT.
2. Performance evidence — posted above: speed and VRAM usage across both short and long audio, comparing WhisperS2T-Reborn against native openai-whisper through docling. Happy to share the benchmark scripts (I can package them as a ZIP or paste piecemeal) and the longer audio sample if useful.
3. Scope — the PR is now focused on the ASR feature, with the earlier unrelated formatting diffs cleaned up.
4. Merge conflicts — resolved; the branch is in sync with main and CI is green.

Most importantly, per your June 17 suggestion, WhisperS2T-Reborn is now opt-in / experimental rather than the default backend (ef212e6) — the default user experience is unchanged, and users must explicitly select it. That keeps docling's core dependency surface untouched while letting the community evaluate the backend in real-world use, which should also help address the maturity/adoption concern.

@cau-git's alignment remarks are addressed (resolved threads), and the two items he flagged as out-of-scope — pinning/exposing the HF model revision for reproducibility, and the offline download facade (artifacts_path / docling-tools models download) — I'm treating as post-merge follow-ups.

Could you re-review when you have a chance? cc @PeterStaar-IBM @dolfim-ibm @vagenas

[mergify]

Merge Protections

🟢 Merge protection satisfied — ready to merge.

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

[ceberam]

🚀