perf: optimize BatchedInferencePipeline — VAD on GPU, feature pipelining, buffer pre-allocation

[RozhinKh]

Summary

End-to-end latency improvements for BatchedInferencePipeline via CPU pipeline restructuring and GPU placement fixes. Config held constant on both branches — code changes only, no model or decoder modifications.

Results (cold-pass, cache disabled, same config on both branches):

Hardware	Config	Latency	Throughput
RTX 3090 24 GB	int8_float16 / beam=5 / batch=32	-15.7 to -21.6%	+27.6%
A100-SXM4-80GB 80 GB	bfloat16 / beam=5 / batch=32	-18.2 to -22.9%	--

SHA1-identical outputs across runs. 6/6 noise robustness conditions pass 3% WER regression threshold. All 9 Artemis ASR scenarios pass validity gate.

Changes

faster_whisper/vad.py

VAD on GPU — SileroVADModel previously hardcoded CPUExecutionProvider. Profiling showed the 1 MB VAD model was consuming ~1.5 s per request due to sequential CPU execution across hundreds of 512-sample windows. The session now prefers CUDAExecutionProvider (falling back to CPU when unavailable), reducing VAD time to ~0.05 s per request.

Single-allocation VAD buffer — __call__ previously required an already-padded array (a full np.pad copy). The new implementation accepts raw audio of any length and constructs the segmented buffer in a single np.empty allocation, zeroing only the segment-0 context strip and the last-segment tail padding. Eliminates ~80 MB of data movement for 21-minute audio. Verified bit-identical to the original across 11 audio lengths (0 to 20,160,001 samples).

faster_whisper/transcribe.py

Feature extraction pipelining — BatchedInferencePipeline previously blocked GPU inference waiting for CPU feature extraction to finish. A ThreadPoolExecutor(max_workers=1) now submits all batch extraction futures upfront. A background thread extracts batch N+1 features while CTranslate2 processes batch N on the GPU. CTranslate2 releases the GIL during CUDA operations, so the background thread runs freely with no synchronization overhead.

Feature buffer pre-allocation — replaced np.stack([pad_or_trim(f) for f in feats]) with a pre-allocated np.zeros((batch_size, n_mels, 3000), dtype=np.float32) result buffer, eliminating two intermediate allocations per batch.

Single-batch executor skip — for audio that fits in a single batch, bypasses the executor entirely since there is nothing to pipeline. Avoids thread creation overhead for short audio without affecting multi-batch workloads.

Eliminated duplicate tokenizer decode — tokenizer.decode(tokens) was called twice per subsegment in forward(). Walrus operator reduces this to one call.

Test plan

• Run benchmark/speed_benchmark.py on baseline vs optimized at matching config
• Verify transcript output is unchanged (SHA1 parity across runs)
• Confirm VAD fallback to CPU when CUDA unavailable
• Confirm single-batch path produces identical output to multi-batch path