perf: cut CPU Whisper latency and prune the catalog by measured WER - #47
Merged
Merged
Conversation
whisper.cpp ran on whisper-cli's own defaults — 4 threads and beam 5 / best-of 5, which are batch-transcription settings — and ignored the WebUI's CPU threads box entirely. Measured on a 17 s clip with ggml-tiny.en and the GPU disabled, asking for the machine's cores and a narrower beam takes the run from 1.83 s to 0.77 s with no change to the transcript. Temperature fallback stays on: it is what rescues a segment from a repetition loop. Thread counts now come from one policy in app.system — physical performance cores, clamped by the cgroup quota. The old min(os.cpu_count(), 8) read the host's core count inside a container and oversubscribed the quota, and it spread work across hyperthread siblings that then held the batch back. faster-whisper now skips silence with Silero VAD and stops decoding timestamp tokens nothing downstream reads, retrying without VAD when VAD finds no speech so a quiet clip cannot turn into a failure. sherpa-onnx builds its waveform with numpy rather than one Python float per sample. Two pipeline fixes outside the engines: FFmpegNormalizer read the whole PCM file and summed its squares on the event loop, stalling every concurrent request in flight; that now runs on a worker and is vectorized where numpy is present. ModelManager.installed_path walked and sized every installed model to return a single path, and now resolves it from the catalog index. Catalog changes are backed by the Open ASR Leaderboard and Argmax's own WhisperKit evaluation rather than by download size alone: - Retire WhisperKit Large v3 Turbo (1610 MB). Argmax runs both builds over the same 2,620 LibriSpeech utterances: 2.40% WER against 2.49% for the 626 MB compressed build. Eight hundredths of a point is not worth 984 MB. - Retire WhisperKit Small (3.95% WER); the compressed Large v3 beats it outright for 142 MB more, and its RAM floor drops to 8 GB to cover that tier. - Retire whisper.cpp Medium, Medium EN and the 3 GB Large v3. - Retire faster-whisper Distil Large v3 for v3.5, the same size at 5.40 average WER — ahead of full Large v3's 5.78. - Add faster-whisper Large v3 Turbo, Large v3, Medium, Medium EN and Distil Large v3.5. Before this the fastest multilingual Whisper a CPU-only host could choose was `small`. - Stop Turbo entries claiming to be the most accurate. The leaderboard puts Turbo at 6.36 against full Large v3's 5.78, and the catalog offers both. Retired entries stay listed and manageable when installed and point at a replacement; nothing is deleted from disk. Pins and docs/models.md are regenerated, and the pin harvester no longer writes pins for models that can never be downloaded again.
✅ Deploy Preview for voca-gateway canceled.
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
Mr-Sunglasses has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
…need Two follow-ups from review of the preceding commit. `_linux_physical_cpus` fell back to the logical count only when /proc/cpuinfo reported no topology at all, not when it reported a degenerate one. Some hypervisors give every vCPU the same (physical id, core id) pair, which parses as exactly one core — so a 16-vCPU guest would have run faster-whisper, sherpa-onnx and whisper.cpp single-threaded, several times slower than the min(os.cpu_count(), 8) this replaced, and silently. No CPU this gateway targets runs more than two threads per core, so a count implying otherwise is now taken as a floor rather than believed. A processor block that omits its own physical id also no longer inherits the previous block's. The sherpa-onnx waveform reader imports numpy unguarded, on a docstring claim that numpy ships with sherpa-onnx. It does not: sherpa-onnx declares only sherpa-onnx-core, and numpy arrives by way of faster-whisper's onnxruntime. The guarantee held only because the engines extra happens to be monolithic, so the extra now declares numpy outright and the docstring says where it comes from. app.audio keeps its stdlib fallback, since a core install still has no numpy and is not meant to.
There was a problem hiding this comment.
Mr-Sunglasses has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
CI type-checks and runs the whole suite against `uv sync --all-groups`, which installs no extras, while `just sync` adds `--all-extras` locally. numpy only ships with the `engines` extra, so the local gate was weaker than CI's and the preceding commit broke the CI environment two ways. `app/models/sherpa_onnx.py` imported numpy unguarded. mypy could not resolve it, and tests do reach `_read_wave_samples` through `transcribe` and `_decode_wave_online`, so the suite would have failed there once mypy passed. It now falls back to the stdlib comprehension it replaced, exactly as `app.audio` already does for its RMS gate, and a test pins the two paths to the same waveform for mono and multi-channel input. Slicing by the channel stride unconditionally also drops a branch: `[::1]` is a no-op. numpy joins the optional-extra modules in the mypy overrides for the same reason `faster_whisper` and `sherpa_onnx` are already there. It stays declared in the `engines` extra so real deployments keep the vectorized path, but the code no longer depends on that being true. Verified against both environments: `uv sync --locked --all-groups` and `uv sync --all-groups --all-extras` each pass ruff, flake8, mypy and pytest.
There was a problem hiding this comment.
Mr-Sunglasses has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
Mr-Sunglasses
deleted the
perf/cpu-whisper-decoding-and-catalog-audit
branch
September 3, 2026 20:50
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Two problems, found while looking at why Whisper is slow on a machine without a GPU.
whisper.cpp was running on
whisper-cli's own defaults — 4 threads and beam 5 / best-of 5, which are batch-transcription settings — and it ignored the WebUI's CPU threads box entirely. That box only ever reached faster-whisper and sherpa-onnx.The catalog carried tiers that nothing measures well. It also had a real gap: the fastest multilingual Whisper a CPU-only host could pick was
small.Performance
Measured on a 17 s clip,
ggml-tiny.en, GPU disabled, page cache warmed:-t 4, beam 5, best-of 5)-t <cores>, beam 2, best-of 2)Same transcript. Full greedy was 0.67 s but measurably degraded the text, so it was rejected. Temperature fallback stays on — it is what rescues a segment from a repetition loop, and disabling it only bought 0.03 s.
Other changes on that path:
system.inference_thread_count) replacing amin(os.cpu_count(), 8)duplicated across two engines. It counts physical cores — performance cores on Apple silicon, 8 rather than 10 on an M1 Pro — and clamps to the cgroup quota. The old code read the host's core count inside a container and oversubscribed the quota.FFmpegNormalizerread the whole PCM file and summed its squares on the event loop, stalling every concurrent request. Now on a worker thread, and vectorized where numpy is present — proved numerically identical to the old formula, with both branches tested.installed_path()walked and sized every installed model to return one path it never used a size from. Now O(1) from the catalog index.Catalog
Retirements are backed by published measurements, not by download size. Retired entries stay listed and manageable when installed and point at a replacement; nothing is deleted from disk.
_626MBcompressed_626MBcompressedfaster-whisper:large-v3Added (all verified against the HF API for real sizes and licences):
faster-whisper:large-v3-turbo,large-v3,medium,medium.en,distil-large-v3.5. Large v3 Turbo is the important one — it keeps Large v3's encoder and swaps its 32-layer decoder for four, and is now recommended for non-Apple hosts.Also corrected a false accuracy claim. Turbo entries advertised "Most accurate", but the Open ASR Leaderboard puts Turbo at 6.36 against full Large v3's 5.78 — and the catalog offers both, plus the Q5 build of the full weights. The label sent people to the least accurate of the three.
Deliberately not changed
whisper-server(port, lifecycle, health, crash recovery) is not worth 4%..envariants. Separately trained weights, measurably better on English (3.56% vs 3.95%), not a second quantization. There is now a test asserting they survive.Testing
ruff format,ruff check,flake8 --select=WPS,mypy --strictclean. 540 tests pass (up from 508). Pins anddocs/models.mdregenerated, both enforced by tests.New invariants: no recommendation may point at a retired model; every replacement id must resolve to an active one; the
.enpairs must both survive; no Turbo build may claim the top accuracy label; andinstalled_pathmust agree with the full scan.Verified end-to-end against the real
ffmpegandwhisper-clibinaries, including that silent audio is still rejected.Known follow-up
whisper.cppreturns a bare string, so the pipeline recordsmodel_load_ms = 0for it — the Pair & test benchmark therefore shows zero model load even though it reloads every request. Pre-existing, but adjacent to this work and worth fixing separately.Review follow-up (second commit)
A review pass after the first commit found two things, both since fixed:
_linux_physical_cpusfell back to the logical count only when/proc/cpuinforeported no topology, not a degenerate one. Some hypervisors give every vCPU the same(physical id, core id)pair, which parses as exactly one core — so a 16-vCPU guest would have run at 1 thread instead of 8, silently, and slower than the code this replaces. A count implying more than two threads per core is now treated as a floor rather than believed. Reproduced in a test before fixing.sherpa-onnx-core, and numpy arrives viafaster-whisper → onnxruntime. The guarantee held only because theenginesextra is monolithic. That extra now declaresnumpyoutright.app.audiokeeps its stdlib fallback, since a core install genuinely has no numpy.543 tests pass.