feat: expand speech model catalog and add native GGUF runtime - #53
Merged
Conversation
✅ Deploy Preview for voca-gateway canceled.
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
Mr-Sunglasses has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
There was a problem hiding this comment.
Mr-Sunglasses has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
There was a problem hiding this comment.
Mr-Sunglasses has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
The transcribe.cpp models in the catalog are unusable without a transcribe-cli the project never ships, but the Overview page listed only FFmpeg, whisper.cpp, and the Python engines. A missing binary showed up as a download that transcribed nothing. Probe the configured binary in detect_system and report it as a "transcribe.cpp CLI" tile, so it reads Missing with the build-and-set instructions until VOCAGATEWAY_TRANSCRIBE_BINARY or PATH resolves it.
A transcribe.cpp card offered a two-gigabyte GGUF download on a host with no transcribe-cli, and nothing on the card said so. The Overview panel knew, but only if the operator went looking before downloading. Collect the per-engine runtime checks into one table that both the panel tiles and the model entries read, so a card can never disagree with the panel about what is installed. A model whose engine is unavailable now carries a "needs <runtime>" badge and the install hint. The weights are still correct without the runtime, so Download stays available and the warning clears once the binary or package resolves.
The doc index advertised 58 models and 108 languages while the generated page listed 76 and 109. The count was already two catalog expansions stale, because nothing checked it — so pin both numbers to DEFAULT_CATALOG in the test that already guards the generated page, and move the additional-models link into the page table where every other doc lives. Also name the two literals a reader has to guess at: the "None" that Qwen3-ASR wants interpolated into its decoder prompt is now LANGUAGE_AGNOSTIC_DECODER rather than a bare string compared in the MLX adapter, and the transcribe.cpp entries use the language and licence constants their neighbours already use.
Correctness: - transcribe.cpp health() probed with shutil.which while the Libraries panel and the model cards used system.resolve_binary, which also expands `~` and accepts a file without the execute bit. The panel read Installed while every recording failed. Both now resolve through one documented probe. - transcribe.cpp resolved the language before checking readiness, so a host with no binary and no model answered "choose the spoken language" instead of "install transcribe-cli". Readiness comes first. - The MLX adapter passed `prompt` without the signature check it applies to `language`, so a build whose generate() lacks it raised TypeError on every recording. The prompt is also catalog data now rather than a key comparison. - _run_huggingface_download promoted a partial tree: it checked the remote listing but skipped the on-disk re-check both sibling download paths do, and installed() only looks for the marker file. - Batch results were matched by exact path in manifest order, so a CLI that canonicalises /var to /private/var, or finishes out of order, failed the recording with advice about length. They match by file name; a duplicate or absent chunk still fails. - _execute discarded stderr, so every failure was a bare exit code; it now reports the tail like whisper.cpp's sibling, turns exec OSError into EngineUnavailableError, and an unreadable WAV into a transcription error. - Selecting a model cleared only transcribe_model, and active_model_path returned the first field set rather than the running engine's, so a stale value made the Models tab mark the wrong card active and handed clients the wrong languages. One mapping now owns the rule for every engine. sherpa-onnx: the new per-stream language override applied to every offline model, silently pinning ones that auto-detect. It is scoped to Cohere, whose recognizer is now built with the requested language and rebuilt when it changes, so a wrong-language decode cannot pass as a fluent transcript. Docs: VOCAGATEWAY_ENGINE's value list omitted transcribe.cpp in README and configuration.md, and VOCAGATEWAY_TRANSCRIBE_BINARY was in no table. Also: model-type strings live in the catalog instead of being duplicated under a second name in the adapter, and the runtime table probes lazily without a dataclass that existed for one statement.
Two takes on the same warning landed on this branch: an aside shown on every transcribe.cpp card, and a badge plus note driven by whether the runtime is actually resolvable on this host. Together a card said the same thing twice, and the aside kept saying it after transcribe-cli was installed. Keep the host-driven one, which clears itself once the binary resolves and covers every engine rather than transcribe.cpp alone, and give it what the aside had that it lacked: the deployment.md link and the reminder that the runtime lives where the gateway runs, inside the container under Docker. DependencyStatus carries the URL so the link is catalog data, not markup. The Settings page keeps the aside: the engine picker offers transcribe.cpp before any model exists, so there is no card there to carry the warning.
There was a problem hiding this comment.
Mr-Sunglasses has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
Canary-Qwen 2.5B and Granite Speech 4.1 both have strong published WER, but that was never the binding constraint, and neither entry earns the runtime behind it: - TranscribeCppEngine is the only adapter that is not a MemoryResidentEngine. It reloads the model for every recording, which for dictation costs more than the accuracy it buys, and the commit before this branch was spent making models resident. - Neither model is recommended on any host, at any RAM size. - Neither streams, while 17 catalog models do. - Granite already ships twice through mlx-audio; only Canary-Qwen was unique. - Both entries are community Q5_K_M conversions whose quantization loss is unmeasured here, so the leaderboard numbers are not these weights. - The runtime is in no image and no extra: it needs a per-platform C++ build pinned to transcribe.cpp 0.2.3 and its -o / --batch / --batch-jsonl flags. That left the chunk splitter, the batch reader, a dependency tile, docs and a build guide serving two models nobody is steered toward. The per-card runtime warning stays and keeps its reach over every remaining engine, so a card still says when whisper.cpp, WhisperKit, sherpa-onnx, faster-whisper or MLX cannot run on this host.
There was a problem hiding this comment.
Mr-Sunglasses has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
Each of these was added by this branch, and this repository's own notes already argued against every one of them: - MLX Hindi to Roman Swift: "higher published error rates than Apex". - Hindi to Roman Prime Q5: better on Common Voice/FLEURS, worse on Indic-Voices, at twice Apex's size — "not an across-the-board upgrade". - MLX Whisper Small Hindi: publisher evaluates read Hindi only, and a gateway smoke test "produced repeated extra words". - MLX Srota and Srota Conversational: the conversational benchmark shares speakers between train and test, so it establishes nothing on unseen speakers. Both are 1.58 GB Apple-silicon-only fine-tunes of Qwen3-ASR 0.6B, while the official Qwen3-ASR 1.7B already covers Hindi and English in 1.61 GB. Apex Q5 stays: it predates this branch and is the baseline the two Roman-Hindi additions were measured against and lost to, so it remains the option for Hindi written in Latin script. Hindi is still covered by 23 models. The literal "None" decoder prefix went with Srota, its only user. The MLX adapter keeps decoder_language_code support, which whisper.cpp also implements for Apex; with no MLX model carrying the field now, its test pins the contract to a synthetic entry rather than to whichever model happens to have it.
There was a problem hiding this comment.
Mr-Sunglasses has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
…picks Twelve entries — Tiny, Tiny EN, Base and Base EN across faster-whisper, whisper.cpp and WhisperKit — were the catalog's weakest transcribers, and every one is beaten at its own size by something already here: Moonshine English Base is 141 MB against Whisper Base's 142-145 MB, and the 20M streaming zipformer is 44 MB against Tiny's 66-75 MB. Both were built for dictation rather than scaled down from it. They were load-bearing for recommendations, so the low rung is rebuilt rather than emptied. Below 8 GB there was never anything Apple-specific to offer -- the smallest WhisperKit and MLX entries both want 8 GB -- so the two branches collapse into one pair that suits either host: SenseVoice Small INT8 for multilingual, Moonshine English Base for English. At 8 GB on Linux, faster-whisper Base gives way to sherpa-onnx Parakeet TDT v2 INT8. Every RAM tier from 2 GB to 64 GB was checked to resolve to models that exist and fit. The Whisper `.en` pairing test now starts at Small, and the troubleshooting checklist no longer tells operators to reach for Tiny/Base. The OpenBLAS and thread-pinning benchmarks in deployment.md still name ggml-tiny.en: those numbers were measured with it, and renaming the subject would misreport what was run.
There was a problem hiding this comment.
Mr-Sunglasses has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
Member
Author
Catalog changes: 65 → 57 modelsTwo different kinds of change here, worth keeping separate. Removed from the existing catalog (12)The Whisper Tiny and Base tiers, across all three Whisper engines:
Each is beaten at its own size by something already in the catalog:
Both replacements were built for dictation rather than scaled down from a general model. These tiers were load-bearing for recommendations, so the low rung was rebuilt rather than emptied:
Every tier from 2 GB to 64 GB, on both host types, was checked to resolve to models that exist and fit. Withdrawn from this PR (7, never on
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The catalog lacks the researched Roman Hindi, mixed Hindi-English, and newer multilingual/English models. Several require decoder-specific language handling or a native runtime before their downloads can produce usable transcripts.
Summary
Add 11 revision-pinned models (76 total): Roman Hindi Swift and Prime Q5; Whisper Small Hindi; two Srota variants; batch and streaming Parakeet Unified INT8; Cohere Transcribe INT8; MLX and native Q5 Granite multilingual; and native Canary-Qwen Q5.
Preserve Roman Hindi as Hindi speech written in Latin letters. Keep Srota's mixed script separate, set Cohere's per-recording language, and prevent MLX Granite language hints from requesting translation.
Add an optional transcribe.cpp adapter with persisted selection, diagnostics redaction, timeout/cancellation cleanup, and bounded audio chunks decoded in one process. Validate every batch result before returning a combined transcript.
Restrict new MLX downloads to required runtime files and support sharded checkpoints. Document hardware, licensing, setup, and evaluation limitations without changing recommended defaults. Whisper Small Hindi remains experimental after repetition in a live check; private Orato weights are excluded.
Document native host requirements and pinned transcribe-cli installation for macOS/Linux, including service PATH and custom Docker requirements.
just doctorchecks the optional executable and its configured override. Python extras do not install this native tool.Show an always-visible transcribe-cli requirement notice with installation instructions on native model cards and in engine Settings. Clarify that model/Python downloads do not install it and Docker needs the executable inside the container.
Verification
just test: lint (including WPS), formatting, strict mypy, dependency/lock/audit checks, 589 tests, and Compose config pass.Generated model documentation check and
git diff --checkpass.Independent verification of all 11 new pin records: zero mismatches or skips.
All 11 entries exercised with public audio on Apple silicon; multilingual Cohere and both Granite variants checked in English and French. These are integration smoke checks, not comparative WER benchmarks or cross-device performance validation.
Built transcribe.cpp 0.2.3 at
e2f82cb6702315a1194f3bf1a6fee67cd2678447. Both native models recovered all 230 words in a synthetic 74-second repetition check after chunking.Docs updated for the separately installed native binary and model limitations.
Container build not applicable: Dockerfiles and lockfile are unchanged. The default image does not bundle transcribe-cli.
Rebuilt the pinned native CLI using the documented Release/static/embedded-Metal flags, copied it outside the build directory, verified required CLI options and system-only library links, and transcribed public audio with the installed copy on macOS.
just doctorchecked with the binary absent, a valid explicit override, and a missing override.UI follow-up: 43 admin tests, Ruff check/format, template render checks for native/non-native cards and Settings, and diff whitespace checks pass. Browser visual inspection was unavailable in this session.
Privacy and security