Custom vocabulary (promptTokens) makes Large v3 Turbo return empty transcripts

[flesler]

Summary

With custom vocabulary enabled, Large v3 Turbo (Compact) (openai_whisper-large-v3-v20240930_turbo_632MB) often returns empty text (Result: ... / “Transcription produced no usable text”). The same audio works on other models (e.g. Distil Large v3 Turbo), and clearing vocabulary makes Large v3 Turbo work again.

This looks separate from the silent-PCM / warm-engine issue in #176 / #178 — here the mic path is fine; WhisperKit returns no usable text when promptTokens are set on this turbo variant.

Repro

1. Set custom vocabulary (e.g. GraphQL, HTML, Git, AWS, MongoDB, Zod, Jira, UTF-8, JSON)
2. Load Large v3 Turbo (Compact)
3. Dictate normally
4. Observe empty transcriptions
5. Clear vocabulary (or switch to Distil Large v3 Turbo) → works again

Likely cause

VocaMac passes vocabulary to WhisperKit as promptTokens (with usePrefillPrompt). There is a known WhisperKit issue where promptTokens + turbo/distilled models hit the firstTokenLogProbThreshold gate and abort as “no speech”:

• WhisperKit: Using promptTokens causes the Transcription to return empty result. argmaxinc/argmax-oss-swift#372
• Related fix discussion: Fix prompt tokens causing empty transcription output argmaxinc/argmax-oss-swift#428

VocaMac 0.7.0 depends on WhisperKit ≥0.9.4; may not include a fix yet.

Environment

• VocaMac 0.7.0
• Apple M5 Max
• Model: openai_whisper-large-v3-v20240930_turbo_632MB
• Custom vocabulary non-empty

Notes

Not proposing a PR yet — not sure the right fix is small (bump WhisperKit vs disable promptTokens on turbo vs expose decode thresholds). Happy to test builds.

[jatinkrmalik]

Thanks @flesler, this is a really useful report.

I traced this through VocaMac and I agree with your read: in 0.7.0, custom vocabulary is encoded as WhisperKit promptTokens, and we enable usePrefillPrompt when those tokens are present. Given that clearing vocabulary makes the same mic/audio path work again, this looks separate from #176 / #178 and much more like the upstream prompt-token decode issue.

I also checked upstream. The original prompt-token issue is still open, #428 has not been merged, and there is now a newer upstream PR, argmaxinc/argmax-oss-swift#497, that root-causes this as early-exit checks firing during prompt prefill. That PR specifically mentions verification against openai_whisper-large-v3-v20240930_turbo_632MB, which matches your affected model.

So I do not think exposing decode thresholds is the right first fix on our side. The likely short-term VocaMac fix is either to disable custom-vocabulary prompt tokens for affected models, or better, retry once without prompt tokens when WhisperKit returns an empty transcript while vocabulary was enabled. That keeps dictation working while preserving vocabulary hints for models where they behave correctly.

For now the workaround is what you found: clear custom vocabulary for Large v3 Turbo (Compact), or use a model that does not reproduce this for you, such as Distil Large v3 Turbo. I’ll keep this open and would love to take you up on testing a build once I have a small mitigation ready.

[jatinkrmalik]

@flesler, we have a candidate fix in #182 and a signed, notarized test build: download the DMG. Could you try it with Large v3 Turbo (Compact) and custom vocabulary enabled, using the same audio that reproduced the issue? Let us know whether it returns text.

[flesler]

Verified on the #182 build — details on the PR: #182

Short version: with Large v3 Turbo + custom vocab, every real dictation hit empty prompted decode then recovered via the no-vocab retry. One short (0.7s) still blanked after retry — that was silent PCM (#176), not this bug.