feat: Custom prompts for transcription context

[jatinkrmalik]

Allow users to provide initial prompt text to WhisperKit to improve transcription accuracy for domain-specific vocabulary:

• Add a text field in Settings for custom prompt
• Useful for technical jargon, proper nouns, specific terminology
• WhisperKit supports initial prompt via DecodingOptions

[m13v]

We implemented exactly this in our transcription pipeline. A few things from our experience.

The DecodingOptions.promptTokens approach works well, but there is a subtle issue - you need to tokenize the prompt yourself using WhisperKit's tokenizer, not just pass raw strings. The API expects token IDs, not text. Call whisperKit.tokenizer?.encode(text:) to get the token array, and make sure to handle the case where the tokenizer is nil (it can be nil if the model is not fully loaded yet).

For the Settings UI, we found a tag-based input works better than a plain text field. Users want to add terms one at a time and easily remove them. A comma-separated text field works too but the UX is worse when the list gets long.

Important constraint: WhisperKit's prompt token budget is around 224 tokens. With English text that is roughly 50-100 words depending on term length. We show a token count next to the input field so users know when they are hitting the limit. If you exceed it, WhisperKit silently truncates from the beginning, which means your most important terms might get dropped.

The format that works best for the prompt: prefix with "Glossary: " followed by comma-separated terms. So it looks like "Glossary: Kubernetes, kubectl, etcd, CoreDNS, Istio". This gives Whisper enough context to understand these are domain terms without confusing it about what kind of audio to expect.

One enhancement worth considering: combine this with screen context. If you can read the active window title or some text from the screen via AXUIElement, you can automatically seed relevant terms without the user having to type anything.

[m13v]

Our transcription service with prompt token support and vocabulary conditioning: https://github.com/m13v/fazm/blob/main/Desktop/Sources/TranscriptionService.swift

For the screen context approach (reading active window text via AXUIElement to auto-seed domain terms): https://github.com/mediar-ai/mcp-server-macos-use/blob/main/Sources/MCPServer/main.swift

[jatinkrmalik]

@m13v Are you an AI bot? Are you trying to farm backlinks to your project? I'm not sure.

[m13v]

haha no, just a dev who's been building voice transcription stuff for a while. I'm working on fazm - a macOS desktop agent with WhisperKit integration - so I've hit a lot of the same prompt tokenization issues you're dealing with in vocamac. figured I'd share what actually worked for us since most of the whisper docs don't mention the token budget gotcha

[jatinkrmalik]

Additional request from a user (relayed from our feedback channel):

On my personal iPhone I occasionally use Wispr Flow, and one of its advertised features is a dictionary of words. I wonder if VocaMac has such a thing on its roadmap? I want my manager's name "Namrata" in there, so it doesn't write "Normada", "Nam orda", etc.

This is the same ask as this issue, a custom vocabulary / dictionary so proper nouns and names stop getting mangled by the model. Adding a +1 with a concrete use case (personal/coworker names), which is probably the most common trigger for this pain point.

Starting implementation now, seeding the user-supplied terms into WhisperKit as prompt context.

[jatinkrmalik]

Tested on local! Working well, you can grab the test build from #169.

This will ship official in next version bump: v0.7.0