feat: add local LLM text post-processing - #177
Draft
jatinkrmalik wants to merge 5 commits into
Draft
Conversation
Member
Author
|
/build |
|
⏳ PR Build started for Build signed & notarized DMG... this usually takes 10–20 minutes. |
|
✅ PR Build ready!
📥 Install
SHA-256 checksum
|
jatinkrmalik
force-pushed
the
jmalik/feat/local-llm-polish
branch
from
July 9, 2026 20:25
524cfa4 to
f5223ae
Compare
Member
Author
|
/build |
|
⏳ PR Build started for Build signed & notarized DMG... this usually takes 10–20 minutes. |
|
✅ PR Build ready!
📥 Install
SHA-256 checksum
|
Member
Author
|
/build |
|
⏳ PR Build started for Build signed & notarized DMG... this usually takes 10–20 minutes. |
|
✅ PR Build ready!
📥 Install
SHA-256 checksum
|
jatinkrmalik
force-pushed
the
jmalik/feat/local-llm-polish
branch
from
July 23, 2026 00:11
a3d314c to
f164f65
Compare
Run llama.cpp in single-turn mode, parse only the assistant reply from chat output, and tighten the setup probe so model preparation only succeeds after a valid response. Switch the default rewrite model to Gemma 3 1B because it behaved more conservatively and reliably in local repros.
Replace human-oriented llama CLI output parsing with a private Unix-socket llama-server session and JSON chat-completions responses. Preserve existing runner settings by resolving adjacent server binaries and add an opt-in real-model integration test for the complete process lifecycle.
jatinkrmalik
force-pushed
the
jmalik/feat/local-llm-polish
branch
from
July 23, 2026 00:35
f164f65 to
4bed40f
Compare
Member
Author
|
/build |
|
⏳ PR Build started for Build signed & notarized DMG... this usually takes 10–20 minutes. |
|
✅ PR Build ready!
📥 Install
SHA-256 checksum
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an opt-in local AI post-processing pass after Whisper transcription and before text insertion.
llama-serverruntime, including migration from previously storedllamaorllama-clipaths-hf, then enables rewrite only after a valid probe responseRuntime design
VocaMac starts
llama-serverfor each post-processing request on a unique private Unix socket under/tmp. It waits for the health endpoint, sends separate system and user messages to the OpenAI-compatible/v1/chat/completionsendpoint, decodeschoices[0].message.content, and shuts the server down after the response.This replaces the earlier
llama-clitranscript parsing approach entirely:Production notes
llamaandllama-clipaths resolve the adjacentllama-serverbinary automatically.Model choice
In local tests for conservative transcript cleanup, Gemma 3 1B followed the rewrite instructions more reliably than Qwen3 1.7B, so Gemma is now the default. Qwen remains available as an explicit larger option.
Related Linux issue: VocaHQ/vocalinux#408
Validation
swift test(236 tests, 2 skipped, 0 failures)VOCAMAC_RUN_LOCAL_LLM_TEST=1 swift test --filter LocalLLMPostProcessorTests/testLocalLlamaServerIntegrationWhenEnabledggml-org/gemma-3-1b-it-GGUF:Q4_K_MQwen/Qwen3-1.7B-GGUF:Q8_0llama-serverprocess running