[Feat]: On-device push-to-talk voice input (Whisper STT) - #786
Draft
a-ghorbani wants to merge 17 commits into
Draft
[Feat]: On-device push-to-talk voice input (Whisper STT)#786a-ghorbani wants to merge 17 commits into
a-ghorbani wants to merge 17 commits into
Conversation
Add whisper.rn, @fugood/react-native-audio-pcm-stream, and react-native-permissions for on-device voice input. Wire the iOS NSMicrophoneUsageDescription, the Android RECORD_AUDIO permission, and a microphone-permission helper (granted/denied/blocked mapping). whisper.rn and llama.rn both vendor a ggml fork under disjoint symbol namespaces (wsp_ggml_* vs lm_ggml_*). Under use_frameworks! :linkage => :static their framework header-copy phases collide on identically-named ggml headers, so force whisper-rn to static_library alongside the existing llama-rn static workaround.
Add the on-device speech-recognition service: per-tier Whisper model manifests (base q5_1, small q8_0, large-v3-turbo q5_0), a whisper.rn-backed engine with per-tier on-disk install + version sentinel (sentinel is the final download write so an interrupted install never reports ready), and an energy-VAD gate that rejects sub-floor buffers before the decoder. Whisper hallucinates on silence, so gating is load-bearing. Transcription runs entirely on-device; the only network call is the HuggingFace model download.
Add a MobX store mirroring the TTS subsystem: a memory-derived availability gate with a persisted boolean|null user override, per-tier download orchestration (reclaim before disk preflight, sentinel-gated install), and a transient push-to-talk capture state machine. The store never touches the LLM ModelStore. Initialize it at app boot alongside the TTS store.
Own the push-to-talk capture lifecycle: request mic permission, capture 16 kHz mono PCM bounded by a max-record cap, run the energy-VAD gate on release, transcribe on-device, and hand the transcript to a callback (never auto-send). Native capture is released on press-out, error, max-ms, app background, and unmount so the microphone is never leaked.
Add a push-to-talk mic button to the composer right-controls that self-gates on the availability gate, routes to Settings when the model is not installed, and appends the transcript to the composer (never auto-sent). ChatView gains an appendTranscript seam that appends to the current input text rather than overwriting it. Settings gains a voice-input toggle (memory-derived default with user override) plus per-tier model install/remove controls. Add a dedicated microphone icon and the en.json strings.
Cover the energy-VAD gate (silence/too-short rejection, PCM decode), the ASRStore availability gate + per-tier download + capture FSM, the whisper engine sentinel gating + offline transcribe path, the push-to-talk capture lifecycle (permission deny, VAD reject, gate-closed, release on unmount), and the composer mic button (self-gating, install routing, append-not-send). Add jest mocks for whisper.rn, the audio pcm-stream, react-native-permissions, and an ASR store mock.
Whisper model files (up to ~547 MB) under files/asr/ must not be uploaded to cloud backup (25 MB limit) or device-transfer snapshots, matching the existing tts/ exclusion.
Adds an __E2E__-gated pocketpal://asr?cmd=state::<...> host that forces the voice-input store into the mic-at-rest, not-installed, and low-memory gate states so a simulator visual-capture spec can screenshot them without a real microphone, model download, or low-memory hardware. Registers the ASR_AUTOMATION_STATE marker in the prod-bundle DCE grep.
Add voice-input append-not-overwrite seam tests, mic-permission mapping tests, and ASR settings-toggle tests; register the voiceInput locale section and align the asrStore mock setters with MobX runInAction.
- Treat native AudioRecord.stop() as void (the .d.ts Promise type is wrong); wrap in Promise.resolve so teardown/transcribe is not aborted. - Guard onPressIn against re-entrant presses, release during the permission dialog, and Android audio-init rejection. - Distinguish a blocked microphone permission from a plain denial. - Release the whisper context once transcription settles.
- Release the whisper context when the app backgrounds so it doesn't sit resident with the LLM (only 'background', not transient 'inactive'). - Auto-select a tier on successful download; reselect a remaining ready tier (or the default) when the active tier is deleted. - Surface the disk-space preflight shortfall on the download-error channel instead of silently reverting to not-installed.
Render a transient snackbar keyed off the ASR capture error so denied, blocked, too-short, and transcribe failures are visible. A blocked microphone permission offers an open-Settings action.
When a tier download is blocked by the disk-space preflight, the Settings row now renders how much space to free up alongside the retry control.
Add error-capture, error-blocked, and error-disk commands so the composer error snackbar, the open-Settings route, and the Settings insufficient-storage line can be captured without a real microphone. Stays behind the E2E gate.
The onPressIn entry guard bailed on any captureState other than 'idle', but 'error' persists until the snackbar's 4 s auto-dismiss. An immediate retry (the common 'too short' case) silently no-opped on a mic button that still looked enabled. Treat 'error' as a re-armable resting state; requesting_perm clears lastError, dismissing the stale error snackbar.
The insufficient-storage line showed the raw model size, but the download preflight requires estimated * headroom free, so freeing the shown amount still failed the gate. Add a single ASR_DISK_HEADROOM_FACTOR constant shared by the preflight writer and the Settings message, and display the headroom-inclusive figure. Also assert the fallback tier is not ready in the delete-reselect test.
Owner
Author
Independent review — APPROVE (2 rounds)A multi-lens independent review (security, mobile, performance, QA, architecture, data, UX, local-invariants) ran over two rounds. All findings are resolved; every lens approves. Round 1 → REQUEST_CHANGES. Substantive issues, since fixed:
Round 2 → APPROVE after a small regression in the re-entrancy guard was caught and fixed. Privacy/offline invariant verified throughout: no audio, PCM, or transcript leaves the device; the only network call is the model download. whisper.rn + llama.rn ggml forks coexist cleanly (disjoint Full test suite green (3719 passed). Remaining gate before merge: a manual on-device smoke of the mic paths — hold→speak→release inserts text, blocked-permission → Open Settings, and context-release-on-background — since the microphone can't be driven in a simulator. Generated by PocketPal Dev Team |
|
When is this feature going to be released? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds multilingual, fully on-device speech-to-text as push-to-talk voice input in the chat composer. Hold the mic button, speak, release — the transcribed text is appended into the input field for the user to review and send. Nothing is auto-sent. The whole path runs locally via Whisper; the only network access is the on-demand model download.
Addresses voice-input demand from #698 (Russian voice models / Whisper), #427 (download local TTS and STT to chat with a local LLM — hands-free / driving), and #655 (Add Voice Mode). Companion to the existing on-device TTS, completing the voice loop.
Feature
boolean | nulluser override over a one-timegetTotalMemory()check, surfaced as a "Voice input" toggle in App Settings with a below-threshold helper line on low-memory devices..mlmodelcdirectory doesn't fit the file-oriented GGUF candidate shape, and it would leak a non-chat entry into every LLM-model surface).NSMicrophoneUsageDescription(iOS),RECORD_AUDIOruntime permission (Android).whisper.rn / llama.rn ggml coexistence
The main technical risk was linking whisper.rn's ggml fork alongside llama.rn's. A
nm -Dscan confirmed the two forks use disjoint symbol namespaces — llama.rn exportslm_ggml_*(with zero bare C-ABIggml_*/whisper_*; the onlywhisper-substring symbols are llama.rn's own mangled CLIP/multimodal encoder), while whisper.rn's source fork useswsp_ggml_*. The prefixes cannot collide. The clean iOS Release build underuse_frameworks! :linkage => :staticrequired no symbol-namespacing fallback; the only Podfile change was forcingwhisper-rnto static linkage (same fix already applied tollama-rn) to resolve ause_frameworks!ggml-header copy collision.Native verification (NATIVE_CHANGES)
pod installclean;ios/Podfile.lockcommitted (whisper-rn 0.6.0,llama-rn 0.12.4)..socoexist; namespaced ggml, no collision).Tests
Full suite green: 252 suites, 3702 passing / 2 skipped, global coverage ~75% statements (≥ 60% threshold). New ASR modules:
energyVad,MicButton, constants and mic-permission mapping at 100%;usePushToTalk~88%;ASRStore~76%. Coverage spans the gate formula, capture FSM transitions, VAD silence rejection, permission grant/deny/blocked mapping, forced re-download on stale sentinel, append-not-overwrite composer seam, and the Settings toggle.A prod-bundle DCE check in CI asserts the E2E-only automation host used for deterministic visual capture is stripped from production builds.
Visual / manual caveat
Three at-rest UI states (mic at rest, gate-open-not-installed setup affordance, low-memory Settings toggle) are captured on an iOS simulator. The two microphone-dependent states (active recording, transcript appended) cannot be driven on a simulator — the simulator yields no audio frames, the same documented limitation that blocks E2E coverage of the Apple Shortcuts feature — so they are covered by unit tests and to be confirmed by manual on-device capture during physical-device verification.
Generated by PocketPal Dev Team