Skip to content

architecture.md

Latest commit

 

History

History
217 lines (185 loc) · 12 KB

architecture.md

File metadata and controls

217 lines (185 loc) · 12 KB
Raw

Architecture

Component boundary

target app text field
  ↕ UITextDocumentProxy
vocaphone keyboard extension
  ↕ atomic App Group JSON + revision numbers
vocaphone containing app
  ↕ bearer-authenticated HTTP/HTTPS through LAN, VPN, or reverse proxy
FastAPI gateway (VocaHQ/vocagateway submodule at gateway/)
  on macOS or Linux (native or multi-architecture container)
  → bounded temporary audio → FFmpeg mono 16 kHz WAV
  → TranscriptionEngine adapter → VocaMac, Handy, MLX Audio, WhisperKit,
                                  sherpa-onnx, faster-whisper, Moonshine,
                                  or whisper.cpp

The gateway implementation and its ops docs live in vocagateway; this repository vendors a pinned revision under gateway/.

The App Group record is the source of truth. Polling is a wake-up strategy, not the data store. Audio references are opaque filenames; tokens, transcripts, and absolute paths are never written to ordinary logs.

SessionRecord.processingLocation is optional and additive. It is how the keyboard and the Live Activity name the place transcription is happening without asking the app, and its absence is a real state — a record written before the field existed, or a session interrupted before the app claimed it. Every reader answers that with neutral wording ("Transcribing") rather than guessing a route, and version-1 records without the field continue to decode unchanged. Nothing on the gateway wire format changed.

Recorded request flow

  1. The keyboard creates a UUID session and atomically writes launchingApp.
  2. If a nonexpired Quick Dictation marker exists, the already-running app sees the request while its background input is active. Otherwise the keyboard opens vocaphone://dictate?session=<uuid> after a short fallback delay.
  3. The app validates the session, claims it, and resolves which speech-to-text route the session will take — onDevice or gateway — writing that processingLocation into the record before any audio moves. It then switches its persistent audio input from discarding buffers to writing a WAV recording, and writes recording plus bounded meter updates. The audio graph is not rebuilt between dictations.
  4. The user manually returns to the original app.
  5. Finish changes shared state to finalizing.
  6. The app negotiates streaming support on the authenticated WebSocket itself, avoiding a separate health round trip. With a ready Moonshine engine, copied float32 buffers reach the streaming endpoint while the app still writes the complete WAV. Batch-only engines receive a structured unsupported response.
  7. The app stops recording and uses the stream result when available. Otherwise it creates the idempotent session and runs the normal upload/batch flow.
  8. The app writes readyToInsert and deletes its audio only after success.
  9. The keyboard verifies its session context, persists inserting, calls insertText, then persists inserted and completed.

After Finish, the app can rearm a Quick Dictation window without tearing down its AVAudioEngine. The window length is a preference — 10 minutes, 20 minutes, or "until I close vocaphone", which takes a short lease the standby heartbeat keeps renewing so a killed process cannot leave a marker that never expires. The same input tap writes buffers only while a dictation is active and deliberately discards every standby buffer. The shared availability file contains only activation and expiry timestamps. It is cleared before active recording, on expiry, on audio failure, when the user turns the feature off, and when the Live Activity's Pause button ends the current window. Pausing sets a flag that the next foreground clears; only the Settings toggle is durable.

Persisting inserting before touching the document intentionally favors avoiding duplicate text if the extension terminates at the worst moment.

Typing intelligence (iOS keyboard)

Completion, correction and next-word prediction run entirely inside the keyboard extension. Nothing about them touches the gateway, the App Group session record, or the network.

keystroke ─▶ WordComposer ─▶ TypingEngine ─▶ TypingCandidates ─▶ TypingStripView
                  ▲               │
     documentContextBeforeInput   ├─ UITextChecker (system dictionaries)
        (reconcile only)          ├─ UILexicon (the user's own replacements)
                                  ├─ TypingWordList (shipped, frequency-ordered)
                                  └─ LearnedWords (App Group, capped at 2 000)

Three constraints shape the design:

  1. There is no composing region. UITextDocumentProxy has no marked-text API, so replacing a word is n × deleteBackward() plus one insertText, applied in a single run loop turn and never while a dictation insertion is in flight. WordComposer is the keyboard's own record of the current word; documentContextBeforeInput only ever reconciles it, because that window is bounded and can be nil while the keyboard loads.
  2. UITextChecker is main-actor. The SDK marks it NS_SWIFT_UI_ACTOR, so it cannot be pushed onto a background queue. Keystrokes stay unblocked by ordering instead: text is inserted synchronously on touch-up and the computation is enqueued as a separate main-actor task, which a generation counter cancels if the user has typed on, and a 64-entry LRU cache usually skips entirely. TypingWordList provides a pure-Swift fallback that can leave the main actor if device measurement ever demands it.
  3. The extension has a hard memory ceiling. One checker, one word list, a bounded cache and a capped learned-word store.

The word list, the bigram table, the emoji catalog and the emoji suggestion table live once at assets/keyboard/ in the repository root. The iOS keyboard target references that directory from project.yml; the Android build merges the same directory through sourceSets in app/build.gradle.kts. Two hand-maintained copies would drift, and nothing would notice until the platforms started suggesting different words.

Preview harness (iOS)

Every user-visible state has a #Preview, and every preview is built from VocaPhoneApp/App/Previews/. The point is not tidiness: states like "gateway reachable but token rejected", "model failed its integrity check" and "transcript ready but the field changed" are expensive enough to reach on a device that they were never looked at.

Three pieces:

  • PreviewFixtures — canned SessionRecords, SetupStatus combinations, transcription sources, a transcript library, and named UserDefaults stores. Stored values go into the registration domain, which UserDefaults keeps in memory, so opening a canvas cannot rewrite the settings of the app installed on the same simulator.
  • PreviewHost and PreviewMatrix — the environment a screen needs, and the four variants every screen has to survive: default, dark, .accessibility5, and right-to-left.
  • Preview initializers on RecordingCoordinator and LocalModelManager. Both are live objects whose designated initializers touch audio, the network, the keychain and the shared container; the preview ones assign state and stop. Both types carry an isInert flag that makes their side-effecting entry points return early, because a canvas is live — without it a home preview's .task would overwrite the fixture with the real system state within a frame, and Download in a model preview would fetch a gigabyte.

The keyboard's surfaces are all UIViews, so they preview through KeyboardViewPreview in the keyboard target, beside the views themselves.

The #if DEBUG boundary is enforced, not assumed. ios/tools/check-preview-isolation.py fails if a preview-only file has code outside #if DEBUG, or if a preview-only symbol is named from release code. It runs in just ios ci and in the iOS workflow. just ios release-build compiles with DEBUG undefined and is the wider version of the same check.

Server states

created → uploaded → transcribing → completed

Failures move to failed while retaining original audio for retry. Repeating session creation or finishing a completed session returns the same job/result.

Engine boundary

TranscriptionEngine exposes health() and transcribe(path, options), while engines that can identify model files also expose best-effort warmup. HandyEngine can reuse Handy's selected downloaded model, WhisperKitEngine runs Core ML folders through one managed loopback-only WhisperKit service on Apple silicon, and WhisperCppEngine is the portable CLI fallback. VocaMacEngine covers the other optional desktop app: VocaMac exposes no headless transcription command, so instead of driving the app it reads the model chosen in VocaMac's preferences, rejects incomplete downloads the way VocaMac's own asset check does, and hands the Core ML folder and VocaMac's tokenizers to WhisperKitEngine. Both desktop apps are optional and Mac-only — Handy needs macOS, VocaMac needs Apple silicon — so app/system.py holds one table of per-engine host requirements that drives the WebUI picker contents, the label shown beside each engine, and the 422 rejection when a host cannot run the selected engine. The service keeps the selected model resident and falls back to the one-shot CLI when an older WhisperKit build cannot serve. FasterWhisperEngine owns one persistent CTranslate2 model and uses CPU INT8 by default. SherpaOnnxEngine owns one portable INT8 recognizer for SenseVoice, Parakeet, GigaAM, Canary, or a streaming Zipformer model, dispatching on the selected model's catalog model_type for both loading and decoding. MLXAudioEngine keeps one Apple-silicon-native model in unified memory. MoonshineEngine owns one persistent transcriber. Any engine can expose the guarded /v1/stream path by implementing the StreamingEngine protocol (app/models/base.py): supports_streaming, streaming_lock, and create_stream() returning an object with add_listener/add_audio/stop. Currently Moonshine's streaming architectures and sherpa-onnx's streaming Zipformer model do; SherpaOnnxEngine wraps sherpa-onnx's separate OnlineRecognizer/OnlineStream API in an adapter presenting that same surface, so the WebSocket handler itself has no engine-specific code. Every other model — including every other sherpa-onnx model — uses the same upload fallback as other batch engines. No engine-specific field is part of the stable session API response.

The default Docker image includes OpenBLAS whisper.cpp, sherpa-onnx, faster-whisper, and Moonshine and persists /data as a volume. Compose also provides host-native CPU, NVIDIA CUDA, and Vulkan images. MLX Audio, WhisperKit, VocaMac, and Handy remain native-macOS-only.

The gateway keeps privacy-safe operational counters in process memory: uptime, active and queued transcriptions, completed/failed/rejected counts, stage-level latency, real-time factor, and peak process memory. These counters reset on restart and never contain audio, transcripts, session identifiers, or model input text.

Liveness (/health/live) is independent of the transcription engine. Readiness (/health/ready) uses a five-second cached engine probe and returns 503 when the selected model cannot transcribe. Startup schedules a best-effort filesystem prefetch for the selected model while the HTTP process remains available.

Deployment boundary

The native macOS gateway can use Apple-platform engines. The container is a Linux process with persistent CPU engines and optional GPU-specific images; Docker Desktop cannot pass the macOS MLX, WhisperKit, or Core ML runtime into that container. Both deployments expose the same API, WebUI, health semantics, and persistent model-selection behavior.

The canonical container project is gateway/compose.yaml. It publishes host loopback by default, mounts /data as the only persistent application volume, and supplies the bearer token through a Compose secret.