Skip to content

PLAN.md

Latest commit

 

History

History
345 lines (290 loc) · 44.4 KB

PLAN.md

File metadata and controls

345 lines (290 loc) · 44.4 KB
Raw

Shriflow — Wispr Flow alternative for macOS — Execution Plan v2

Hold a key → speak → LLM-polished text lands in whatever input you were using, in under a second. Local and cloud transcription, batch and realtime, minimal UI.

Framework (user-mandated): Native SDK (native-sdk.dev, vercel-labs/native v0.4.1, Apache-2.0) — UI in declarative .native markup, logic in Zig, compiled to a real native macOS binary. No Electron, no webview. All research below verified against live docs on 2026-07-10; items marked ⚠️UNCERTAIN get re-verified by the owning agent at build time.

How this plan is used: §0–§2 are shared context pasted to every coding agent. §3 contains one self-contained work order per phase — each is spawned as an Opus agent with max effort. §4 is the orchestration playbook (waves, gates, merge rules).

§0. Product spec (shared context — every agent reads this)

Core flow

idle ──key-down──► recording ──key-up / ✓──► transcribing ──► formatting ──► injecting ──► done
 ▲                    │ Esc or ✕ = cancel                                        │
 │                    ▼                                                          ├─ focused editable field → text inserted at cursor
 └────────────── HUD hides ◄── success flash / error state ◄────────────────────┴─ no valid target → clipboard + fallback card (Copy)
  • Two engines classes: Local (on-device whisper.cpp; later sherpa-onnx/Parakeet) and Cloud (BYO API key: Groq, Deepgram, OpenAI, Mistral, ElevenLabs).
  • Two delivery modes: Batch (default; paste once after speech) and Realtime (streaming partials; only finalized segments are typed into the field — never rewrite text already committed to the user's app). Realtime with no focused editable field at key-down → auto-degrades to batch for that utterance.
  • Formatting step: fast cloud LLM fixes grammar/punctuation/casing, strips fillers and self-corrections ("no wait, I mean"), applies dictated structure (lists, "new paragraph"). Never adds content. Output = text only. Skipped for ≤3-word utterances; hard 1.2 s timeout → paste raw transcript (speed is never held hostage). User-switchable model + Off.
  • Keybinds (rebindable, recorder UI): dictation key default Fn/Globe — hold = push-to-talk (release ends), quick tap (<300 ms) = toggle; Esc cancel; paste-last-transcript default ⌥⇧V. Fallback preset ⌥Space offered in onboarding.
  • Fallback card (matches screenshot): when injection has no target (no field focused, secure field, or AX failure) — dark card above the HUD pill: hint "Select a text field, then dictate", the transcript, a Copy button, ✕. Text is already on the clipboard when the card appears.

Latency budgets (each stage timed; telemetry in history + debug overlay)

Path Target p50 key-up→inserted Stage budget
Cloud batch (Groq turbo), 10 s utterance ≤ 900 ms finalize 30 → upload+STT 450–650 → LLM 250–350 → inject 80
Local batch (whisper turbo-q5, M-series) ≤ 1.2 s progressive chunk transcription during speech; only tail remains at key-up
Realtime partial lag ≤ 500 ms behind speech provider interim + render

Surfaces (minimal, cream/paper light theme + dark HUD)

  1. Main window — "Welcome back, {name}"; day-grouped history list (hover: copy / delete); right stat rail: total words · avg WPM · day streak (serif numerals); gear → settings.
  2. Settings — Transcription (Local|Cloud segmented; model dropdown; per-provider API-key fields → Keychain + Test button; local model manager with download progress), Formatting (provider+model, On/Off), Keybinds, Behavior (realtime toggle, launch at login, history on/off + clear), Language hint.
  3. HUD pill — bottom-center of the active screen, dark rounded pill ✕ | live waveform | ✓; states: recording / processing (spinner) / ✓ flash / ⚠ error. Must never steal focus.
  4. Fallback card — above the pill, as described.
  5. Tray — menu-bar app (Dock-hidden accessory): Start dictation, Copy last transcript, Settings, Quit.
  6. Onboarding — 3 permission steps (Microphone → Accessibility → Input Monitoring) with live status + System Settings deep links, then a playground text field.

§1. Verified research (2026-07-10)

Native SDK essentials

  • Scaffold: npm i -g @native-sdk/cli && native init shriflow · dev loop native dev (hot reload) · native check (ms-fast view validation) · native test (headless UI tests) · native build · native package --target macos.
  • Agent skills (mandated): npx skills add vercel-labs/native, plus pipe native skills get core --full | native-ui | automation into .claude/skills/…/SKILL.md. Every app embeds an automation server: native automate snapshot | widget-click | screenshot | wait — our agent-driven E2E loop.
  • App model: Model struct + Msg tagged union + update(); effects via .update_fx/.init_fx — "subprocesses, HTTP, files, timers flow through the effects channel and back into update as ordinary messages". Background threads/modules reach the loop via the bridge dispatcher (RuntimeOptions.bridge) / runtime.dispatchEvent ⚠️exact idiom from core --full skill.
  • Runtime API: createWindow(.{label,title,default_frame: RectF}) / listWindows / focusWindow / closeWindow (max 16), createTray/updateTrayMenu/removeTray (NSStatusItem; menu clicks → CommandEvent source=.tray), read/writeClipboard, set/get/deleteCredential (Keychain), showNotification, dialogs, openExternalUrl.
  • app.zon: id, name, display_name, version, icons, platforms, permissions, capabilities (gpu_surfaces, native_views, menus, shortcuts…), commands(≤256), shortcuts(≤64, in-window only), menus, url_schemes, security; windows via .windows or .shell view trees; titlebar standard|hidden_inset|chromeless. Validate: native validate app.zon, native doctor --strict.
  • Custom drawing: kind=.gpu_surface (Metal), present_mode=.timer for animated content (waveform), GpuSurfaceFrameEvent timing, runtime.supports(.gpu_surfaces).
  • Extensibility: extensions.Module (info + start/command/stop hooks) in a ModuleRegistry; custom-build examples ship build.zig → we link C libs/frameworks freely (Zig compiles C and Obj-C .m files natively).
  • Signing: native package --target macos --signing identity --identity "Developer ID Application: …" [--entitlements file] [--team-id]; adhoc mode for local; template assets/native-sdk.entitlements; notarize via xcrun notarytool … && xcrun stapler staple; zig build dmg. ⚠️Info.plist extra keys (NSMicrophoneUsageDescription) undocumented → post-process bundle with PlistBuddy in a build step if app.zon can't express it.
  • SDK gaps we fill in Zig (pure C APIs unless noted): global hotkeys w/ down+up (CGEventTapCreate), focused-element probe + text injection (AXUIElement*), mic capture (AudioUnit HAL), non-activating floating panel (NSPanel via small Obj-C shim — Risk 1), local STT (link whisper.cpp), accessory activation policy (NSApp.setActivationPolicy(.accessory) via shim).

Cloud STT (v1 set)

Provider Model Mode Price Notes
Groq ⭐batch default whisper-large-v3-turbo batch $0.04/hr, 10 s min bill 228× realtime; OpenAI-compatible POST https://api.groq.com/openai/v1/audio/transcriptions; flac/mp3/m4a/ogg/wav/webm; verbose_json word timestamps; 25 MB free-tier file cap; multilingual; no streaming
Deepgram ⭐realtime default nova-3 (+ flux-general-en end-of-turn model) WS streaming + batch ~$0.0077/min stream ⚠️ wss://api.deepgram.com/v1/listen, interim results ~300 ms, endpointing/utterance-end events; raw PCM16 accepted
ElevenLabs scribe_v1; Scribe v2 Realtime batch + WS $0.22/hr; $0.28–0.39/hr rt ~150 ms claimed rt latency (p95 ~250); 90+ langs
Mistral Voxtral Mini Transcribe 2 (v26.02); Voxtral Realtime (open-weights 4B, sub-200 ms) batch (+rt later) ⚠️~$0.001–0.002/min diarization, context biasing, ≤3 hr files
OpenAI gpt-4o-transcribe / gpt-4o-mini-transcribe / whisper-1 batch + rt WS ⚠️~$0.006/$0.003 per min the OpenAI-compat reference shape
Custom any OpenAI-compatible base URL batch — free coverage of Fireworks etc.

Cleanup LLMs (v1 set — all via one OpenAI-compat chat client)

Provider Model $/1M in/out Speed Role
Groq ⭐default llama-3.1-8b-instant 0.05/0.08 840 tok/s ~150-tok cleanup ≈ 250–400 ms; same key as Groq STT
Groq gpt-oss-20b 0.075/0.30 1,000 tok/s alt on same key
Cerebras gpt-oss-120b free tier ~3,000 tok/s quality+speed monster
Google gemini-2.5-flash-lite 0.10/0.40 fast, free tier budget alt (gemini-3.1-flash-lite = 0.25/1.50)
— Off — 0 ms raw passthrough

Local STT

  • v1 engine: whisper.cpp v1.9.1 (MIT; C API whisper.h; Metal first-class; built-in Silero VAD --vad). Models (ggml, Hugging Face ggml-org/whisper.cpp): large-v3-turbo-q5_0 ≈574 MB (default "Quality"), small-q5_1 ≈190 MB ("Balanced"), base.en-q5_1 ≈60 MB ("Light"). Progressive chunked transcription during speech → only the tail transcribes at key-up.
  • v1.5: sherpa-onnx (Apache-2.0, C API, prebuilt macOS arm64): Parakeet tdt-0.6b-v2 int8 offline + streaming Zipformer (true local realtime) + Moonshine + Silero VAD. Same abstraction, new backends.
  • Prior art validating stack: VoiceInk (GPL-3, whisper.cpp + Parakeet, KeyboardShortcuts lib, SelectedTextKit).

macOS mechanics (platform module)

  • Hotkeys: one session CGEventTap for keyDown/keyUp/flagsChanged. Fn/Globe = flagsChanged + kCGEventFlagMaskSecondaryFn; active tap consumes the event to suppress the system Globe action ⚠️(if macOS still triggers input-source switcher, onboarding instructs Keyboard → "Press 🌐 key to" = Do Nothing — the standard peer-app workaround). Needs Accessibility + Input Monitoring TCC.
  • Focus probe: AXUIElementCreateSystemWide → kAXFocusedUIElementAttribute; editable = role ∈ {AXTextField, AXTextArea, AXComboBox, web text roles} ∧ AXValue settable; secure = kAXSecureTextField subrole → never inject.
  • Injection router: ① paste-restore (default): save pasteboard → write text + org.nspasteboard.ConcealedType → synthetic ⌘V CGEventPost → restore old contents after ~300 ms; ② CGEventKeyboardSetUnicodeString typing in ~20-char chunks (terminals / paste-hostile apps; per-app override map); ③ AX kAXSelectedTextAttribute insert where reliable. Focus re-probed immediately before injecting.
  • Mic: AudioUnit HAL, pre-warmed at launch (instant start on key-down), 16 kHz mono f32→i16 ring buffer, RMS levels ~30 Hz to HUD.
  • Permissions: AXIsProcessTrustedWithOptions, IOHIDCheckAccess/IOHIDRequestAccess, mic via AVFoundation request; deep links x-apple.systempreferences:com.apple.preference.security?Privacy_{Accessibility,ListenEvent,Microphone}. Dev builds must sign with a stable identity (adhoc re-signs reset TCC grants).

§2. Architecture, contracts & conventions (frozen by P0; agents MUST NOT edit contracts.zig — propose changes to the orchestrating session instead)

shriflow/
  PLAN.md  app.zon  build.zig  assets/…
  .claude/skills/{native-core,native-ui,native-automation}/SKILL.md
  vendor/whisper.cpp/                 # pinned submodule, Metal ON
  src/
    main.zig            # app root: Model, update(), update_fx wiring
    contracts.zig       # ← single source of truth: types + Msg + module command API
    session.zig         # dictation state machine (P3)
    app.native          # markup: main window, settings, onboarding
    hud.zig             # HUD pill + fallback card views (P4)
    platform/  module.zig hotkeys.zig focus.zig inject.zig mic.zig panel.zig panel_shim.m permissions.zig
    engines/   stt.zig stt_openai_compat.zig stt_deepgram.zig stt_elevenlabs.zig stt_whispercpp.zig models.zig format.zig format_openai_compat.zig
    store/     settings.zig history.zig keys.zig
    telemetry.zig
  tests/               # native test suites per phase

contracts.zig (authoritative sketch — P0 lands the real file)

pub const AudioChunk   = struct { samples: []const i16, sample_rate: u32 = 16_000, ts_ms: u64 };
pub const FocusInfo    = struct { editable: bool, secure: bool, app_bundle_id: [:0]const u8, role: [:0]const u8 };
pub const InjectMethod = enum { paste_restore, type_unicode, ax_insert };
pub const InjectOutcome= enum { injected, no_target, secure_field, failed };
pub const EngineId     = enum { groq, openai, mistral, elevenlabs, deepgram, custom_compat, whispercpp };
pub const Transcript   = struct { text: []const u8, duration_ms: u32, engine: EngineId };
pub const Partial      = struct { text: []const u8, final: bool };   // streaming
pub const PermKind     = enum { microphone, accessibility, input_monitoring };
pub const PermState    = enum { unknown, denied, granted };

pub const Msg = union(enum) {
    // platform → app
    hotkey: struct { action: enum { down, up, tap, cancel }, },
    audio_level: f32,                       // RMS 0..1 @ ~30 Hz
    inject_done: InjectOutcome,
    permission: struct { kind: PermKind, state: PermState },
    // engines → app
    stt_partial: Partial,
    stt_done: union(enum) { ok: Transcript, err: EngineError },
    format_done: union(enum) { ok: []const u8, timeout_raw: []const u8, err: EngineError },
    model_download: struct { model: []const u8, pct: u8, done: bool, err: ?[]const u8 },
    // ui/store
    settings_changed: SettingsPatch, history_loaded: …, ui: UiMsg,
};

// Platform module command surface (dispatched via extensions dispatchCommand)
pub const PlatformCmd = union(enum) {
    start_mic, stop_mic,
    probe_focus,                             // → replies with FocusInfo msg
    inject: struct { text: []const u8, method: ?InjectMethod },  // → inject_done
    set_binding: Binding, // dictation key / paste-last
    hud: union(enum) { show_recording, show_processing, show_success, show_error: []const u8, show_card: []const u8, hide },
    check_permissions, request_permission: PermKind, open_settings_pane: PermKind,
    paste_last,
};

// Engine interfaces (vtable structs)
pub const SttEngine = struct { /* caps():SttCaps; transcribe(audio,opts,post:*PostFn) void; openStream(opts,post) ?*StreamSession */ };
pub const StreamSession = struct { /* pushAudio(AudioChunk); finish(); cancel(); */ };
pub const Formatter = struct { /* format(text, ctx: FormatCtx, post) void — owns skip-heuristic + 1.2 s timeout → .timeout_raw */ };

Conventions (all agents)

  • Zig 0.14+ style per repo zig fmt; errors as typed error unions — no panics on user paths; every fallible boundary returns a Msg error variant that the UI can render.
  • All engine/network work off the UI thread; results only enter the app through the bridge dispatcher as Msgs (idiom documented by P0 in contracts.zig header after reading core --full).
  • Allocation: arena per request/session; no leaks under native test (leak-check on).
  • Secrets only via runtime.credentials; never in settings JSON or logs. Audio buffers never touch disk (except opt-in debug).
  • Telemetry: every stage wraps telemetry.span(stage); spans land on the history record.
  • Each phase adds native test coverage + (where UI-visible) a native automate smoke script under tests/; native check must pass before an agent reports done.
  • Commit style: P<phase>: <what> — one phase per branch/worktree (see §4 merge rules).

§3. Phase work orders (each = one Opus agent, max effort)

P0 — Foundation & contracts (serial; everything waits on this)

Objective: compilable skeleton + frozen contracts so waves can run in parallel. Tasks:

  1. git init; commit this plan as PLAN.md.
  2. npm i -g @native-sdk/cli; native init shriflow (flatten into repo root); verify native dev renders.
  3. Install skills: npx skills add vercel-labs/native; mkdir -p .claude/skills/…; native skills get core --full > .claude/skills/native-core/SKILL.md; same for native-ui, automation. Read core --full and document in contracts.zig header: (a) the exact background-thread → Msg dispatch idiom, (b) window creation/show-without-focus behavior, (c) module registration boilerplate.
  4. app.zon: id com.shriflow.app, display_name Shriflow, icons placeholder, permissions/capabilities (clipboard, credentials, notifications, gpu_surfaces, native_views, menus), commands for tray, url_schemes = shriflow.
  5. Land src/contracts.zig (per §2, adjusted to real SDK idioms), src/main.zig with empty Model/Msg/update compiling, stub files for every module in §2's tree (compile-clean, TODO(P<n>) markers), build.zig with whisper.cpp submodule wired but behind a -Dlocal-stt flag (so wave-1 agents aren't blocked by its build).
  6. Tray with placeholder menu; accessory activation policy call (panel_shim.m: [NSApp setActivationPolicy:NSApplicationActivationPolicyAccessory]) proving the Obj-C shim compiles.
  7. tests/smoke.zig + automation script: app boots headless, snapshot returns the main window. Acceptance: native check + native test + native dev all green; skills present; contracts committed; every stub compiles; README with dev commands. Out of scope: any real feature logic.

P1 — Platform module (the hard 20%; unblocks P3)

Objective: all macOS system integration behind PlatformCmd, emitting contract Msgs. Tasks:

  1. hotkeys.zig: CGEventTap (keyDown/keyUp/flagsChanged); binding model {Fn} | {modifiers+key}; Fn via kCGEventFlagMaskSecondaryFn with event consumption; hold-vs-tap (<300 ms) disambiguation; Esc-while-recording → .hotkey .cancel; tap auto-re-enable on kCGEventTapDisabledByTimeout; binding-capture mode for the settings recorder (report next chord pressed).
  2. mic.zig: AudioUnit HAL input, pre-warmed engine; 16 kHz mono i16 ring buffer; start_mic/stop_mic; audio_level RMS @30 Hz; AudioChunk stream to registered consumer (STT); device hot-swap (default-device-changed listener).
  3. focus.zig: system-wide AX probe → FocusInfo (editable/secure/bundle-id/role); cache per-app quirks table (terminals → prefer typing).
  4. inject.zig: the three methods + router + verification (re-probe before inject; no_target/secure_field outcomes); pasteboard save/restore with ConcealedType; paste_last replay.
  5. permissions.zig: check/request all three + deep links; emit .permission on app-activate re-checks.
  6. panel.zig + panel_shim.m: HUD host — Path A: SDK chromeless window shown unfocused, then munge NSWindow.level=.statusBar, collectionBehavior=[canJoinAllSpaces,fullScreenAuxiliary,stationary], preventsActivation where possible; Path B (fallback, required if A steals focus even once in testing): real NSPanel(.nonactivatingPanel) hosting a CALayer-drawn pill (waveform bars from audio_level, ✕/✓ hit zones → PlatformCmd events). Bottom-center of the screen owning the frontmost window; multi-display correct. Decide A vs B in the first day and record the decision in PLAN.md.
  7. Debug harness window (dev-only) exercising every command with logs. Acceptance: manual QA checklist (documented in tests/platform-qa.md): Fn hold/tap/Esc all fire correct Msgs; recording starts <50 ms after key-down (log timestamps); paste lands in TextEdit/Chrome/Slack/Notes/VS Code; Terminal via typing; password field → secure_field; no-focus → no_target; clipboard restored byte-identical; HUD visible over fullscreen apps and never steals key status (verified: type in Chrome while pill shows). native test for pure-logic parts (binding disambiguation state machine, ring buffer). Out of scope: session logic, UI polish, engines.

P2a — Cloud engines (parallel)

Objective: all cloud STT + formatter clients behind SttEngine/Formatter. Tasks:

  1. Shared HTTP: SDK effects HTTP (or std.http fallback) with keep-alive + connection pre-warm on record-start; multipart builder; WAV encoder (16 kHz mono PCM16).
  2. stt_openai_compat.zig: Groq/OpenAI/Mistral/custom base URL; verbose_json; error taxonomy (auth/rate/net/format) → EngineError.
  3. stt_deepgram.zig: WS streaming client (nova-3; flux flag) — raw PCM16 frames, interim → .stt_partial{final:false}, speech_final/utterance-end → final:true; also batch REST path. WS client: minimal RFC6455 impl or vetted Zig lib — document choice.
  4. stt_elevenlabs.zig: scribe batch REST + Scribe-v2-realtime WS.
  5. format.zig + format_openai_compat.zig: system prompt (contract in §0; temperature 0.2; max_tokens 2×input); skip-heuristic; 1.2 s deadline → .timeout_raw; provider matrix Groq/Cerebras/Gemini-compat/custom.
  6. Key handling via store/keys.zig interface (stub until P2c; compile-time interface already in contracts). "Test key" command per provider (1 s micro-request).
  7. Mock-server-based native test suite: happy path, 401, 429, timeout, malformed JSON, WS interim ordering. Record real-latency fixture script (dev tool hitting real APIs when keys present, printing stage timings vs §0 budgets). Acceptance: tests green without network; with a real Groq key, fixture shows 10 s clip → transcript+cleanup within budget; Deepgram stream yields ordered partials with correct final flags. Out of scope: UI, session wiring, local engine.

P2b — Local engine + model manager (parallel)

Objective: whisper.cpp fully integrated; models downloadable in-app. Tasks:

  1. build.zig: compile vendored whisper.cpp with Metal (+Accelerate), -Dlocal-stt default ON from this phase; pin submodule @v1.9.1.
  2. stt_whispercpp.zig: context lifecycle (load once, keep warm); Silero VAD chunker; progressive mode: transcribe VAD-closed chunks during speech, concatenate, tail-only at finish; language hint; greedy decode params tuned for dictation; memory cap by model tier.
  3. models.zig: registry {turbo-q5_0 574 MB "Quality", small-q5_1 190 MB "Balanced", base.en-q5_1 60 MB "Light"}; HF resumable download (Range), SHA256 verify, progress Msgs, delete; storage ~/Library/Application Support/Shriflow/models/.
  4. Benchmark harness (dev cmd): RTF per model on this machine → stored, shown in settings. Acceptance: native test with a bundled tiny model fixture: JFK sample transcribes correctly; progressive mode: simulated 20 s stream → final text identical to one-shot (± whitespace); key-up→text for 10 s audio ≤ 1.2 s with small-q5 on dev machine; downloads resumable after kill. Out of scope: sherpa-onnx (v1.5), UI.

P2c — Store & telemetry (parallel)

Objective: settings, secrets, history/stats, latency spans. Tasks:

  1. settings.zig: versioned JSON at ~/Library/Application Support/Shriflow/settings.json; schema: engine selections, per-provider model, keybinds, mode, language, behavior flags; atomic writes; migrations.
  2. keys.zig: Keychain via runtime.credentials (service com.shriflow.app, account = provider id).
  3. history.zig: append-only JSONL (history.jsonl); record {ts, text, raw_text, app_bundle_id, duration_ms, words, engine, spans, outcome}; derived stats: total words, avg WPM (words/speech-minutes), day streak; retention toggle + clear-all; paste-last source.
  4. telemetry.zig: span API + per-session rollup; debug overlay data feed. Acceptance: native test: settings roundtrip + migration from v0; history 10k rows loads <50 ms; stats match hand-computed fixtures; keychain mocked interface conformance. Out of scope: UI rendering.

P3 — Session orchestrator (after P1 + P2a interfaces are real; the product's brainstem)

Objective: session.zig state machine wiring everything per §0's flow. Tasks:

  1. States idle/recording{batch|realtime}/transcribing/formatting/injecting/card/error; single active session; cancellation at every stage (Esc, ✕, hotkey re-press).
  2. Batch: key-down → HUD show + mic + (cloud: connection pre-warm); key-up → finalize → engine.transcribe → formatter → probe+inject → outcome (success flash | card) → history append (with spans).
  3. Realtime: probe at key-down — editable target? open stream: partials → HUD ghost text; finals → inject via router (typing method for increments); no target → silently run batch semantics; key-up → flush finals (+ optional whole-utterance cleanup OFF by default).
  4. Fallbacks: engine error → if cloud failed and a local model is installed + "offline fallback" enabled → retry locally once; formatter timeout → raw; every error → HUD error state with human message.
  5. Paste-last keybind → platform paste_last with history head.
  6. Debug overlay (dev flag): live stage timings vs budget. Acceptance: native automate E2E: scripted fake-engine run walks all states (snapshot assertions per state); real-run checklist: Groq batch into Chrome ≤ 900 ms p50 over 10 tries (spans logged); realtime into TextEdit streams finals only; no-focus run ends with card + clipboard set; cancel leaves no HUD residue, no stray audio. Out of scope: visual polish.

P4 — UI (parallel with P3 once P0 lands; consumes fake data until stores wire in)

Objective: all §0 surfaces in .native markup + Zig views, matching the screenshots' minimal aesthetic. Tasks:

  1. Design tokens: cream #FAF9F6 canvas, ink #1A1A1A, muted #8A8781, serif stat numerals, 8-pt spacing grid, pill dark #161618 @ 92% opacity, radius 999.
  2. Main window: header, day-grouped virtualized history list (copy/delete hover actions), stat rail, empty state.
  3. Settings views per §0 (segmented sections; provider key fields masked with Test button state; model manager rows with progress bars; keybind recorder using P1 capture mode; behavior toggles).
  4. HUD pill + fallback card contents (waveform bars ← audio_level, processing spinner, success/error states, card with Copy) rendered into P1's host (A or B — coordinate on the chosen path's drawing API: SDK gpu_surface vs CALayer spec handed to P1).
  5. Onboarding flow with live permission states + deep-link buttons + playground field.
  6. Tray menu final items. Acceptance: native automate snapshot suite per surface (all states, incl. HUD states via debug commands); native check clean; visual review screenshots attached to PR description. Out of scope: business logic (drive everything through Msgs/fixtures).

P5 — Hardening, packaging, QA (last)

Objective: shippable, signed, honest-to-budget build. Tasks:

  1. Latency pass on real hardware vs §0 table; optimize (connection reuse, model warm-up, chunk sizes); publish measured table in README.
  2. Injection QA matrix: Chrome, Safari, Arc, Slack, Discord, Notes, Mail, VS Code, Cursor, Terminal, iTerm, secure fields, IME (at least one CJK layout) — record per-app method matrix into focus.zig quirks table.
  3. Permissions lifecycle: fresh-machine onboarding run-through; signature-stable dev cert documented; Sequoia/Tahoe permission-nag behavior checked.
  4. Packaging: icons, native package --signing identity + entitlements (mic), PlistBuddy step for usage strings if needed, notarization + staple + DMG; auto-update wiring per SDK /updates.
  5. Full native test + automation suite green in one command (./scripts/verify.sh); crash/edge sweep (mic unplug mid-dictation, wifi drop mid-upload, 5-min dictation, rapid key spam). Acceptance: signed notarized DMG installs on a second machine; all QA checklists green; budgets met or regressions documented with cause.

§4. Orchestration playbook (how we spawn the fleet)

Waves (all coding agents = Opus, max effort, worktree-isolated):

  1. Wave 0: P0 (one agent, serial). Gate: repo builds + contracts frozen.
  2. Wave 1 (5 parallel): P1, P2a, P2b, P2c, P4. Each gets: §0–§2 + its own §3 order + instruction to load the three native-sdk skills first. P1 must report its Path A/B panel decision immediately (P4 depends on the drawing API).
  3. Wave 2: P3 (one agent) once P1 + P2a merge; P4 integration PRs land alongside.
  4. Wave 3: P5 after M2.

Gates: M1 = Fn-hold → Groq → cleaned text in Chrome ≤ 1 s (post-P3). M2 = local whisper + fallback card + settings complete. M3 = realtime mode. M4 = signed DMG.

Merge rules: one worktree/branch per phase; only the orchestrating session merges; contracts.zig changes require orchestrator sign-off; every merge runs ./scripts/verify.sh (native check + test + automation smoke).

Verification loop for agents: build → native automate snapshot/widget-click/screenshot against the running app → assert → iterate. No agent reports done with red checks or unmet acceptance criteria; each ends with an adversarial self-review pass of its own diff.

⚠️ Budget prerequisite: today's research agents all died on the account monthly spend limit. Raise it before Wave 1 (5 concurrent max-effort Opus agents + a long P3/P5 tail), or the fleet stalls mid-phase.

§5. Defaults chosen (say the word to change any)

  1. Name/bundle: Shriflow / com.shriflow.app.
  2. Hotkey: Fn hold/tap (fallback preset ⌥Space); paste-last ⌥⇧V.
  3. v1 providers: Groq (STT+LLM defaults), Deepgram (realtime), OpenAI, Mistral, ElevenLabs, Cerebras, Gemini, custom OpenAI-compat endpoint.
  4. v1 local: whisper.cpp (turbo-q5 default, small/base tiers); sherpa-onnx + Parakeet + local streaming in v1.5; local formatter LLM deferred (cloud or Off in v1).
  5. Realtime commits only finalized segments; per-segment cleanup off by default.
  6. History on by default, 100% local, toggle + clear.
  7. Signing assumes an Apple Developer ID exists (needed for stable TCC + distribution) — confirm.

§6. Risks

  1. HUD focus-steal — dual path planned (P1 task 6), decided day one. Highest technical risk; everything else is well-trodden.
  2. Fn suppression quirks per macOS version → documented workaround + alternate binding.
  3. Info.plist usage strings via app.zon unknown → PlistBuddy post-process fallback (P0 spike, P5 owns).
  4. native-sdk v0.4 churn → pin CLI+SDK versions in repo; isolate SDK touchpoints behind thin wrappers.
  5. Effects/threading idiom partially undocumented → P0 resolves from core --full skill before Wave 1; contracts encode the answer.
  6. Spend limit → §4 prerequisite.

§7. Sources

native-sdk.dev: /windows /capabilities /extensions /tray /skills /packaging /packaging/signing /app-zon /state /runtime /native-surfaces /app-model /keyboard-shortcuts · github.com/vercel-labs/native · console.groq.com/docs/speech-to-text · groq.com/pricing · developers.deepgram.com (models overview) · elevenlabs.io/realtime-speech-to-text, /pricing/api · docs.mistral.ai (audio) · inference-docs.cerebras.ai · ai.google.dev/gemini-api/docs/pricing · github.com/ggml-org/whisper.cpp v1.9.1 · github.com/k2-fsa/sherpa-onnx · github.com/Beingpax/VoiceInk · Apple developer docs (CGEventFlags, AX attributes).

Decision log (updated as phases land)

  • P0 (done, 2026-07-10): scaffold ejected (native eject) so build.zig can link the Obj-C shim + ApplicationServices/CoreGraphics/AudioToolbox/IOKit and, behind -Dlocal-stt, whisper.cpp. SDK pinned via committed symlink vendor/native-sdk-cli (absolute-target → resolves from any worktree). contracts.zig frozen; the Msg union lives in main.zig (markup dispatch needs flat tags there) with the contract-required variants already present as no-op arms. Verified: zig build, zig build test, native check, and a live automation smoke (boot → tray installed → view asserts → clean stop) all green.

  • Wave-1 file ownership: only P4 edits src/main.zig + src/app.native; P1/P2a/P2b/P2c stay inside their own files/dirs and report any main.zig needs in their final summary.

  • P1 — HUD Path A/B decision (2026-07-10): PATH B. The HUD is a real non-activating NSPanel created and drawn by the shim (panel_shim.m), not the SDK-declared hud window of Path A. Rationale: (1) focus-steal is the #1 risk (§6) and an NSPanel with NSWindowStyleMaskNonactivatingPanel + canBecomeKeyWindow == NO + .statusBar level + collectionBehavior {canJoinAllSpaces, fullScreenAuxiliary, stationary, ignoresCycle} cannot become key/main by construction — no reliance on munging an SDK-owned window; (2) Path A needs Options.windows_fn/window_view wired in src/main.zig, which P1 does not own (frozen against P1 edits) — Path B is entirely inside P1's files (panel.zig + panel_shim.m), host AND drawing, matching the P1 work order and needing zero coordination with P4/P3; (3) the pill (✕ | waveform | ✓), spinner, success/error, and fallback card are one AppKit drawRect fed from theme.zig tokens (via exported Zig adapters — single source of truth) and the tested placement math in panel.zig. Cost: the HUD is invisible to native automate snapshots/screenshots (those cover SDK gpu_surface views only), so HUD visual + focus-steal QA is manual (tests/platform-qa.md). P4's planned "draw HUD contents into P1's host" is therefore unnecessary — P1 owns HUD drawing end to end; P4/P3 talk only PlatformCmd.hud.

  • Wave-1 launch BLOCKED (2026-07-10): account monthly spend limit is hit — every subagent (research agents earlier, then the P1 worktree agent) dies instantly with "You've hit your monthly spend limit." The 5-agent parallel fleet cannot run until the limit is raised at claude.ai/settings/usage. Main session is unaffected, so phases that verify fully headlessly (P2c store, most of P2a, P4 UI, P2b logic) are being built directly in the main worktree meanwhile; P1 (needs real TCC grants + mic + event-tap hardware testing) waits for the fleet + a human at the keyboard.

  • Toolchain note (from the dead P1 agent's partial output): bare native build --release errors (optimize-mode .any hard-errors in the native_sdk dependency's own build.zig). Use zig build (what we use), native build, or native build --release=safe|fast. The app source compiles clean.

  • P2c (done, 2026-07-10): store & telemetry built + verified in the main session (fleet still blocked). settings JSON with defensive parsing + migration hook; Keychain wrapper with redaction; history JSONL + streaming stats (words/WPM/streak); telemetry spans + budget overlay. 45/45 tests green.

  • Status after this session: P0 ✅, P2c ✅ (both verified, committed). Remaining P1/P2a/P2b/P4 are the parallel-fleet work — blocked on raising the monthly spend limit (claude.ai/settings/usage). P1 additionally needs the user's Mac for TCC grants + event-tap/mic testing.

  • Wave-1 integration COMPLETE (2026-07-10): all five implementation phases merged to main and verified together — P4 UI, P2a cloud engines, P2b local whisper (+Metal), P2c store/telemetry, P1 platform. Merged tree: 153/154 tests (cloud-only) / 155/157 (-Dlocal-stt), native check clean, app boots into onboarding correctly. Environment note: a mid-session OS filesystem-lock/EPERM incident killed the first fleet run; recovered on its own; the resumed/relaunched agents finished cleanly (concurrent builds ≤3 were fine).

  • HUD = Path B (P1's call): a real non-activating NSPanel drawn by panel_shim.m, never-key by construction. Consequence: the HUD is invisible to native automate (manual QA only), and P4 does NOT draw HUD content — P1 owns it end to end; everyone else talks PlatformCmd.hud.

  • Resolved integration seams: stt.zig whispercpp test assertion gated on -Dlocal-stt; formatter honors the frozen (no-*Effects) vtable via a worker-thread watchdog (P3 may instead drive fx.fetch using the exported buildChatBody/SseAccumulator); mic/CoreAudio reached via dlopen (no build.zig change). DownloadSink lives in models.zig (mirrors SttSink).

  • P3 wiring contract (from P1's report): hold a platform.Platform; start() on boot (main thread); registerDeliver for the Bridge fast path; repurpose the poll_tick timer to repeating contracts.poll_interval_ms calling drain(); route every PlatformEvent through the session state machine; write transcript to clipboard BEFORE PlatformCmd.hud{.card}; replace P4's fake open_perm/recheck_perms/quit_app/test_key stubs with real platform/store/engine calls.

  • P3 (done, 2026-07-11): the session orchestrator is wired — the app is now feature-complete. session.zig holds the testable core (thread-safe engine bus + STT/formatter sinks, per-utterance Session with bounded audio accumulation + spans + settings snapshot + seq-guard, pure pipeline decisions); main.zig holds the runtime singletons + boot + the repeating poll_tick drain loop + the batch pipeline (hotkey → mic → STT via engineFor + cached key → formatter/skip → inject → outcome→success|clipboard+card → stats) + offline local fallback + cancel/paste-last, and makes the tray/permission/key stubs real. Boots clean (dispatch_errors=0), 159/160 tests, native check green. M1 is code-complete but needs the user's Mac + a Groq key to verify live (real hotkey/mic/inject + TCC grants).

  • P3 → P5 follow-ons (documented, non-blocking): (1) the 15ms drain poll rebuilds the idle view ~66×/s — replace with P1's dispatch_async fast-path deliver (needs a *Runtime/window handle captured in main) + a slow fallback poll; this is the top perf item. (2) history JSONL persistence via fx.writeFile + settings load-at-boot + Keychain key load (only in-memory key cache today). (3) realtime finals-injection (batch is complete; streaming partials are ignored). (4) native automate assert races the rebuild churn → verify.sh reads snapshot.txt directly.

  • Changes 1–4 (done, 2026-07-11): owner-requested post-M1 changes, planned by Fable, built by 2 parallel Opus agents, merged + verified. (1) Cleanup formatter default → Vercel AI Gateway (openai/gpt-oss-20b at https://ai-gateway.vercel.sh/v1; Groq llama-3.1-8b-instant one-key alt; Gemini ids refreshed). (2) All realtime/streaming removed — DeliveryMode/Partial/StreamSession/SttCaps/openStream + Deepgram/ElevenLabs engines + net_ws.zig/local_vad.zig deleted; cloud STT = Groq/OpenAI/Mistral + local whisper; batch push-to-talk only; settings schema stays v1 (defensive reader). (3) HUD redesigned — 210×48 dark pill, ~10 thick rounded equalizer bars with rolling-max ripple, gray ✕ circle / white ✓ circle (panel_shim.m draw + pure tested math in panel.zig). (4) All dummy/seed UI removed — 0/0/0 stats, empty "No dictations yet" history, honest not-installed downloads + boot install-scan + real P2b downloader wired (models.download on a detached thread → dlSink seq 0; .download case hoisted above the stale-seq guard). Merged main green: cloud 135/137, -Dlocal-stt 137/140, native check clean, automation smoke confirms empty-state + Gateway option + no realtime + dispatch_errors=0. HUD visual + real download + live gateway latency need the owner's Mac.

  • Changes 1–4 Fable review (2026-07-11): reviewed the C1–C4 diff at xhigh; confirmed C2 surgery clean (no dangling removed-symbols, vtable consistent, freeOwned correct), C3 HUD math clean, C1 correct (gpt-oss-20b fits Buf(64), 401→raw never blocks paste). FIXED: (1 MAJOR) the download dlSink shared round-robin stamp slot 0 with every 64th dictation → dropped transcript + stuck session + data race — now a dedicated EngineBus.dl_stamp; (3 MAJOR) startModelDownload had no in-flight guard → double-click spawned two workers corrupting the .part — added if dl_state==.downloading return; (6 MINOR) keyless install paid a doomed ~1.2 s formatter round-trip per dictation — kickoffFormat now injects raw when the key is empty. All green after fixes.

  • Deferred hardening from the review (top follow-ons): (5, most important — CORE) net_http.Transport ignores timeout_ms → a hung cloud STT upload sticks the session at .transcribing until Esc (audio lost); (2) same root cause makes a network-dropped model download hang at .downloading forever (recovery: relaunch → .part resumes) — proper fix is real read/connect deadlines in net_http (+ optionally a cancel button and cancel-on-quit). (4) failed download discards de.err (reverts to available; no user-facing reason). (nits) validate a 206 Content-Range vs requested offset; boot install-scan runs 3 stats on the loop thread; stale comments (local_whisper "progressive", contracts "WS", models.verifyFile unused param).

  • Persistence + history fix + notch HUD (done, 2026-07-11): three owner-reported issues from live testing, planned via explore/plan agents + live-web notch research, built in-session. (1) Persistence wired — the tested-but-unwired P2c stores now run: settings.json loads synchronously in boot() (drives route/bindings/placement pushes) + saves via fx.writeFile on every settings mutation; history.jsonl's byte mirror (g_hist_bytes, 512 KB cap, whole-line eviction) is the source of truth — the visible ring + stats derive from it on every change (boot/dictation/delete/clear), so display always matches disk; API keys live in the macOS Keychain (owner's choice) via SecItem helpers in the shim + a duck-typed keychain.Runtime plugged into the existing keys.Store; hydrateKeychain restores g_keys + the "•••• last4 saved" UI at boot. (2) History display fixed — root cause was a zero-capacity history: [0]ui.HistEntry + a stats-only recordHistory (stats counted words; rows could never exist). Now [200]ui.HistEntryOwned with owned text, newest-first render, day-group/time labels computed at render from ts_ms via localtime_r (Today rolls to Yesterday honestly). (3) Notch HUD — new default recording overlay: pure-black wings (opaque #000, square top / 14 pt bottom corners, no shadow, level mainMenu+3) extending the physical housing on the built-in display, 5-bar equalizer in the left wing, spinner/✓ there, card/error as a black dropdown below the notch (18 pt bottom corners, Copy/✕ live); no chips — Esc cancels, release confirms, all other clicks pass through. Geometry is pure tested Zig (hasNotch/notchRect/notchLayout/resolveMode, notch APIs are @available(12.0)-gated, built-in found via CGDisplayIsBuiltin); mode resolved per show; screen-change observer re-anchors (250 ms debounce). Settings → Behavior → "Recording indicator" (Auto | Notch | Bottom) persisted + pushed via new PlatformCmd.set_hud_placement; Bottom keeps the old pill byte-identical (owner: external displays stay bottom-pill). Smokes are hermetic now (scratch $HOME) since the app persists real state. Verified: full verify.sh green, live two-boot round-trip (onboarding_done + hud_placement survive relaunch), history-from-disk renders Today/Yesterday groups + stats, dispatch_errors=0. Notch visuals need the owner's notched Mac (tests/platform-qa.md updated).

  • ce1a4ae Fable review (2026-07-11, max scrutiny): no criticals; byte-mirror memory ops, lifetimes, notch geometry/unpacking, markup enum dispatch, bottom-pill regression, and hermetic smokes all verified correct. FIXED (this commit): (M1 MAJOR) stats used a fixed 1024-row buffer → total_words silently froze once history outgrew it — now STREAMS StatsBuilder over every line; (M2 MAJOR) ring eviction memmoved ~234 KB per line past 200 on the loop thread at paste time — now a modulo slice-ring that full-parses only the newest 200; (M5) a crash-torn history.jsonl tail got appended onto, merging records — loadHistory truncates the mirror to the last complete line; (M3) a save racing an in-flight same-key write was rejected + only logged → latest state could miss disk — .rejected now sets a dirty flag the 15 ms drive tick re-issues; (M7) save_key claimed "saved" even when the Keychain write failed — persists first, only marks saved on success (failure shows unsaved + failed test chip); (M6) row Copy served the ring's 1024-byte display copy — now re-parses the full transcript from the JSONL line; (M4) settings.zig header falsely claimed tmp+rename atomic writes — corrected (atomic writes = follow-on); dead newestText removed. RIDING (documented): non-atomic fx.writeFile (M4 proper), boot Keychain reads can prompt on ad-hoc re-signs (M8, in platform-qa.md), delete-by-ts picks the first duplicate (M9, unreachable live), >512 KB external history file loads empty then is overwritten (M10), UTC-vs-local streak day seam, per-frame label recompute (P5 view-rebuild item).