Voice via mobile browser: ASR + TTS on iOS Safari + Android Chrome

[kovtcharov]

Goal

Make voice input + output work reliably in mobile Safari + Chrome browsers — the path users hit when they reach GAIA via the tunnel from their phone. Without this, the mobile-via-tunnel story is text-only.

Why this matters for consumer adoption

The morning-brief / voice-research / daily-companion use cases all assume voice. Telegram covers async voice (#889 + voice notes). The tunnel covers sync mobile access. Voice via the tunnel is the connector — "I'm holding my phone, I open GAIA, I tap mic, I speak" — and mobile browser voice has known platform quirks.

Scope (single PR, v0.18.2 or v0.19.0)

A. Audit current state

• Test the existing Agent UI mic button on:
  • iOS Safari 17+ (mobile)
  • iOS Chrome / Edge (uses WebKit on iOS — same engine, different policies)
  • Android Chrome
  • Android Firefox (different engine)
• Document what works, what's broken, what's flaky
• Output: short audit committed to PR description

B. Known mobile-browser pitfalls to address

• MediaRecorder MIME types — iOS Safari historically restricted to audio/mp4; codecs=mp4a.40.2; Android Chrome prefers audio/webm;codecs=opus. Need codec detection + fallback.
• Microphone permission UX — first-tap should request permission with a clear in-app explanation; rejected permission must show actionable recovery (Settings → Safari → Microphone)
• Background tab handling — recording stops when tab goes background on iOS; document expected behavior
• TTS playback autoplay restrictions — mobile Safari blocks autoplay without user gesture; first agent voice response must be triggered by the same gesture that sent the message
• Lock-screen / silent mode — on iOS, audio output respects silent switch; explicit volume + audio session config helps

C. Mobile-specific UX improvements

• Push-to-talk button (long-press to record, release to send) — better than tap-to-start/tap-to-stop on mobile
• Visible recording indicator + audio level meter
• Transcript preview before sending (gives confidence the ASR worked)
• Voice playback controls per agent message (play/pause, speed)

D. Backend coordination

• Confirm Whisper ASR endpoint accepts the codecs mobile browsers actually emit
• Confirm Kokoro TTS output format works in mobile browsers (<audio> element or Web Audio API)
• Streaming TTS chunks if available (faster perceived latency than wait-for-full-response)

E. Tests

• Playwright Agent UI: Playwright + Strix Halo E2E suite (validate every UI feature on real AMD hardware) #883 — add mobile-browser voice scenarios: tap mic → record → assert transcript appears → assert TTS audio element fires play event
• Manual smoke test matrix: 4 browsers × 3 use-case skills
• Doc: docs/guides/voice-on-mobile.mdx with troubleshooting for permission issues

What this is NOT

• ❌ Not Telegram voice notes (Telegram messaging adapter (Phase 0) — pull forward from v0.23.0 #889 scope expansion)
• ❌ Not a native iOS / Android voice integration
• ❌ Not the broader voice-first parity work in v0.21.0 (Voice-first interaction: parity between voice and text as P0 enabling technology #702) — this is specifically the mobile-browser path

Acceptance criteria

• Tap mic → speak → transcript appears within 3 seconds on iOS Safari + Android Chrome
• Agent voice response plays automatically in response to the tap that sent the message
• Permission rejection shows actionable recovery message
• Push-to-talk works smoothly on touch
• Playwright mobile voice scenarios in Agent UI: Playwright + Strix Halo E2E suite (validate every UI feature on real AMD hardware) #883 pass

Attribution / prior art

• MDN MediaRecorder docs — codec compatibility tables
• WebKit blog — iOS Safari audio policies

Dependencies

• Adjacent: feat(ui): friendly ngrok tunnel diagnostics + cookie auth for mobile #872 (tunnel — entry path), M1 (PWA shell), M2 (responsive UI for the mic button), Agent UI: Playwright + Strix Halo E2E suite (validate every UI feature on real AMD hardware) #883 (Playwright)
• Reuses: existing Whisper ASR (src/gaia/audio/whisper_asr.py) and Kokoro TTS (src/gaia/audio/kokoro_tts.py)
• Adjacent: Voice-first interaction: parity between voice and text as P0 enabling technology #702 (voice-first parity v0.21.0) — this is the mobile-browser leg of that broader work

[kovtcharov]

Spec — Issue #896: Voice via mobile browser (M3)

Depends on:

• mobile_design_system.md — §7 (voice
  button placement, states, PTT default), §6 (touch hierarchy), §9
  (a11y).
• Mobile-responsive Agent UI: touch targets, single-pane layout, virtual keyboard #895 (M2) — needs <TouchTarget> and the redesigned chat input row
  to land first; voice button slots into bottom-right of that row.

Touches:

• Frontend: src/gaia/apps/webui/src/components/mobile/VoiceButton.tsx (new)
  • supporting hooks.
• Backend: src/gaia/audio/whisper_asr.py (codec/MIME accept-list),
  src/gaia/ui/routers/ (new /api/voice/transcribe endpoint or
  extension of an existing one).
• Docs: docs/guides/agent-ui.mdx — note mobile voice support.

---

1. Goal

Voice input works in iOS Safari and Android Chrome through the M1
tunnel, with the same Whisper backend already used by gaia talk.
Push-to-talk on mobile, click-to-toggle on desktop, all behind one
<VoiceButton> component placed per design-system doc §7.

Success = a mobile user can hold the mic button, speak, release, and
see the transcribed message land in the input (or auto-send,
configurable). Permission denials are recoverable. iOS Safari and
Android Chrome both work.

---

2. Codec detection + fallback

Browsers expose different MediaRecorder MIME types. Pick the best
supported one at runtime:

// src/gaia/apps/webui/src/utils/voice/codec.ts (new)
const PREFERRED_MIME_TYPES = [
  'audio/webm;codecs=opus',  // Android Chrome, desktop Chrome/Firefox/Edge
  'audio/mp4',               // iOS Safari 14.5+ (mp4a/AAC)
  'audio/webm',              // generic webm fallback
  'audio/ogg;codecs=opus',   // Firefox older versions
];

export function pickSupportedMimeType(): string | null {
  if (typeof MediaRecorder === 'undefined') return null;
  for (const t of PREFERRED_MIME_TYPES) {
    if (MediaRecorder.isTypeSupported(t)) return t;
  }
  return null; // explicit null — caller surfaces error, no silent fallback
}

export function fileExtensionForMime(mime: string): string {
  if (mime.startsWith('audio/webm')) return 'webm';
  if (mime.startsWith('audio/mp4'))  return 'm4a';
  if (mime.startsWith('audio/ogg'))  return 'ogg';
  return 'bin';
}

No silent fallback if pickSupportedMimeType() returns null —
fail loudly per CLAUDE.md project rules. Show a friendly error in the
button's tooltip: "Your browser doesn't support voice input. Try
Safari 14.5+ or Chrome."

Backend accept-list

whisper_asr.py (or the new /api/voice/transcribe router) must
accept all of:

• audio/webm (opus and vorbis variants)
• audio/mp4 / audio/m4a (AAC)
• audio/ogg (opus)
• audio/wav (kept for gaia talk compatibility)

Whisper itself uses ffmpeg under the hood and handles all of these,
but the FastAPI router must not reject the upload at the multipart
boundary. Concretely: the router validates content-type against the
list above and rejects unknown types with HTTP 415 + a JSON error
naming the supported list (per CLAUDE.md "actionable errors" rule).

---

3. Permission UX flow

[Idle button tap/long-press] → navigator.mediaDevices.getUserMedia({audio:true})
                                       │
                ┌──────────────────────┼──────────────────────┐
                ▼                      ▼                      ▼
            granted              prompt-default            denied
          start recording       browser shows prompt    show recovery card

Recovery from rejection

If getUserMedia rejects with NotAllowedError (user clicked block):

1. Show a non-modal <BottomSheet> with:
   • Headline: "Microphone is blocked"
   • Body: "GAIA needs microphone access to use voice input. Tap
     the lock icon in your browser's address bar to allow."
   • Per-browser hint, detected from navigator.userAgent:
     • iOS Safari: "Settings → Safari → Microphone → Allow"
     • Android Chrome: "Tap the lock icon → Permissions → Microphone"
   • CTA buttons: "Try again" (re-prompts) and "Use text instead"
     (closes the sheet).
2. Persist the rejection in localStorage
   (gaia.voice.permissionDenied) so the button starts in a
   disabled-with-tooltip state on next visit, instead of springing
   another rejection on the user.
3. Tapping the disabled button shows the recovery sheet again — no
   silent failure.

If getUserMedia rejects with anything else (NotFoundError,
OverconstrainedError, AbortError), show the same sheet with the
specific error name and a link to the troubleshooting doc.

iOS-specific gotcha

iOS Safari requires getUserMedia to be called from a user gesture
on the same task tick. The PTT long-press handler must call
getUserMedia synchronously inside pointerdown, not after an
await. Document this in a code comment so a future refactor
doesn't break it.

---

4. Push-to-talk implementation

State machine

idle ──pointerdown──> permission-check ──granted──> recording
                            │
                            └──denied──> recovery-sheet (back to idle)

recording ──pointerup──> stopping ──blob ready──> uploading ──response──> idle
recording ──pointercancel──> stopping (discard) ──> idle
recording ──[duration > MAX_PTT_MS]──> stopping ──> idle (auto-send)

MAX_PTT_MS = 60_000 to prevent runaway recordings if the user's
finger gets stuck.

Long-press handler

Use pointerdown / pointerup / pointercancel (covers touch +
mouse + pen uniformly). Don't use touchstart/touchend — they
don't fire on desktop, and the handler should be the same code path
for both platforms.

// VoiceButton.tsx — sketch, not final
const onPointerDown = (e: React.PointerEvent) => {
  e.preventDefault();
  (e.currentTarget as HTMLElement).setPointerCapture(e.pointerId);
  startRecording();   // synchronous getUserMedia call inside
};
const onPointerUp = () => stopRecording({ send: true });
const onPointerCancel = () => stopRecording({ send: false });

setPointerCapture keeps the button receiving pointerup even if
the user's finger drifts off it — important on small touchscreens.

Audio level meter

Use AudioContext + AnalyserNode.getByteFrequencyData() to render
a 5-bar meter inside the button while recording. RMS over the bins,
clamped to [0, 1]. Update at 30Hz, NOT every frame — saves battery.
Respect prefers-reduced-motion by replacing the meter with a single
solid pulse.

Tap-to-toggle (settings opt-in)

A settings toggle "Hands-free voice" switches mode from ptt to
toggle. In toggle mode, tap to start, tap to stop. Useful for
desktop and accessibility (users who can't long-press). Default on
mobile is ptt; default on desktop is toggle.

---

5. Backend coordination

Endpoint contract

POST /api/voice/transcribe (new, in src/gaia/ui/routers/voice.py):

• Multipart form with one field: audio (the recorded blob).
• Optional query param: language (default en).
• Response: { "text": "...", "duration_ms": 1234, "model": "whisper-..." }.
• Errors:
  • 415 — unsupported MIME type, body lists supported types
  • 413 — payload too large (>10MB)
  • 503 — Whisper not available, body says how to fix
  • 500 — surfaces the underlying error with a correlation ID

The router instantiates the existing WhisperASR from
src/gaia/audio/whisper_asr.py. Do not add a fallback to a
smaller model or a "best effort" path — if the configured Whisper
fails, return 503 with an actionable message (per CLAUDE.md
fail-loudly rule).

Streaming vs. one-shot

For PTT < 60s, one-shot upload is fine and simpler. Don't add
chunked WebSocket streaming in M3 — it's a tempting rabbit-hole and
not needed for the stated UX. If/when transcription latency becomes
a complaint, file a follow-up.

Reuse gaia talk infra

WhisperASR already exists and is tested. M3's job is the web
endpoint + the React component, not a new ASR. Route both gaia talk and the web endpoint through the same singleton model loader
to avoid double-loading Whisper into memory.

---

6. Tests

Frontend (Playwright, mobile profiles from #883)

1. Codec detection: stub MediaRecorder.isTypeSupported to return
   only audio/mp4; assert the recorder uses audio/mp4.
2. Codec none: stub all codecs unsupported; assert the button
   shows the "browser unsupported" tooltip and is disabled.
3. Permission denied: stub getUserMedia to reject with
   NotAllowedError; assert the recovery <BottomSheet> appears
   with the iOS-vs-Android hint matching the UA.
4. Permission persisted: with localStorage.gaia.voice.permissionDenied
   set, assert the button starts disabled and tap reopens the sheet.
5. PTT happy path: mock MediaRecorder to emit a synthetic blob
   on stop; long-press the button, release, assert a POST to
   /api/voice/transcribe with the right Content-Type.
6. PTT cancel: start recording, fire pointercancel, assert no
   network request.
7. Reduced motion: assert audio meter renders as static pulse,
   not animated bars.
8. a11y: axe-core scan on the recovery sheet — zero violations.

Backend (pytest)

1. Upload audio/webm (opus) — 200 + plausible text.
2. Upload audio/mp4 (AAC) — 200 + plausible text.
3. Upload audio/ogg (opus) — 200 + plausible text.
4. Upload application/octet-stream — 415, error body lists
   accepted types.
5. Upload >10MB — 413.
6. With Whisper not loaded — 503, error body names the env var/CLI
   command to fix.

Manual QA

Real devices through the M1 tunnel:

• iOS 16.4 Safari on iPhone — long-press, speak, release, transcribe.
• Android 13 Chrome on Pixel — same.
• iOS 15 Safari (older) — graceful degradation if codec unsupported.

---

7. Acceptance criteria

• <VoiceButton> exists in components/mobile/, slotted into the
  chat input row per design-system doc §7.
• Codec picker chooses audio/webm;codecs=opus on Android Chrome
  and audio/mp4 on iOS Safari (verified on real devices).
• PTT works on both platforms; release sends the audio.
• getUserMedia rejection shows the recovery <BottomSheet> with
  a correct browser-specific hint.
• /api/voice/transcribe accepts all four MIME families and
  rejects others with 415 + actionable body.
• No silent fallbacks anywhere in the path — every failure has an
  actionable error string per CLAUDE.md.
• All Playwright tests in §6 pass on all mobile profiles.
• All pytest tests in §6 pass (with Whisper available) or skip
  cleanly (without).
• axe-core gate passes on the recovery sheet.
• Manual QA on iOS + Android passes.

---

8. Failure modes & mitigations

Failure	Likelihood	Mitigation
getUserMedia not called from user gesture on iOS Safari → silent fail	high if refactored	Code comment + Playwright test that intercepts the gesture timing
MediaRecorder.isTypeSupported lies on some Android OEMs	medium	Try the recorder; if start() throws, fall back to next codec; if all fail, surface the error (still no silent success)
Whisper cold-start latency hides the audio "sent" feedback	medium	Show "Transcribing…" toast immediately on stop; don't wait for response
User long-presses, button mounts inside a scroll container, scroll absorbs the pointer	medium	touch-action: none on the button + setPointerCapture (already in §4)
Tunnel adds 200–500ms RTT → PTT feels sluggish	low	Show optimistic local recording feedback immediately; latency is on the upload, not the UI
Multipart upload over 10MB on long PTT despite 60s cap	low	Whisper's input rate makes 60s ≈ 1MB; cap exists as belt-and-suspenders
Two voice buttons (e.g., voice in welcome + chat) compete for mic	low	Singleton useMicrophone() hook in hooks/voice/useMicrophone.ts enforces one active recorder at a time

---

9. Out of scope (track separately)

• TTS playback of assistant responses on mobile — separate issue;
  mobile autoplay restrictions need their own design.
• Wake-word / hands-free continuous listening — privacy and battery
  concerns, deserves a dedicated spec.
• Streaming partial transcripts — see §5; defer until UX justifies.
• Voice in standalone apps (apps/llm/, apps/jira/, etc.) — only
  Agent UI in M3.

---

10. References

• Design system: mobile_design_system.md §7
• Whisper ASR: src/gaia/audio/whisper_asr.py
• Talk SDK: src/gaia/talk/
• UI router pattern: src/gaia/ui/routers/
• M1 tunnel context: Issue Starter skill pack: 10 agentskills.io skills demonstrating the platform #893
• M2 chat input redesign: Issue Mobile-responsive Agent UI: touch targets, single-pane layout, virtual keyboard #895 (this issue's frontend depends
  on it)
• MediaRecorder API: https://developer.mozilla.org/en-US/docs/Web/API/MediaRecorder
• iOS Safari WebRTC support: https://webkit.org/blog/11353/mediarecorder-api/