feat(ios): iOS client with QR pairing, E2E-encrypted tunnel, and push-to-talk#1420
Draft
senamakel wants to merge 8 commits into
Draft
feat(ios): iOS client with QR pairing, E2E-encrypted tunnel, and push-to-talk#1420senamakel wants to merge 8 commits into
senamakel wants to merge 8 commits into
Conversation
Introduces `src/openhuman/devices/` — the Rust core pairing domain that lets iOS clients connect to the desktop core over the tinyhumans backend tunnel. Includes: - `crypto.rs`: X25519 keypair gen + XChaCha20-Poly1305 frame sealing/opening with sliding-window replay protection (5 unit tests: round-trip, tamper, replay, version mismatch, DH parity). - `store.rs`: SQLite persistence for paired devices using the existing `with_connection` pattern (4 unit tests). - `tunnel_client.rs`: emit_register / emit_connect / emit_frame helpers over the global SocketManager; one-shot ack registry for tunnel:registered. - `rpc.rs`: devices_create_pairing / devices_list / devices_revoke handlers with in-memory PENDING_KEYPAIRS / PENDING_SESSIONS / PEER_STATUS maps (3 unit tests). - `bus.rs`: DeviceTunnelSubscriber — resolves tunnel:registered acks, updates PEER_STATUS on peer-status events, completes X25519 handshake on tunnel:frame, persists PairedDevice and publishes DevicePaired. - `schemas.rs`: controller schemas + registered controllers following the cron/schemas.rs pattern (8 unit tests). - Wired into src/core/all.rs (build_registered_controllers + build_declared_controller_schemas + namespace_description). - Added DomainEvent variants DevicePaired / DeviceRevoked / DevicePeerOnline / DevicePeerOffline / DeviceTunnelFrame / DeviceTunnelRegistered. - Added tunnel:peer-status / tunnel:frame / tunnel:registered / tunnel:evicted dispatch in socket/event_handlers.rs. - Added x25519-dalek = "2" to Cargo.toml (chacha20poly1305 already present).
…2 of iOS PR) Part A — Rust follow-ups from Layer 1: - Encrypt pending-pairing private keys in SecretStore (enc2: ChaCha20-Poly1305), persist via PERSISTED_KEYPAIRS, drop on devices_revoke. - Implement sealed handshake frame in bus.rs: version 0x01 || eph_pub(32) || nonce(24) || ciphertext+tag; backward-compat plaintext fallback for pre-L2. - Publish DomainEvent::DeviceRevoked from devices_revoke. - Register DeviceTunnelSubscriber at startup (register_device_tunnel_subscriber called from jsonrpc.rs bootstrap path). - Expose LAN RPC URL from OPENHUMAN_CORE_RPC_PORT env (defaults to 7788). Part B — TS transport refactor: - app/src/lib/tunnel/crypto.ts: generateKeypair, deriveSharedSecret, seal/open (XChaCha20-Poly1305), sealHandshake/openHandshake, ReplayTracker. Uses @noble/curves/ed25519 x25519 and @noble/ciphers/chacha xchacha20poly1305. - app/src/lib/tunnel/framing.ts: chunk, Reassembler, TokenBucket (100 fps burst). - app/src/services/transport/CoreTransport.ts: interface (call/stream/isHealthy/close). - app/src/services/transport/LocalTransport.ts: wraps existing local HTTP path. - app/src/services/transport/LanHttpTransport.ts: HTTP to LAN rpcUrl. - app/src/services/transport/TunnelTransport.ts: socket.io E2E encrypted relay. - app/src/services/transport/CloudHttpTransport.ts: HTTP to cloud core. - app/src/services/transport/TransportManager.ts: races LAN vs Tunnel on iOS. - app/src/services/transport/profileStore.ts: localStorage desktop backend with iOS stub (TODO Layer 5). - app/src/services/coreRpcClient.ts: setActiveCoreTransport() override point; dispatch through active transport when set; local HTTP path unchanged. - Tests: crypto (27 tests), framing (8 tests), TransportManager (7 tests), TunnelTransport (4 tests). 1885 total, 0 regressions.
Adds a Devices settings panel that lists paired phones and a Pair iPhone modal that calls devices_create_pairing, renders a QR code with the openhuman://pair?cid&pt&cpk&rpc&exp URL, and polls devices_list to detect handshake completion. - New DevicesPanel + PairPhoneModal under app/src/components/settings/panels/ - Settings nav extended with a "Devices" entry - qrcode.react dependency for SVG QR rendering - Poll-based pairing detection (2s interval) + 3s auto-close on success - Expiry handling with regen flow
Replace the desktop-only compile_error! guard with proper
#[cfg(not(target_os = "ios"))] / #[cfg(target_os = "ios")] gating so
the Tauri shell compiles for aarch64-apple-ios without any CEF, CDP,
or core-sidecar symbols being in scope.
Desktop behaviour is completely unchanged. iOS gets a minimal WRY-based
Tauri builder with tauri-plugin-barcode-scanner registered (needed for
Layer 5 QR pairing UI). A TODO(Layer 6) marker is left for
tauri-plugin-ptt.
Modules gated as desktop-only (not(target_os = "ios")):
cdp, cef_preflight, cef_profile, core_process, core_rpc,
dictation_hotkeys, discord_scanner, fake_camera, file_logging,
gmessages_scanner, imessage_scanner, mascot_native_window,
meet_audio, meet_call, meet_scanner, meet_video, native_notifications,
notification_settings, process_kill, process_recovery, screen_capture,
slack_scanner, telegram_scanner, webview_accounts, webview_apis,
whatsapp_scanner, window_state
iOS-only: tauri-plugin-barcode-scanner
Shared (both platforms): app_quit
Other changes:
- app/src-tauri/Cargo.toml: [target.'cfg(target_os = "ios")'.dependencies]
with tauri-plugin-barcode-scanner = "2"
- app/src-tauri/capabilities/ios.json: new iOS capability file
- app/src-tauri/tauri.conf.json: add bundle.iOS block with frameworks +
minimumSystemVersion
- app/package.json: add tauri:ios:init / tauri:ios:dev / tauri:ios:build
scripts using stock @tauri-apps/cli (not the vendored CEF CLI)
- main.rs: gate the cef_entry_point macro to desktop; add plain fn main()
for iOS
Verification:
cargo check --manifest-path app/src-tauri/Cargo.toml → PASS
cargo check --manifest-path app/src-tauri/Cargo.toml
--target aarch64-apple-ios → fails only on
C build-script deps (aws-lc-sys, ring, objc2-exception-helper) that
require the iphoneos SDK — no errors from our Rust source files.
Prerequisite: rustup target add aarch64-apple-ios +
Xcode with iOS platform (xcode-select --install + iOS SDK).
pnpm compile → PASS
Adds the iOS-only React app surface: platform detection, routing branch, QR pairing screen, and full-screen mascot chat screen wired through the existing TransportManager and profileStore. - lib/platform.ts: isIOS detection (userAgent + isTauri()) with setTestPlatform / clearTestPlatform test hooks - AppRoutes.tsx: top-level iOS branch -> AppRoutesIOS (/pair + /mascot) - AppRoutesIOS.tsx: minimal iOS routes with profile-gated redirect - App.tsx: gate SocketProvider and desktop chrome on !isIOS; split AppShellDesktop from AppShellIOS to respect rules-of-hooks - pages/ios/PairScreen.tsx: QR scan via tauri-plugin-barcode-scanner, openhuman://pair URL parsing + expiry check, X25519 keypair generation, ConnectionProfile save, TransportManager health probe -> navigate /mascot - pages/ios/MascotScreen.tsx: full-screen YellowMascot + scrolling chat transcript, text input, disabled PTT placeholder (Layer 6 hook point), Disconnect button that clears profile and returns to /pair - services/transport/profileStore.ts: real iOS backend (pragmatic interim: localStorage, app-sandboxed WKWebView); replaces stub from Layer 2; SECURITY TODO comment for Layer 7 Keychain migration - package.json: add @tauri-apps/plugin-barcode-scanner@^2 Tests: 27 new Vitest tests across platform, PairScreen, MascotScreen, profileStore (desktop + iOS paths). Full suite: 1937 passing, 0 regressions.
Introduces packages/tauri-plugin-ptt/ — a Tauri v2 plugin that wraps AVAudioEngine + SFSpeechRecognizer (STT) and AVSpeechSynthesizer (TTS) on iOS, and wires the PTT button in MascotScreen.tsx. Plugin layout: - Rust: lib.rs + commands.rs + mobile.rs + error.rs + models.rs On iOS: delegates to PttMobile<R> via PluginHandle::run_mobile_plugin. On non-iOS: PttHandle stub returns NotSupported for all commands. - Swift: PTTPlugin / PTTRecorder / PTTSpeaker / AudioSessionManager AVAudioSession category: .playAndRecord + .defaultToSpeaker + BT options (pattern from chat4000/Sources/Services/VoiceNotes.swift) SFSpeechRecognizer: single task per session, torn down on stopListening. Interruption + route-change observers stop the recorder gracefully. App-background observer stops recording and cancels TTS. - guest-js: startListening / stopListening / speak / cancelSpeech / listVoices + onTranscriptPartial / onTranscriptFinal / onTtsStarted / onTtsEnded / onError Events: ptt://transcript-partial, ptt://transcript-final, ptt://tts-started, ptt://tts-ended, ptt://error. MascotScreen changes: - PTTButton is now live: onPointerDown -> startListening, onPointerUp -> stopListening -> chatSend with transcript. - Partial captions shown above button while recording. - cancelSpeech called on each PTT press to cancel any active TTS. - chat_done handler speaks the assistant reply via AVSpeechSynthesizer. - Error toast for permission denial / interruption. Quality gates: - cargo check packages/tauri-plugin-ptt: clean - cargo check app/src-tauri: clean (desktop unaffected) - pnpm typecheck: clean - pnpm lint: 0 new errors - pnpm test:unit: 218/218 pass (13 new guest-js tests + 7 new PTT UI tests)
…cked Sendable SFSpeechRecognitionTask exposes no `result` property; the recognitionTask result handler is the only source of partial transcripts. Mirror the latest into a stored property so stopListening() can return the final text. AVSpeechSynthesizer is not Sendable; PTTSpeaker serializes access through the plugin command actor, so the unchecked conformance is sound.
…8 of iOS PR) - app/src-tauri/Info.ios.plist: NSCamera, NSMicrophone, NSSpeechRecognition privacy strings for the iOS bundle (copied to gen/apple after tauri ios init) - scripts/ios-init.sh: wrapper that runs tauri ios init and prints next steps - package.json (root): adds tauri:ios:init / tauri:ios:dev / tauri:ios:build top-level delegates so iOS builds work from the repo root like pnpm dev - .github/workflows/ios-compile.yml: macOS CI sanity check (hard gate on host cargo check + tsc + Vitest; soft gate on aarch64-apple-ios cargo check) - src/openhuman/about_app/: new Mobile category + three capabilities: mobile.device_pairing, mobile.ios_client, mobile.push_to_talk - docs/ios/SETUP.md: setup guide covering init, Info.plist, pairing flow, transport selection, security notes, known limitations, CI policy - docs/ARCHITECTURE.md: iOS client subsection (transport, pairing, key paths, security, backend dependency) - CLAUDE.md: iOS client section after Runtime scope (transport strategies, key paths, build entry, docs pointer) - PR_DESCRIPTION.md: draft PR description following project template - cargo fmt / prettier auto-fixes across ptt plugin and MascotScreen - PairPhoneModal polling kept (no domain-event bridge exists yet); existing TODO comment documents the follow-up
Contributor
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
tauri-plugin-pttSwift plugin (packages/tauri-plugin-ptt/) for AVAudioEngine + SFSpeechRecognizer on iOS.Problem
Users with iOS devices had no way to interact with their OpenHuman assistant on the go. The desktop app required a local machine. This PR adds the client-side scaffolding and transport layer needed to bridge iOS to an existing desktop core.
Solution
The iOS app is a subset of the existing React/TypeScript UI, compiled by Tauri v2 into an iOS bundle. A
TransportManagerselects the best transport at runtime. Pairing is secured by an X25519 key agreement; all tunnel traffic uses XChaCha20-Poly1305 encryption. The backend is a blind socket.io forwarder -- it never sees plaintext.Layer-by-layer commits
a99537f3src/openhuman/devices/)4ea14b78TransportManager,LanHttpTransport,TunnelTransport,CloudHttpTransport, tunnel crypto (app/src/services/transport/,app/src/lib/tunnel/)ba651705/settings/devicesUI --DevicesPanel,PairPhoneModalwith QR generation and 2-second poll3e0e2a67#[cfg(target_os = "ios")]guards on CEF-specific code621fec98PairScreen(QR scan viaAVCaptureSession),MascotScreen(chat UI)5ca6cf21tauri-plugin-ptt-- Swift PTT plugin (AVAudioEngine, SFSpeechRecognizer, AVSpeechSynthesizer)41a6a895@unchecked Sendableon PTTSpeakerTest coverage
What is gated behind the iOS target
The following only activates on
cfg(target_os = "ios")or when explicitly called from iOS screens:app/src-tauri/(accounts webviews, etc.)tauri-plugin-pttcommands (start_listening,stop_listening,speak,cancel_speech,list_voices) -- returnNotSupportedon non-iOS targets.packages/tauri-plugin-ptt/ios/Swift sources -- not compiled for desktop.Desktop users see no change.
Known TODOs for follow-up PRs
PairPhoneModalpollsdevices_listevery 2 s. Switch to a socket event subscription when the SSE/socket bridge forDomainEvent::DevicePairedlands.cargo check --target aarch64-apple-iosruns withcontinue-on-error: truein the new CI workflow because third-party C deps (cef-dll-sys) may fail without full Xcode on the runner. A follow-up should pin an Xcode-enabled runner and harden this to a hard gate.Info.ios.plistkeys into the generated Xcode project aftertauri ios init. Should automate viabundle.iOS.templateonce Tauri v2 stabilises the iOS template pipeline.Backend dependency
tinyhumansai/backend#709must be merged and deployed before end-to-end pairing works. Thedevices_create_pairingRPC will return a tunnel registration error until thetunnel:register/tunnel:connect/tunnel:framesocket.io contract is live.Manual test plan for iOS reviewer
(Requires a physical iPhone or iOS 17+ simulator paired with the desktop app.)
From
packages/tauri-plugin-ptt/README.md:startListeningcall.ptt://errorwithcode: interrupted.cancelSpeechduring TTS emitstts-endedwithfinished: false.listVoicesreturns non-empty list ofAVSpeechSynthesisVoiceentries.Additional pairing flow checks:
Screenshots
Submission Checklist
src/openhuman/devices/was covered in Layer 1 tests; new TS code inapp/src/services/transport/andapp/src/lib/tunnel/covered by Vitest suites. PTT Swift layer cannot be unit-tested without iOS toolchain (noted in README).docs/RELEASE-MANUAL-SMOKE.mdyet -- tracked as follow-up.Impact
packages/tauri-plugin-ptt/is a new crate workspace member; adds to build time only when targeting iOS.mobile.*entries and a newMobilecategory.Related
AI Authored PR Metadata (required for Codex/Linear PRs)
Linear Issue
Commit & Branch
feat/ios-clientValidation Run
pnpm --filter openhuman-app format:check-- cleanpnpm typecheck-- cleancargo fmt --all+cargo checkon all three Cargo.toml -- cleanValidation Blocked
cargo check --target aarch64-apple-ioscontinue-on-error: truein CI.Behavior Changes
Parity Contract
NotSupportedon non-iOS. Transport falls back gracefully.Duplicate / Superseded PR Handling