Skip to content

feat(mobile): Expo companion app [beta]#1245

Draft
Jinwoo-H wants to merge 50 commits intomainfrom
Jinwoo-H/mobile-ui-polish
Draft

feat(mobile): Expo companion app [beta]#1245
Jinwoo-H wants to merge 50 commits intomainfrom
Jinwoo-H/mobile-ui-polish

Conversation

@Jinwoo-H
Copy link
Copy Markdown
Contributor

@Jinwoo-H Jinwoo-H commented Apr 29, 2026

Summary

  • Expo mobile app with QR code pairing and WebSocket RPC transport to connect to Orca desktop
  • End-to-end encrypted WebSocket transport between mobile and desktop
  • Live terminal streaming — view and interact with desktop terminals from the phone, including fit-to-phone mode with robust restore
  • Worktree management — list, create, pin, sleep, and delete worktrees from mobile with optimistic UI
  • Custom keyboard shortcuts — built-in terminal shortcuts (Ctrl+L/Z/R/A/E/W/U) plus a picker to add persistent Ctrl+Key, Alt+Key, or text macro shortcuts
  • Desktop integration — Mobile settings pane, device registry, pairing QR generation, terminal buffer serialization for mobile subscribers
  • Unified modal design language — all modals use consistent bottom drawer pattern with inner action groups
  • Haptic feedback on long-press and destructive actions
  • CI — GitHub Actions workflow for mobile type-checking

Test plan

  • Pair mobile app with desktop via QR scan
  • Verify WebSocket connection and reconnection
  • Open a worktree terminal and confirm live streaming
  • Test fit-to-phone toggle and verify TUI rendering
  • Send input from mobile keyboard including custom shortcuts
  • Create, sleep, and delete worktrees from mobile
  • Verify haptic feedback on long-press actions
  • Test modal transitions (no backdrop flicker between action sheet → confirm)
  • Verify custom shortcut persistence across app restarts
  • Run pnpm tsc --noEmit in mobile directory

Made with Orca 🐋

@Jinwoo-H Jinwoo-H changed the title feat(mobile): Expo companion app with terminal streaming and polished UX feat(mobile): Expo companion app [beta] Apr 29, 2026
@Jinwoo-H Jinwoo-H force-pushed the Jinwoo-H/mobile-ui-polish branch from 82fda41 to 28fb6ca Compare April 29, 2026 08:30
Jinwoo-H and others added 27 commits April 30, 2026 02:41
…ransport

Adds the Orca Mobile companion app (Expo SDK 55 / React Native 0.83) with:
- QR code pairing flow: scan orca://pair# URL, validate auth, save host profile
- WebSocket RPC client with auto-reconnect, exponential backoff, streaming support
- Three screens: host list, worktree list, terminal session with live output
- Mock WebSocket server for development without a running Orca desktop instance
- Desktop-side: TLS certificate management, device registry, mobile IPC handlers,
  WebSocket transport layer, and pairing offer encode/decode with Zod validation
- Unit tests for WebSocket transport and pairing codec

Co-authored-by: Orca <help@stably.ai>
- 7 unit tests for RpcDispatcher streaming: scrollback emit, live data
  chunks, unsubscribe cleanup, one-shot fallback, error handling
- GitHub Actions workflow for mobile: typecheck + lint on PR, Android
  debug build with JDK 17 (temurin)

Co-authored-by: Orca <help@stably.ai>
…ocket transport

The hand-rolled X.509 certificate generation produced malformed certs
that OpenSSL and Node's HTTPS server couldn't parse, silently preventing
the WebSocket transport from starting. Replace with openssl CLI call
which is available on all target platforms.

Also enable WebSocket transport by default so mobile clients can connect.

Co-authored-by: Orca <help@stably.ai>
…implemented

React Native's built-in WebSocket rejects self-signed TLS certs with no
way to pin by fingerprint. Use plain ws:// for now — per-device token
auth still prevents unauthorized access. TLS will be re-enabled once the
Expo native module for certificate pinning is built (Phase 1 checklist).

Co-authored-by: Orca <help@stably.ai>
Co-authored-by: Orca <help@stably.ai>
Co-authored-by: Orca <help@stably.ai>
Co-authored-by: Orca <help@stably.ai>
Use the WebView's native viewport scaling and pinch-to-zoom instead of
custom CSS transforms and touch handlers. The terminal renders at
desktop dimensions with initial-scale=0.5 for a readable default, and
the browser's built-in zoom handles pinch correctly (center point,
scroll directions, no swipe conflicts).

Co-authored-by: Orca <help@stably.ai>
Co-authored-by: Orca <help@stably.ai>
Co-authored-by: Orca <help@stably.ai>
Add end-to-end mobile-fit override system that lets a phone client resize
a PTY to phone dimensions and restore to desktop dimensions on disconnect
or user request.

Key changes:
- Runtime: resizeForClient() manages override lifecycle with ownership,
  clamping, and auto-restore on client disconnect. Always resizes PTY
  directly on restore instead of relying on fragile renderer chain.
- RPC: terminal.resizeForClient method for mobile clients via WebSocket.
- Renderer: safeFit() respects active overrides, desktop banner with
  Restore button, override change listener with rAF + fallback safety net.
- Mobile: fit-to-phone with CSS scale zoom, auto-fit only for terminals
  created during the mobile session (not existing desktop terminals).
- Tests: 14 unit tests + 3 integration tests for override lifecycle.

Co-authored-by: Orca <help@stably.ai>
Replace native WebView scroll/zoom with custom touch handlers that
work correctly with the CSS scale transform:

- Single-finger scroll with scale-compensated sensitivity and momentum
- Pinch-to-zoom (1x-5x) centered on pinch point, with pan when zoomed
- Tap/click suppression to prevent cursor jumping
- Row fill: resize xterm viewport to fill WebView height (display-only)
- Disable native zoom (user-scalable=no) since custom handler replaces it

Co-authored-by: Orca <help@stably.ai>
Add module-level Map to persist which terminals have been phone-fitted,
so navigating away and back auto-refits them instead of reverting to
desktop dimensions. Also adds fitPendingRef to prevent duplicate fits.

Co-authored-by: Orca <help@stably.ai>
Replace the disabled TLS cert pinning approach with application-layer
E2EE using tweetnacl (Curve25519 ECDH + XSalsa20-Poly1305). The
WebSocket stays plain ws:// but all RPC traffic is encrypted after
a one-round-trip handshake.

Desktop: generate persistent Curve25519 keypair, E2EE channel module
handles handshake state machine and transparent encrypt/decrypt.
Mobile: ephemeral keypair per connection for forward secrecy, expo-crypto
PRNG polyfill for Hermes compatibility.

Pairing schema updated from v1 (certFingerprint) to v2 (publicKeyB64).
Mock server updated with E2EE support. 24 tests covering crypto,
channel handshake, and integration.

Co-authored-by: Orca <help@stably.ai>
- Add expo-haptics medium impact feedback on all long-press interactions
  (terminal tabs, worktree cards, host cards)
- Add "Sleep" option with Moon icon to worktree action sheet, calling
  worktree.sleep RPC method
- Change send button from bright blue (accentBlue) to subtle dark
  (bgRaised) background with textSecondary icon color

Co-authored-by: Orca <help@stably.ai>
Co-authored-by: Orca <help@stably.ai>
Co-authored-by: Orca <help@stably.ai>
Co-authored-by: Orca <help@stably.ai>
- Add custom keyboard shortcuts (Ctrl+Key, Alt+Key, text macros) with persistent storage
- Add built-in terminal shortcuts (Ctrl+L/Z/R/A/E/W/U) and resize toggle
- Allow sending Enter on empty input for terminal convenience
- Fix delete worktree: correct RPC method (worktree.rm) with optimistic UI
- Fix modal transition flicker between action sheet and confirm dialog
- Unify all modals to consistent bottom drawer pattern with inner action groups

Co-authored-by: Orca <help@stably.ai>
- Remove unused 'startup' parameter in useIpcEvents onActivateWorktree
- Compact attach-main-window-services to stay within max-lines limit

Co-authored-by: Orca <help@stably.ai>
- Extract send helper in attach-main-window-services to stay under max-lines
- Remove unused startup parameter in useIpcEvents

Co-authored-by: Orca <help@stably.ai>
- Add missing randomUUID import in pty.ts
- Add getForegroundProcess to RuntimePtyController test mocks
- Add terminalFitOverrideChanged to RuntimeNotifier test mocks
- Fix vi.fn() type assertions in e2ee-integration tests

Co-authored-by: Orca <help@stably.ai>
Use performAndroidHapticsAsync on Android which uses HapticFeedbackConstants
instead of the Vibrator API. This fixes haptics not working on Android and
removes the VIBRATE permission requirement.

Co-authored-by: Orca <help@stably.ai>
…structions

- Add shared BottomDrawer component with pan gesture tracking via reanimated
- Drawer follows finger on swipe down, dismisses past threshold or on velocity
- Rubber-band resistance on upward pull for tactile feel
- Migrate all 5 modal components to use BottomDrawer
- Replace plain text QR instructions with numbered steps with icons

Co-authored-by: Orca <help@stably.ai>
Remove duplicate onPtyData call in pty.ts that caused every daemon-backed
PTY data event to be dispatched twice through dataListeners, doubling all
terminal output sent to mobile clients.

Also: clean up registerSubscriptionCleanup to evict stale listeners on
reconnect, add regression tests, fix CI lint/type errors, add mobile UX
improvements (worktree creation, settings, about screen).

Co-authored-by: Orca <help@stably.ai>
- BottomDrawer: shift drawer up when keyboard opens using Keyboard events,
  add ScrollView for content overflow
- NewWorktreeModal: replace FlatList with plain map to avoid nested
  VirtualizedList error
- Home: replace Server icon with Monitor, hide IP from cards (show in
  action sheet), add connection status subtitle, update hero copy
- About: remove redundant link labels, use neutral text color for URLs

Co-authored-by: Orca <help@stably.ai>
The full Gradle build is heavy for every PR push. Keep only typecheck +
lint as the PR gate; run the APK build on main pushes and manual dispatch.

Co-authored-by: Orca <help@stably.ai>
Jinwoo-H and others added 23 commits April 30, 2026 02:43
Trigger on mobile-v* tags instead of main pushes. Builds a release APK,
uploads as artifact, and creates a GitHub Release with the APK attached.
Manual dispatch still available for on-demand builds.

Co-authored-by: Orca <help@stably.ai>
The mobile-fit-override code calls safeFit with ManagedPane (from
getPanes()), but safeFit was typed to require ManagedPaneInternal.
Widen the parameter type since safeFit only needs fields on ManagedPane.

Co-authored-by: Orca <help@stably.ai>
- Add shouldRunSetupForCreate to hooks mock in orca-runtime tests
- Fix send() helper to use rest args, avoiding extra undefined in IPC calls

Co-authored-by: Orca <help@stably.ai>
- Update runtime-client test to use transports array (matches schema change)
- Remove 3 preallocated-handle-survives-reload tests that tested unreachable
  behavior (preallocated handles can't survive epoch changes)
- Revert unnecessary readTerminal fallback added during debugging

Co-authored-by: Orca <help@stably.ai>
Parallel e2e workers each launch an isolated Electron instance. All
instances were racing to bind the same fixed WebSocket port (6768),
causing EADDRINUSE crashes that manifested as "Target page, context
or browser has been closed" in every Playwright spec.

Pass port 0 when ORCA_E2E_USER_DATA_DIR is set so the OS assigns a
random available port per instance. The full WebSocket startup path
is still exercised, preserving coverage for future mobile pairing
e2e tests.

Co-authored-by: Orca <help@stably.ai>
- Redesign home screen with "Welcome back" hero and usage stats cards
  (agents spawned, agent time, PRs created) fetched via stats.summary RPC
- Add dedicated notifications settings page with monochrome push toggle
- Add troubleshooting page with diagnostics and common issues accordion
- Add splash screen with Orca logo using expo-splash-screen
- Fix pin sync: prioritize pinned/unread worktrees in compareWorktreePs
  sort so they survive the 200-worktree truncation limit in worktree.ps
- Sync local pin state from server on each poll to reflect desktop pins
- Add isPinned support to worktree.set RPC handler
- Add stats.summary and notification subscription RPC methods
- Clean up debug logging from notification handlers

Co-authored-by: Orca <help@stably.ai>
- Add network interface selector so users can choose LAN or overlay
  network (Tailscale, ZeroTier) address for QR code pairing
- Fall back to OS-assigned port when preferred port is in use
- Filter paired devices list to only show actually connected devices
- Poll device list after QR generation until phone connects
- Mark device lastSeenAt on first WebSocket connection
- Make CustomKeyModal toggle and button monochrome (remove blue accent)
- Make clear filters text and group options monochrome

Co-authored-by: Orca <help@stably.ai>
…-fit

When the desktop restored a terminal from mobile-fit, the notification
only reached the Electron renderer via IPC. Mobile WebSocket clients
were never informed, leaving the phone UI stuck in fitted mode.

Add fit-override change listeners to the runtime (parallel to existing
dataListeners) and emit fit-override-changed events through the
terminal.subscribe stream. The mobile session screen handles the event
by clearing fitted state and resubscribing for a fresh scrollback
snapshot at the restored desktop dimensions.

Co-authored-by: Orca <help@stably.ai>
Co-authored-by: Orca <help@stably.ai>
Use a ref to break the self-reference cycle that caused TS7022.

Co-authored-by: Orca <help@stably.ai>
…ence

- Add worktree.sleep RPC method with full notifier chain (runtime → IPC → renderer)
- Optimistic UI update for sleeping worktrees in mobile client
- Fix splash screen: configure expo-splash-screen plugin, hide splash only after navigator layout
- Regenerate app icon and splash assets at correct density sizes
- Fix host removal not persisting by disabling Android auto-backup (allowBackup=false)
- Guard against invalid host credentials (base64 crash) and prevent navigating into auth-failed hosts
- Extract ActionSheetContent for inline drawer content switching
- Rewrite BottomDrawer as always-mounted Animated.View overlay (replaces Modal)
- Add ConfirmModal component with inline confirm/cancel buttons

Co-authored-by: Orca <help@stably.ai>
…ck actions

- Move settings gear to header, remove bottom footer
- Stat cards match desktop's minimal style (subtle border, icon box)
- Host cards show worktree count and active count via worktree.ps RPC
- Status ring on host icon corner instead of inline dot
- Resume section surfaces most recent active worktree across hosts
- Quick actions: Pair Desktop and New Worktree
- Empty state: welcoming onboarding with "How it works" steps
- Add card radius token (14px) to theme
- Include HTML mock for design reference

Co-authored-by: Orca <help@stably.ai>
- Move status dot inline with status text, remove icon overlay ring
- Fix worktree.ps response parsing (returns {worktrees:[...]} not array)
- Fix Resume card route, sizing, and text alignment to match host cards
- Rename stat labels to be more explicit (Agents spawned, PRs created)
- Wire New Worktree button to navigate to host detail with auto-open drawer
- Cache worktrees for instant host detail page loads
- Defer RPC connection until after navigation animation
- Remove desktop icon and greeting from onboarding screen
- Request OS notification permissions when toggle is enabled

Co-authored-by: Orca <help@stably.ai>
…ut of KAV

- Add disposed guards to session screen RPC connection to prevent
  state updates and client leaks after unmount
- Fix nested timer leak in reconnect effect with addTimer helper
  that checks disposed flag before scheduling
- Move TerminalWebView touch/mouse listeners out of init() to IIFE
  top level so they register once, not on every re-init
- Move session screen modals outside KeyboardAvoidingView so drawer
  overlays cover the full screen height
- Track last-visited worktree in AsyncStorage so the home screen
  Resume card reflects mobile session history, not just desktop
  activity ordering
- Re-fetch worktreeInfo on home screen focus so Resume card updates
  after navigating back from a session
- Add stale guards to NewWorktreeModal async effects
- Fix worktree cache LRU eviction (delete-before-set for Map ordering)
- Use useState initializer for host detail cache to avoid re-reading
  on every render

Co-authored-by: Orca <help@stably.ai>
Co-authored-by: Orca <help@stably.ai>
Co-authored-by: Orca <help@stably.ai>
Co-authored-by: Orca <help@stably.ai>
Co-authored-by: Orca <help@stably.ai>
Co-authored-by: Orca <help@stably.ai>
Co-authored-by: Orca <help@stably.ai>
Co-authored-by: Orca <help@stably.ai>
Co-authored-by: Orca <help@stably.ai>
@Jinwoo-H Jinwoo-H force-pushed the Jinwoo-H/mobile-ui-polish branch from 9e30754 to 2ad7577 Compare April 30, 2026 06:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant