feat(mobile): add local voice and sandbox routing#7850
Conversation
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Claude encountered an error —— View job I'll analyze this and get back to you. |
LifeOps Multi-Tier BenchmarkSuite:
|
Dexploarer
changed the title
|
Claude encountered an error —— View job I'll analyze this and get back to you. |
LifeOps Multi-Tier BenchmarkSuite:
|
|
Claude encountered an error —— View job I'll analyze this and get back to you. |
LifeOps Multi-Tier BenchmarkSuite:
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| tailWorkerEvents(options: { | ||
| id: string; | ||
| afterSequence?: number; | ||
| limit?: number; | ||
| }): RemotePluginWorkerEventsTailSnapshot { | ||
| const record = this.workers.get(options.id); | ||
| if (!record?.handle || record.status.state !== "running") { | ||
| throw new Error( | ||
| `remote-plugin events: target ${options.id} is not running.`, | ||
| ); | ||
| } | ||
| const limit = this.normalizeEventTailLimit(options.limit); | ||
| const events = this.workerEvents.get(options.id) ?? []; | ||
| const afterSequence = options.afterSequence; | ||
| const filtered = | ||
| typeof afterSequence === "number" | ||
| ? events.filter((event) => event.sequence > afterSequence) | ||
| : events.slice(-limit); | ||
| const selected = filtered.slice(0, limit); | ||
| const currentSequence = this.workerEventSequences.get(options.id) ?? 0; | ||
| return { | ||
| id: options.id, | ||
| events: selected, | ||
| nextSequence: | ||
| selected.length > 0 | ||
| ? selected[selected.length - 1].sequence | ||
| : (afterSequence ?? currentSequence), | ||
| }; | ||
| } | ||
|
|
||
| dispose(): void { |
There was a problem hiding this comment.
tailWorkerEvents drops old events via splice(0, n) when the buffer exceeds maxWorkerEvents, but the response never communicates the minimum available sequence to the caller. A polling consumer holding a stale afterSequence will silently skip any evicted events between the last seen sequence and the oldest surviving entry — there is no way to detect the gap from the returned nextSequence. Under a sustained burst (>1 000 events between two poll cycles with the default buffer), an entire window of events disappears without any error or gap marker in the returned snapshot.
| const musicPlayer = getMusicPlayerManager(); | ||
| const browserWorkspace = getBrowserWorkspaceManager(); | ||
| const remotePluginHost = getRemotePluginHost(); | ||
| registerBuiltInDynamicViews(); | ||
| const dynamicViewRegistry = getDynamicViewRegistry(); | ||
| const dynamicViewSessions = getDynamicViewSessionManager({ | ||
| registry: dynamicViewRegistry, | ||
| canvas, | ||
| workerStatusProvider: { | ||
| getWorkerStatus: (id) => remotePluginHost.getWorkerStatus(id), | ||
| }, | ||
| }); | ||
| remotePluginHost.setDynamicViewHost( | ||
| createDynamicViewHostForRuntime(dynamicViewSessions), | ||
| ); | ||
| const traceService = getTraceService({ | ||
| dynamicViewRegistry, | ||
| dynamicViewSessions, | ||
| }); | ||
| remotePluginHost.setTraceHost(createTraceHostForRuntime(traceService)); | ||
| const voiceService = new VoiceService({ traceService }); | ||
| remotePluginHost.setVoiceHost(createVoiceHostForRuntime(voiceService)); | ||
| const launchOrchestrator = new LaunchOrchestrator({ | ||
| agent, | ||
| readBootProgress: async () => { | ||
| const status = agent.getStatus(); | ||
| return composeBootProgressSnapshot( | ||
| { ...status, port: resolveRpcAgentPort(status.port) }, | ||
| readAgentHealthSnapshotViaHttp, | ||
| ); | ||
| }, | ||
| readAuthStatus: readAuthStatusViaHttp, | ||
| readOnboardingStatus: readOnboardingStatusViaHttp, | ||
| readDiagnostics: getStartupDiagnosticsSnapshot, | ||
| readDatabaseStatus: () => agent.getDatabaseSnapshot(), | ||
| readDiagnosticLogTail: getStartupDiagnosticLogTail, | ||
| listSatelliteStatuses: () => | ||
| getFirstPartySatelliteDefinitions({ includeDev: true }).map( | ||
| (definition) => { | ||
| const status = remotePluginHost.getWorkerStatus(definition.id); | ||
| return { | ||
| id: definition.id, | ||
| state: status?.state ?? "stopped", | ||
| error: status?.error ?? null, | ||
| required: definition.kind === "required", | ||
| }; | ||
| }, | ||
| ), | ||
| createBugReportBundle: (params) => desktop.createBugReportBundle(params), | ||
| dynamicViewRegistry, | ||
| dynamicViewSessions, | ||
| }); | ||
| configureRemotePluginHostEvents(sendToWebview); | ||
|
|
There was a problem hiding this comment.
VoiceService and LaunchOrchestrator created inline without disposal
buildBunRpcHandlers instantiates new VoiceService(...) and new LaunchOrchestrator(...) directly inside the function body, then wires them into the remotePluginHost singleton via setVoiceHost / setTraceHost / setDynamicViewHost. If the function is ever called a second time — for example during a webview reconnect or in test harness setups — the previous service instances have their host references on the singleton overwritten without being disposed, leaking any timers or subscriptions they hold. The other services in this function use module-level singleton getters (getTraceService, getDynamicViewRegistry), making the inline instantiation stand out.
|
Claude encountered an error —— View job I'll analyze this and get back to you. |
LifeOps Multi-Tier BenchmarkSuite:
|
| if (prompts.length === 0) { | ||
| return fallbackCaptureSession(sessionId); | ||
| } |
There was a problem hiding this comment.
sessionId paired with local fallback prompts on malformed response
When the server returns a sessionId but yields no parseable prompts (empty array, all entries fail validation, or both prompts and script missing), fallbackCaptureSession(sessionId) now produces a VoiceCaptureSession that carries the server's own sessionId while substituting hardcoded OWNER_CAPTURE_FALLBACK_PROMPTS. All subsequent appendOwnerCapture calls route audio to /api/voice/onboarding/profile/append?id=<server-sessionId>, so the server receives audio recorded against scripts it never sent. On finalizeOwnerCapture the server creates a profile from a mismatched script/audio pairing, silently corrupting the voice model. The original code returned null from normaliseCaptureSession in this branch, which the call site resolved to a fully local session with a local- prefixed ID that the server never saw.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Claude encountered an error —— View job I'll analyze this and get back to you. |
LifeOps Multi-Tier BenchmarkSuite:
|
|
Merged into develop manually as part of post-merge cleanup; head ref fully contained in develop (git rev-list --count develop..pr-7850 = 0). See merge commit b172b5a. |
…ts-onboarding # Conflicts: # bun.lock # packages/agent/src/runtime/eliza.ts # packages/agent/src/services/e2b-capability-router.coding-remote-runner.test.ts # packages/agent/src/services/e2b-capability-router.test.ts # packages/agent/src/services/e2b-capability-router.ts # packages/app-core/platforms/electrobun/README.md # packages/app-core/platforms/electrobun/remotes/fs/plugin.json # packages/app-core/platforms/electrobun/remotes/git/plugin.json # packages/app-core/platforms/electrobun/remotes/local-model/plugin.json # packages/app-core/platforms/electrobun/remotes/pty/plugin.json # packages/app-core/platforms/electrobun/remotes/runtime/README.md # packages/app-core/platforms/electrobun/remotes/runtime/plugin.json # packages/app-core/platforms/electrobun/remotes/runtime/src/bun/worker.ts # packages/app-core/platforms/electrobun/remotes/surface/README.md # packages/app-core/platforms/electrobun/remotes/surface/plugin.json # packages/app-core/platforms/electrobun/remotes/surface/src/bun/worker.ts # packages/app-core/platforms/electrobun/remotes/surface/src/dev/phase4-smoke.ts # packages/app-core/platforms/electrobun/remotes/surface/src/protocol/runtime-client.ts # packages/app-core/platforms/electrobun/src/dynamic-views/README.md # packages/app-core/platforms/electrobun/src/first-party-remotes.test.ts # packages/app-core/platforms/electrobun/src/first-party-remotes.ts # packages/app-core/platforms/electrobun/src/rpc-handlers.ts # packages/cloud-api/__tests__/users-me-wallet-attach.test.ts # packages/cloud-services/coding-remote-runner/Dockerfile # packages/cloud-services/coding-remote-runner/README.md # packages/cloud-services/coding-remote-runner/__tests__/server.test.ts # packages/cloud-services/coding-remote-runner/package.json # packages/cloud-services/coding-remote-runner/src/index.ts # packages/cloud-shared/src/lib/config/containers-env.ts # packages/cloud-shared/src/lib/services/coding-containers.test.ts # packages/homepage/tests/e2e/aesthetic-audit.spec.ts # packages/homepage/tests/e2e/marketing-cloud-download.spec.ts # packages/scripts/audit-capability-router-naming.self-test.ts # packages/ui/src/components/onboarding/VoicePrefixGate.tsx # packages/ui/src/components/onboarding/VoicePrefixSteps.test.tsx # packages/ui/src/components/onboarding/VoicePrefixSteps.tsx
- Address greptile review findings: - rename ELIZA_CARROT_MAX_WORKER_EVENTS → ELIZA_REMOTE_PLUGIN_MAX_WORKER_EVENTS (with legacy fallback) - drop redundant satelliteId field from RemotePluginWorkerEventRecord - restore readonly index signature on JsonObject - fix VoicePrefixGate useEffect dep that was never invoked - Sweep remaining satellite/Satellite/SATELLITE identifiers in elizaOS source/tests/docs - Rename example dirs: hello-carrot → hello-remote-plugin, carrot-clock → remote-plugin-clock - Update i18n keys settings.sections.carrots → settings.sections.remote-plugins - Repurpose audit-capability-router-naming script: guards against legacy satellite/carrot vocabulary returning instead of tracking the migration in progress - Preserve electrobun build config field names (carrot/carrotOnly) — external API - Preserve YOLO COCO class label "carrot" in plugin-vision — unrelated Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ts-onboarding # Conflicts: # packages/agent/vitest.config.ts # plugins/plugin-hearwear/src/index.ts # plugins/plugin-xr/src/__tests__/xr-feature-parity.test.ts
|
Claude encountered an error —— View job I'll analyze this and get back to you. |
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…file - Take develop's VoicePrefixGate/VoicePrefixSteps versions for elizaOS branding (bg-[#F7F9FF] / bg-[#0B35F1] / text-[#0B35F1] palette enforced by static-smoke.sh) - Bump coding-remote-runner bun-types to 1.3.14 to match root lockfile entry, fixing env:audit:check diff against bun install Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
LifeOps Multi-Tier BenchmarkSuite:
|
- first-party-remotes.ts: path.join(platformRoot, "remotePlugins") was produced by the global satellites→remotePlugins sed but the actual on-disk directory in this PR's rename is "remotes". Fixes all first-party-remotes.test.ts failures. - apps/app-xr/e2e/all-views-crud.spec.ts: drop hearwear → facewear in the view-id list to match plugin-xr/src/__tests__/xr-feature-parity.test.ts (24 views, facewear) as landed on develop. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
…ts-onboarding # Conflicts: # packages/app-core/scripts/build-llama-cpp-dflash-targets.test.mjs # packages/app-core/scripts/build-llama-cpp-dflash.mjs # packages/app-core/src/runtime/platform-policy-docs.test.ts # plugins/plugin-coding-tools/src/actions/ls.test.ts # plugins/plugin-coding-tools/src/actions/ls.ts
|
Claude encountered an error —— View job I'll analyze this and get back to you. |
LifeOps Multi-Tier BenchmarkSuite:
|
…ent fixture Side-effect of the satellite→remotePlugin sed: the test had both satelliteId and remotePluginId on the same object literal; the sed turned both into remotePluginId, producing a duplicate key. Tests still pass because the values matched, but the literal is now valid. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sort order changed after the satellite→remotePlugin/RemotePlugin rename because tokens changed length. Pure mechanical sort, no logic changes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Claude encountered an error —— View job I'll analyze this and get back to you. |
LifeOps Multi-Tier BenchmarkSuite:
|
|
Claude encountered an error —— View job I'll analyze this and get back to you. |
LifeOps Multi-Tier BenchmarkSuite:
|
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…luginId - packages/app-core/vitest.config.ts: drop duplicate pluginWorkerRuntimeSrc declaration introduced by the develop merge (esbuild rejected the dup) - phase4-smoke.ts: drop duplicate remotePluginId object key (sed artifact) - biome --write across PR-touched files (import sort, unused import drop, no semantic changes) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Claude encountered an error —— View job I'll analyze this and get back to you. |
LifeOps Multi-Tier BenchmarkSuite:
|
2 participants
Relates to
mode: "direct" | "remote"). This branch accepts the landed remote-plugin naming and keeps the implementation compatible with that direction.Background
What does this PR do?
This PR makes the iOS app and remote coding path usable from mobile while preserving the new remote-plugin architecture boundary.
plugin-coding-toolsparity scoped:lscan route througheliza.fs; grep/glob remain plugin-owned until explicit search/glob semantics exist; edit remains plugin-owned until a first-class patch/edit capability exists.origin/develop, including the Babylon -> Feed rename and the upstream remote-plugin vocabulary changes.Why is this needed?
The product goal is that a user can open Eliza on a phone and still use the agent to talk, set up local voice/model state, sign into cloud, and start real coding work from any device. iOS cannot safely or App-Store-compatibly run arbitrary PTYs, Codex/Claude/OpenCode agent runners, or native build tools inside the client. Those jobs need to route to an Eliza Cloud sandbox or a trusted home/remote worker, while the mobile app stays the controller and renderer.
This PR also removes ambiguity around voice readiness. Having the files on disk is not the same as having a working native synth engine, and onboarding should not pretend otherwise.
Risks
Medium. This touches mobile onboarding, iOS local voice, capability routing tests, and the remote-plugin merge boundary. The riskiest areas are native iOS voice/runtime behavior on physical devices and CI paths that moved under the recent
developrename wave.Documentation changes needed?
Docs are included where this branch already introduced or updated the remote coding/sandbox and mobile paths. No separate migration guide is required for app users in this PR.
Testing
Local validation run after reconciling
origin/developbun run test:remote-capabilities:surface-auditELIZAOS_STATIC_SOURCE_ONLY=1 ./scripts/static-smoke.shfrompackages/os/linux/variants/milady-tailsbun run --cwd packages/ui typecheckbun run --cwd packages/app typecheckbun run --cwd packages/app-core/platforms/electrobun typecheckbun run --cwd packages/app-core/platforms/electrobun testbun run --cwd plugins/plugin-capacitor-bridge typecheckbun run --cwd plugins/plugin-coding-tools typecheckbun run --cwd plugins/plugin-coding-tools test src/actions/ls.test.ts src/actions/bash.test.ts --reporter verbosenode --check packages/app-core/scripts/kernel-patches/cpu-simd-kernels.mjsnode --check packages/app-core/scripts/kernel-patches/cpu-polar-kernels.mjsnode --check packages/app-core/scripts/build-llama-cpp-dflash.mjsnode --check packages/app-core/scripts/run-mobile-build.mjsnode --check packages/test/cloud-e2e/src/fixtures/stack.tsnode --check packages/tui/src/utils.tsgit diff --checkgit diff --cached --checkReviewer start point
Start with:
packages/ui/src/components/onboarding/VoicePrefixGate.tsxpackages/ui/src/components/onboarding/VoicePrefixSteps.tsxpackages/app-core/scripts/build-llama-cpp-dflash.mjsplugins/plugin-coding-tools/src/actions/ls.tsDeploy notes
CI is rerunning on the pushed merge commit. #7844 is still open and dirty; this PR preserves compatibility with its stated direction without depending on it landing first.
Greptile Summary
This PR wires up the iOS local-voice onboarding path, migrates remote sandbox/coding routing from "satellite/carrot" to a "remote-plugin" vocabulary, and adds a direct
invokeWorker/tailWorkerEventsAPI to theRemotePluginHost. The UI onboarding states are removed in favour of a refactoredVoicePrefixStepscomponent, andlsinplugin-coding-toolsgains a capability-router fast path so mobile can delegate filesystem listings to Eliza Cloud.E2BSatelliteCapabilityRouterServiceand >40 env-variable names renamed; URL settings keep backward-compat fallbacks.RemotePluginHostextended – newinvokeWorker,tailWorkerEvents, and ring-buffered event log; service singletons inrpc-handlers.tsmoved to module-level.lsrouting – capability-router fast path with local-FS fallback onCAPABILITY_UNAVAILABLE.Confidence Score: 5/5
Safe to merge; all findings are naming or type-strictness nits with no runtime impact.
The core logic changes are all correct. The four flagged items are an env-variable name using old vocabulary, a redundant wire-type field, its mirror in the RPC schema, and a loosened TypeScript index-signature that could mask future mutation bugs but has no runtime consequence today.
packages/app-core/platforms/electrobun/src/native/remote-plugin-host.ts and packages/app-core/platforms/electrobun/src/rpc-schema.ts carry the redundant satelliteId field and old-vocabulary env var.
Important Files Changed
Comments Outside Diff (1)
packages/ui/src/components/onboarding/VoicePrefixGate.tsx, line 523-537 (link)refreshVoiceBundleReadinessis listed in the dependency array, but the effect body never invokes it — it defines its own innerrefreshclosure that callsloadVoiceBundleReadinessdirectly. BecauserefreshVoiceBundleReadinessis stable (emptyuseCallbackdeps), this doesn't cause extra re-runs, but the dependency is misleading and thealiveguard inside the effect duplicates logic already present inrefreshVoiceBundleReadiness.Reviews (4): Last reviewed commit: "fix: update capability naming audit self..." | Re-trigger Greptile