M1: Browser use in the platform-hosted virtual desktop - #42624
Conversation
…rough velay (#41949) Claude-Session: https://claude.ai/code/session_012YDF6hHcX7aPZsc6TuByhy Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012YDF6hHcX7aPZsc6TuByhy Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…top/stream (#41952) * feat(assistant): on-demand pod desktop session streamed over /v1/desktop/stream Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012YDF6hHcX7aPZsc6TuByhy * chore(feature-flags): align pod-desktop registry wording with the web client entry Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012YDF6hHcX7aPZsc6TuByhy --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…stream tests, comment trims (#41957) Claude-Session: https://claude.ai/code/session_012YDF6hHcX7aPZsc6TuByhy Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…hromium, 4xxx close codes (#41966) Claude-Session: https://claude.ai/code/session_012YDF6hHcX7aPZsc6TuByhy Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…hared gateway WS resolver (#41960) * fix(web): pod desktop clipboard leak, 4xxx close codes, lazy noVNC, shared gateway WS resolver Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012YDF6hHcX7aPZsc6TuByhy * fix(web): drop stray generated-client symlink from the commit Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012YDF6hHcX7aPZsc6TuByhy * fix(web): spell out desktop end-reason keys for the catalog-usage guard Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012YDF6hHcX7aPZsc6TuByhy --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…utdown latch, image and env hardening (#41985) Claude-Session: https://claude.ai/code/session_012YDF6hHcX7aPZsc6TuByhy Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
… comments (#41983) Claude-Session: https://claude.ai/code/session_012YDF6hHcX7aPZsc6TuByhy Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…stream suite (#41982) Claude-Session: https://claude.ai/code/session_012YDF6hHcX7aPZsc6TuByhy Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…al (#41992) Claude-Session: https://claude.ai/code/session_012YDF6hHcX7aPZsc6TuByhy Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…ways releases on start failure (#41993) Claude-Session: https://claude.ai/code/session_012YDF6hHcX7aPZsc6TuByhy Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
The flag is not in LaunchDarkly yet, so this is a pure rename with no migration or compatibility shim. - registry id/key `pod-desktop` -> `assistant-desktop`, label `Pod Desktop` -> `Assistant Desktop`, bundled copies resynced - `isPodDesktopEnabled` -> `isAssistantDesktopEnabled` and the flag constant in `assistant/src/desktop/desktop-feature.ts` - web selector `useAssistantFeatureFlagStore.use.podDesktop()` -> `.assistantDesktop()` - `PodDesktopAffordance` -> `AssistantDesktopAffordance`, files renamed to `assistant-desktop-affordance.tsx` / `.test.tsx` - i18n `podDesktop.*` -> `assistantDesktop.*` in all five locale catalogs and in the desktop-panel key map, keys still spelled out for the catalog guard - ARCHITECTURE.md section "Pod Desktop Stream" -> "Assistant Desktop Stream", plus the prose that carried the old name The route `/v1/desktop/stream`, the `assistant/src/desktop/` directory and the web `domains/chat/desktop/` directory are unchanged. Claude-Session: https://claude.ai/code/session_012YDF6hHcX7aPZsc6TuByhy Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
openbox alone draws no panel, so the streamed desktop had no taskbar, window list or clock. Launch tint2 with its stock config after the window manager, preflight the binary with the other three, and treat a panel exit as non-fatal: the desktop keeps running, and teardown still kills it. Claude-Session: https://claude.ai/code/session_012YDF6hHcX7aPZsc6TuByhy Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…iewer (#42124) * feat(assistant): dock-style desktop taskbar with pinned apps, wider viewer Generate a tint2rc at desktop start instead of running tint2 stock: a bottom-centered floating dock with rounded translucent chrome, icon-only launchers and window buttons, and generated .desktop launchers for Playwright's Chromium and xterm. Widen the viewer modal to min(96vw, 1600px) and shrink its header to a 36px band. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012YDF6hHcX7aPZsc6TuByhy * feat(assistant): run a compositor so the dock renders rounded tint2 only gets an ARGB visual when a compositor is already running, so the rounded translucent dock config rendered as an opaque square strip. Spawn xcompmgr right after openbox and before the dock, as a cosmetic child whose spawn failure and exit are logged and left alone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012YDF6hHcX7aPZsc6TuByhy --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
noVNC is MPL-2.0, which asks that recipients of the executable form be told the licence and where to get the source. The bundler preserves its copyright headers but nothing carried the licence text or a durable source link. Adds a notice served from the web build root, so it travels with the bundle rather than living only in the repo. Records the component, version, copyright, licence, the pinned v1.7.0 source and npm tarball, and an explicit "not modified by Vellum", followed by noVNC's own notice (which says which of its files are MPL and which are more permissive) and the full MPL-2.0 text. The macOS and Windows clients package this same web build so they carry the file; iOS and Android load the hosted web app, so they reach it over the network. Claude-Session: https://claude.ai/code/session_012YDF6hHcX7aPZsc6TuByhy Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…laywright (#42169) The shared frame pump behind /v1/stt/stream, /v1/watch/stream and /v1/desktop/stream closed both sides with 1011 whenever a downstream send returned 0. That teardown is desktop-specific: RFB is an ordered byte stream with no resync, so a lost frame corrupts the framebuffer for good, while the two JSON routes carry self-contained lifecycle and transcript frames. - `closeOnDroppedFrame` is now an opt-in on the handler factory, taken only by the desktop stream. stt and watch log and carry on, at debug rather than warn: the usual trigger is a runtime frame landing just after the browser socket closed, which is benign and once per session. - Bun's send returns a byte count, so an empty frame also returns 0. A drop now requires a non-empty payload. The image bakes Chromium from `/app/assistant`'s pinned playwright, but the runtime installed a floating one into the external dir. A different playwright looks for a different browser build id, so the bake bought a pod nothing and the first browser call or PDF export re-downloaded Chromium. The runtime install now asks for the version `assistant/package.json` pins, and replaces a copy left at any other version. Also restores the dropped live-voice assertion that a resolve without a conversationId produces a URL with no conversationId parameter. Claude-Session: https://claude.ai/code/session_012YDF6hHcX7aPZsc6TuByhy Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Preserve display/window selection through the shared WebSocket resolver and runtime upgrade parser, along with main's velay bridge authentication checks. Add regression coverage for Watch target validation and independent desktop, voice, dictation, Watch, and Twilio connections through the velay bridge. Validation: 498 targeted tests; assistant, gateway, and web type checks; scoped lint and formatting checks; web production build. The pre-commit secret scan passed. Its generic-examples scan rejected three fixtures already on main. The same scanner passed against this PR's staged diff from main; all remaining applicable checks were run separately.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1107f9dc14
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bb9d565ebc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Codex Review: Didn't find any major issues. Can't wait for the next one! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review Please review commit b85e1c1, focusing on stale Chrome singleton cleanup after container replacement and active-turn browser routing. Recovery was reproduced in a fresh package-free Linux container with an existing profile, then verified after the fix with the saved session cookie intact. Live-owner and existing-socket cases have regression coverage. |
|
Codex Review: Didn't find any major issues. Nice work! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
Scope audit: the recent first-use, cursor and recovery commits added 844 lines across five non-merge commits. The intervening merge's additions belong to main. The full feature diff is 4,654 additions after this cleanup: 2,082 runtime lines, 2,252 test/smoke lines, and 320 docs/help/schema/translation lines. The cleanup shortens repeated CLI and per-turn guidance and consolidates this PR description. Shared ownership/takeover code remains because M1 browser automation requires it and M2 consumes the same implementation. No additional computer-control features were pulled into M1. @codex review d27db08. Please focus on the recovery changes in b85e1c1: preservation of live Chrome profile locks, restored packages with persistent profiles, and current-turn browser routing. The final commit only shortens guidance. Local recovery tests and the 70 affected CLI/tool-resolution tests pass. |
|
Codex Review: Didn't find any major issues. Chef's kiss. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review acc0c64. Scope cleanup only: remove computer-only lease IDs, native observation sequencing and cleanup extension options from M1. They remain in the single shared lease implementation in stacked M2 #42714. Remove the flag-helper test duplicated by setup-route gate tests. Keep browser ownership, takeover, setup cancellation and IPC disconnect cancellation intact. Do not reintroduce the deferred headless-browser recovery changes. 74 focused M1 tests, type checking and lint pass. The PR diff shrinks by 72 lines. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: acc0c64bda
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| constructor(protected readonly runCommand = runDesktopCommand) {} | ||
| async setViewerInput(enabled: boolean): Promise<void> { | ||
| const value = enabled ? "1" : "0"; | ||
| await this.runCommand("tigervncconfig", [ |
There was a problem hiding this comment.
When a platform image provides only vncconfig, which resolveDesktopBinaries() explicitly supports as a fallback, every lease acquisition still invokes /usr/bin/tigervncconfig. Setup therefore reports ready and starts the desktop successfully, but the first browser action fails with ENOENT; cleanup invokes the same missing binary and can leave the lease stuck in the assistant state. Reuse the configurator selected by desktop dependency resolution instead of hardcoding this executable.
Useful? React with 👍 / 👎.
|
|
||
| ### Browser CLI surface defaults | ||
|
|
||
| The browser execute and tab routes share `browser/virtual-desktop-target.ts`. Platform-hosted web guardian conversations use installed, enabled virtual desktop Chrome by default. The shared `isVirtualDesktopEnabled` gate requires both `IS_PLATFORM` and `IS_CONTAINERIZED` alongside the `assistant-desktop` flag. Native renderer turns also use the `web` transport, so the frozen turn `clientOs` excludes native apps from automatic streamed-browser selection. Native apps retain their existing backend selection and fallback behavior. Explicit desktop/backend/client targets and existing personal-browser sessions override the surface default. Disabled or uninstalled desktop support retains the existing browser path; selection never triggers installation. |
There was a problem hiding this comment.
Fresh evidence in the reviewed tree is that this rule again says uninstalled support keeps the existing browser path and never triggers installation, while DesktopControlLease.runBrowser() still calls ensureReady() and continues the original browser action after installation. This contradicts the retained first-use behavior and the CLI and architecture documentation, so future work following this mandatory guidance could reintroduce the fallback regression that the earlier review thread addressed.
Useful? React with 👍 / 👎.
1 participant
Summary
assistant-desktop. First browser use and opening picture-in-picture start automatic installation, with serialized automation and cancellation.assistant browser --virtual-desktop, retaining--desktopand existing API/storage names for compatibility. The centered title is Desktop. Picture-in-picture stays view-only; the expanded desktop accepts direct input without ownership buttons, status banners or a handoff API. Native computer actions remain in stacked PR M2: Unified computer use for connected and assistant desktops #42714.Original prompt
Split browser use and computer use into independently reviewable milestones while preserving the existing browser CLI and native client behavior. Keep M1 in this PR with visible cursor feedback, platform-only feature gating and automatic installation.
Current state
The implementation is based on the requested rollback to
bb9d565ebce59cc4caa04b388456ec367d36c41a. Recovery and guidance changes after that point remain reverted. A subsequent scope cleanup moves computer-only lease IDs, observation sequencing and native-input cleanup hooks into M2, and removes a flag-helper test duplicated by the setup-route gate tests. Explicit handoff UI, APIs, viewer input locks and their unused state are removed. Cancellation and conversation isolation remain in M1.The reported behavior where the assistant still launches headless Chrome remains unresolved and is deferred. Prior validation claims for the reverted recovery changes do not apply to this PR's current state.
Cleanup validation
After handoff removal, 69 focused tests pass across browser sessions, CDP clients, streaming access gates, automatic setup, PiP/modal continuity and viewer input. Assistant and web type checking, targeted lint and OpenAPI regeneration pass. The Linux smoke script was updated for direct input but was not rerun for this change.
Milestone boundary
M1 serializes automation between assistant conversations. Users can interact directly with browser chrome, native dialogs and other apps in the expanded desktop. Opening or clicking the viewer does not pause automation. Merge this PR first, then retarget #42714 to main.