vellum-ai
diff --git a/‎ARCHITECTURE.md‎
Lines changed: 12 additions & 4 deletions b/‎ARCHITECTURE.md‎
Lines changed: 12 additions & 4 deletions
@@ -810,15 +810,15 @@ graph LR
 
 ## Assistant Desktop Stream
 
-A containerized assistant can serve an interactive desktop on demand. The setup-capable modal installs desktop-only system packages and Google Chrome after the guardian clicks **Install desktop**. Authenticated direct stream requests also start or join the same background setup for clients without the setup UI. A client that times out during installation can reconnect after setup finishes; a disconnected viewer does not start a desktop process tree. `GET /v1/desktop/setup` checks readiness without installing anything; `POST` starts one shared background installation. Both flat and assistant-scoped gateway paths require guardian authentication and proxy to a gateway-service-only runtime route. `assistant:self:desktop` sync invalidations refresh setup status as installation progresses, and a reopened modal or reconnected event stream refetches it. Older assistants returning 404 retain the direct streaming flow.
+An enabled platform-hosted assistant can serve an interactive desktop on demand. Opening the virtual desktop picture-in-picture view automatically starts installation of desktop-only system packages and Google Chrome. The first eligible browser command waits for the same installation job and continues its original action once ready. Setup progress uses the same sync invalidations as the viewer. Cancellation or feature revocation prevents a delayed action. Browser CLI requests opt into IPC cancellation on disconnect so timed-out commands cannot act after setup finishes. Authenticated direct stream requests also start or join the same background setup for clients without the setup UI. A client that times out during installation can reconnect after setup finishes; a disconnected viewer does not start a desktop process tree. `GET /v1/desktop/setup` checks readiness without installing anything; `POST` starts one shared background installation. Both flat and assistant-scoped gateway paths require guardian authentication and proxy to a gateway-service-only runtime route. `assistant:self:desktop` sync invalidations refresh setup status as installation progresses, and a reopened modal or reconnected event stream refetches it. Older assistants returning 404 retain the direct streaming flow.
 
-`desktop-dependencies.ts` installs the X server, window manager, dock, compositor, clipboard bridge, terminal, wallpaper setter (`feh`), fonts and Chrome libraries through image-root `/usr/bin/apt-get` with `--no-upgrade` and `--no-remove`. The installer uses only system command paths, bypassing Kata persistent-apt wrappers so binaries, X assets and shared libraries live in the same filesystem. Packages must be installed again when a Kata save or container recreation discards that root; Chrome and its profile remain in persistent storage. Google Chrome is an exact-version, SHA-256-verified download for Linux x64 or ARM64, extracted under the assistant's internal external-dependency directory. Extraction does not run Chrome package scripts, register its repository or change the system's default browser. Successful setup is recorded only after Chrome runs; readiness also checks the desktop binaries and X fonts so a recreated container offers setup again. Configured `NODE_EXTRA_CA_CERTS` are combined with the system CA bundle for apt and Chrome downloads without changing system trust. Installation errors remain retryable. The base image carries no desktop-only packages or baked browser; existing browser-tool and PDF installations retain their own Playwright behavior.
+`desktop-dependencies.ts` installs the X server, window manager, dock, compositor, clipboard bridge, terminal, wallpaper setter (`feh`), fonts and Chrome libraries through image-root `/usr/bin/apt-get` with `--no-upgrade` and `--no-remove`. The installer uses only system command paths, bypassing Kata persistent-apt wrappers so binaries, X assets and shared libraries live in the same filesystem. Packages must be installed again when a Kata save or container recreation discards that root; Chrome and its profile remain in persistent storage. Google Chrome is an exact-version, SHA-256-verified download for Linux x64 or ARM64, extracted under the assistant's internal external-dependency directory. Extraction does not run Chrome package scripts, register its repository or change the system's default browser. Successful setup is recorded only after Chrome runs; readiness also checks the desktop binaries and X fonts so a recreated container installs missing components on first use. Configured `NODE_EXTRA_CA_CERTS` are combined with the system CA bundle for apt and Chrome downloads without changing system trust. Installation errors remain retryable. The base image carries no desktop-only packages or baked browser; existing browser-tool and PDF installations retain their own Playwright behavior.
 
-`DesktopSessionManager` owns `Xtigervnc` on display `:99` with VNC on `localhost:5999`, `openbox`, `xcompmgr`, `plank`, `tigervncconfig` and Google Chrome. A Python standard-library helper supervises Openbox and handles native title-bar move requests by restoring maximized windows before continuing the drag. The helper uses the installed X11 library and shares the window manager process group for shutdown. Chrome starts directly, without Playwright or automation switches, using the existing `data/desktop-profile` directory. The dock configuration and launcher paths remain under `data/desktop-panel`.
+`DesktopSessionManager` owns `Xtigervnc` on display `:99` with VNC on `localhost:5999`, `openbox`, `xcompmgr`, `plank`, `tigervncconfig` and Google Chrome. A Python standard-library helper supervises Openbox and handles native title-bar move requests by restoring maximized windows before continuing the drag. The helper uses the installed X11 library and shares the window manager process group for shutdown. Chrome starts directly with a loopback-only CDP port, using the existing `data/desktop-profile` directory. The assistant verifies that the listener belongs to its managed Chrome process before connecting. The dock configuration and launcher paths remain under `data/desktop-panel`.
 
 Openbox loads a generated `data/desktop-panel/openbox.xml` with one workspace and no workspace-switching bindings or menus. The generator adapts the existing user config or the installed system config, preserving themes, window controls, shortcuts and application menus in managed copies. Nested XML includes are adapted into managed copies with their original lookup bases and XPointer selections preserved. Source files remain unchanged. If a custom XML configuration cannot be adapted, Openbox loads the existing source config and a warning is logged, preserving desktop availability. Restored windows are assigned to the sole workspace, and session-manager restoration is disabled. Each process tree owns a fresh X display; viewer reconnects retain its existing windows. The config is regenerated at desktop start, so existing installations adopt it without changing browser profiles or dock preferences.
 
-Children receive only an allowlisted environment. One viewer holds the slot at a time; the tree lingers five minutes after disconnect. Chrome launches once when the desktop starts. Closing or crashing Chrome leaves the desktop running, and viewer reconnects keep it closed; the dock launcher can reopen it. Required child failures tear down the tree, while cosmetic dock/compositor failures leave the desktop running. Dock startup failures and exits receive up to three restart attempts per desktop session, one second apart. Shutdown uses SIGTERM followed by SIGKILL after a two-second grace, and a subsequent start waits for teardown. The `assistant-desktop` flag and `IS_CONTAINERIZED` gate both setup and streaming. The stream rechecks the gate after asynchronous setup and before forwarding either direction of traffic; revocation closes the viewer with `4008` and releases its slot. The companion platform PR adds authenticated desktop routing through velay; it does not change pod memory or shared-memory provisioning.
+Children receive only an allowlisted environment. One viewer holds the slot at a time; the tree lingers five minutes after disconnect. Chrome launches once when the desktop starts. Closing or crashing Chrome leaves the desktop running, and viewer reconnects keep it closed; the dock launcher or an explicit browser CLI action can reopen it. Required child failures tear down the tree, while cosmetic dock/compositor failures leave the desktop running. Dock startup failures and exits receive up to three restart attempts per desktop session, one second apart. Shutdown uses SIGTERM followed by SIGKILL after a two-second grace, and a subsequent start waits for teardown. The `assistant-desktop` flag, `IS_PLATFORM` and `IS_CONTAINERIZED` gate both setup and streaming. Self-hosted Docker assistants cannot use the virtual desktop. The stream rechecks the gate after asynchronous setup and before forwarding either direction of traffic; revocation closes the viewer with `4008` and releases its slot. The companion platform PR adds authenticated desktop routing through velay; it does not change pod memory or shared-memory provisioning.
 
 Plank runs under `dbus-run-session` with private XDG configuration/data paths and the keyfile settings backend; its BAMF matcher shares that session bus and process group. The manager retains each exited dock group during recovery so its applications keep running, then clears all surviving groups at teardown. Shutdown cancels pending dock restarts. Chrome and Terminal launchers use their real X11 identities, with Chrome's official packaged icon and profile-specific identity. Pinned launchers represent running applications, with Plank providing focus, minimize/restore, window selection, and explicit new-window gestures. Default pins and preferences are published atomically on first use; subsequent starts refresh managed launcher paths while preserving user customization. An installation missing Plank or BAMF requests on-demand setup.
 
@@ -828,6 +828,14 @@ Before launching Chrome or exposing its dock launcher, the Linux container sessi
 
 **Transport.** `/v1/desktop/stream` is a pure RFB byte pipe: after the upgrade, every frame in both directions is binary and `DesktopStreamBridge` (`desktop-stream-bridge.ts`) pumps it to and from the VNC port, buffering client bytes that arrive before that socket is up. Nothing is signaled in-band; outcomes are close codes in the application range so they can neither collide with velay's own `1013` nor be remapped by the gateway's velay bridge. The manager decides them and the bridge only relays (`DesktopLoss`, through the viewer-slot result, `onDesktopLost`, or the `DesktopStartError` a start rejects with): `4008` desktop disabled or unsupported on this daemon, `4013` another viewer holds the slot, `4011` the desktop failed to start, died under the viewer, or the viewer fell too far behind (a dropped `ws.send`), and the standard `1001` when the runtime is shutting down, whether the socket arrived after shutdown began or a live viewer is cut off by it. On the managed path velay's bridge carries the runtime's `1001` as `4001` and the gateway's `1011` as `4011`, both of which the panel treats as retryable endings. The daemon upgrade is gated exactly as `/v1/watch/stream` (private-network peer and origin, gateway service token, one shared `upgradeRuntimeStream` path); the feature gate runs after the upgrade because the gateway relays close codes, not HTTP statuses, to the browser. VNC needs no password: only same-pod processes can reach the loopback port, and the authenticated upgrade is the only bridge to it.
 
+### Virtual desktop browser use
+
+`assistant browser --virtual-desktop` controls the managed Chrome window. Web conversations select it automatically when available; native desktop clients retain their existing browser behavior and require an explicit choice to use streamed Chrome. The shared browser target resolver honors explicit modes, client targets and existing personal-browser sessions. It distinguishes Electron from plain web using the frozen turn client OS while retaining the original transport identity. Usage guidance lives in `assistant browser --help`. Platform hosting, the default-off `assistant-desktop` flag, completed automatic installation and identified guardian conversation gate automation. No connected host desktop client is needed. Existing host computer-use tools keep their connected-client routing.
+
+`DesktopAutomationLease` serializes browser automation and binds ownership to one conversation and actor. The browser CLI borrows a scoped direct CDP client and dispatches through the existing browser operation handlers. Snapshot references use a separate desktop namespace and are invalidated on navigation, tab changes and release. A page overlay animates the CDP pointer in the stream. Page screenshots come directly from Chrome as color JPEGs. Native application control and whole-desktop screenshots are outside this browser interface. See [desktop browser CLI](assistant/docs/desktop-browser-cli.md).
+
+An automation slot keeps the desktop alive independently of the viewer. The picture-in-picture preview is view-only; opening the desktop allows direct mouse, keyboard and clipboard input without a handoff step. Viewer interaction does not pause browser automation. Cancellation, desktop loss, inactivity and `detach` release the automation slot and clean up held browser input. No control state or screenshots are persisted outside normal tool history.
+
 ## Screen Annotation
 
 The assistant points at things on the screen the user is sharing with a call, so they can go and do the thing themselves. It is the opposite errand from computer use and shares none of its actions: nothing here clicks, types or takes the mouse. The bundled `screen-annotation` skill (`assistant/src/config/bundled-skills/screen-annotation/`) offers two tools, `screen_point_at` and `screen_clear_marks`, and a request replaces whatever is currently drawn. Clearing is its own tool because it is a thing the model decides to do rather than an argument shape it has to remember; on the wire it is the same request carrying no marks.