feat: add browser use tool#2193
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a new built-in Browser Use tool provider that controls a managed, visible Chrome-for-Testing session, and wires it into the UI/provider selection and built-in tool initialization.
Changes:
- Adds “Browser Use” to the web UI tool/provider options and default test payloads.
- Registers the “Browser Use” provider in backend provider construction and built-in tool seeding.
- Introduces
tool/browser_use.go: persistent visible Chrome session + MCP tools (open, snapshot, click, type, press, tabs, switch tab, play media, close), including runtime Chrome-for-Testing download/install.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| web/src/common/TestToolWidget.js | Adds a default “Browser Use” test JSON entry. |
| web/src/ToolEditPage.js | Enables “Browser Use” type handling and proxy toggle visibility. |
| web/src/Setting.js | Adds “Browser Use” provider metadata + type/subtype options. |
| tool/tool.go | Registers “Browser Use” in NewProvider. |
| tool/browser_use.go | Implements the Browser Use provider and built-in tools, including Chrome-for-Testing download/launch/session management. |
| object/init.go | Seeds a built-in “Browser Use” tool configuration. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| func downloadBrowserUseArchiveOnce(downloadURL, archivePath string) error { | ||
| if err := os.MkdirAll(filepath.Dir(archivePath), 0755); err != nil { | ||
| return fmt.Errorf("failed to create browser download directory: %w", err) | ||
| } | ||
|
|
||
| client := &http.Client{Timeout: 10 * time.Minute} |
There was a problem hiding this comment.
When EnableProxy is set for Browser Use, Chrome itself is configured to use the SOCKS5 proxy, but the HTTP requests used to fetch Chrome-for-Testing metadata and to download the archive do not use the app’s proxy transport. In environments where outbound internet requires the proxy, the managed browser download will fail even though the tool is “proxy enabled”. Consider using proxy.ProxyHttpClient.Transport (similar to Web Fetch/Search) for these HTTP clients when enableProxy is true, and plumb that flag into ensureBrowserUseChromeForTesting/download helpers.
| func downloadBrowserUseArchiveOnce(downloadURL, archivePath string) error { | |
| if err := os.MkdirAll(filepath.Dir(archivePath), 0755); err != nil { | |
| return fmt.Errorf("failed to create browser download directory: %w", err) | |
| } | |
| client := &http.Client{Timeout: 10 * time.Minute} | |
| func newBrowserUseDownloadHTTPClient() *http.Client { | |
| client := &http.Client{Timeout: 10 * time.Minute} | |
| if proxy.ProxyHttpClient != nil && proxy.ProxyHttpClient.Transport != nil { | |
| client.Transport = proxy.ProxyHttpClient.Transport | |
| } | |
| return client | |
| } | |
| func downloadBrowserUseArchiveOnce(downloadURL, archivePath string) error { | |
| if err := os.MkdirAll(filepath.Dir(archivePath), 0755); err != nil { | |
| return fmt.Errorf("failed to create browser download directory: %w", err) | |
| } | |
| client := newBrowserUseDownloadHTTPClient() |
| func (p *BrowserUseProvider) run(actions ...chromedp.Action) error { | ||
| session := globalBrowserUseManager.get(p) | ||
| session.mu.Lock() | ||
| defer session.mu.Unlock() | ||
|
|
||
| if err := session.ensureLocked(); err != nil { | ||
| return err | ||
| } | ||
|
|
||
| timeoutCtx, cancel := context.WithTimeout(session.ctx, browserUseDefaultTimeout) | ||
| defer cancel() | ||
| return chromedp.Run(timeoutCtx, actions...) | ||
| } |
There was a problem hiding this comment.
Browser Use actions are hard-coded to browserUseDefaultTimeout (30s) via run(), but the tool schemas don’t expose a timeout parameter like web_browser / web_fetch. This can make navigation/snapshots flaky on slower pages and gives callers no way to tune behavior. Consider adding an optional timeout argument (with sane min/max) to the Browser Use builtins and using it when creating the per-call context.
| func ensureBrowserUseChromeForTesting() (string, error) { | ||
| browserUseDownloadMu.Lock() | ||
| defer browserUseDownloadMu.Unlock() | ||
|
|
||
| platform, err := chromeForTestingPlatform() | ||
| if err != nil { | ||
| return "", err | ||
| } | ||
|
|
||
| metadata, err := fetchChromeForTestingMetadata() | ||
| if err != nil { | ||
| return "", err | ||
| } | ||
|
|
||
| channelMeta, ok := metadata.Channels[chromeForTestingDefaultChannel] | ||
| if !ok { | ||
| return "", fmt.Errorf("Chrome for Testing channel %q is not available", chromeForTestingDefaultChannel) | ||
| } | ||
|
|
||
| var downloadURL string | ||
| for _, item := range channelMeta.Downloads.Chrome { | ||
| if item.Platform == platform { | ||
| downloadURL = item.URL | ||
| break | ||
| } | ||
| } | ||
| if downloadURL == "" { | ||
| return "", fmt.Errorf("Chrome for Testing channel %q does not provide a chrome download for platform %q", chromeForTestingDefaultChannel, platform) | ||
| } | ||
|
|
||
| cacheDir := browserUseCacheDir() | ||
| installDir := filepath.Join(cacheDir, strings.ToLower(chromeForTestingDefaultChannel), channelMeta.Version, platform) | ||
| executablePath := browserUseChromeExecutablePath(installDir, platform) | ||
| if fileExists(executablePath) { | ||
| return executablePath, nil | ||
| } | ||
|
|
||
| if err = os.RemoveAll(installDir); err != nil { | ||
| return "", fmt.Errorf("failed to reset Chrome for Testing install directory %s: %w", installDir, err) | ||
| } | ||
| if err = os.MkdirAll(installDir, 0755); err != nil { | ||
| return "", fmt.Errorf("failed to create Chrome for Testing install directory %s: %w", installDir, err) | ||
| } | ||
|
|
||
| archivePath := filepath.Join(cacheDir, "downloads", fmt.Sprintf("chrome-%s-%s.zip", channelMeta.Version, platform)) | ||
| if err = downloadBrowserUseArchive(downloadURL, archivePath); err != nil { | ||
| return "", err | ||
| } | ||
| if err = unzipBrowserUseArchive(archivePath, installDir); err != nil { | ||
| return "", err | ||
| } | ||
| if !fileExists(executablePath) { | ||
| return "", fmt.Errorf("Chrome for Testing executable was not found after install: %s", executablePath) | ||
| } | ||
| if err = os.Chmod(executablePath, 0755); err != nil { | ||
| return "", fmt.Errorf("failed to mark Chrome for Testing executable as runnable: %w", err) | ||
| } | ||
| return executablePath, nil | ||
| } |
There was a problem hiding this comment.
The Chrome-for-Testing download/install path verifies only that the ZIP can be opened; it doesn’t verify the provenance/integrity of the downloaded browser before executing it. Since this code downloads and runs an external binary at runtime, consider adding stronger integrity checks (e.g., pinned version + SHA256 validation from a trusted source, or requiring an operator-provided executable path) to reduce supply-chain risk.
| func (p *BrowserUseProvider) run(actions ...chromedp.Action) error { | ||
| session := globalBrowserUseManager.get(p) | ||
| session.mu.Lock() | ||
| defer session.mu.Unlock() | ||
|
|
||
| if err := session.ensureLocked(); err != nil { | ||
| return err | ||
| } | ||
|
|
||
| timeoutCtx, cancel := context.WithTimeout(session.ctx, browserUseDefaultTimeout) | ||
| defer cancel() | ||
| return chromedp.Run(timeoutCtx, actions...) | ||
| } |
There was a problem hiding this comment.
BrowserUseProvider.run derives its timeout from session.ctx and none of the Browser Use builtins pass/propagate the ctx they receive in Execute. This means request cancellation/deadlines won’t stop in-flight browser actions (navigation/click/type), which can leave the browser mutating state after the caller has aborted and can also leak work under load. Consider threading the Execute context into run/runSession and creating a derived context that cancels when either the request context is done or the per-action timeout elapses.
| if err = os.MkdirAll(s.userDataDir, 0755); err != nil { | ||
| return fmt.Errorf("failed to create browser profile directory %s: %w", s.userDataDir, err) | ||
| } |
There was a problem hiding this comment.
The managed browser profile directory is created with mode 0755. Since this directory is intended to persist cookies/local storage across sessions, 0755 can allow other local users to read it on multi-user machines. Use a more restrictive mode (e.g., 0700) for userDataDir (and ensure created files inherit restrictive permissions) to reduce the risk of credential/session leakage.
| if err = os.MkdirAll(s.userDataDir, 0755); err != nil { | |
| return fmt.Errorf("failed to create browser profile directory %s: %w", s.userDataDir, err) | |
| } | |
| if err = os.MkdirAll(s.userDataDir, 0700); err != nil { | |
| return fmt.Errorf("failed to create browser profile directory %s: %w", s.userDataDir, err) | |
| } | |
| if err = os.Chmod(s.userDataDir, 0700); err != nil { | |
| return fmt.Errorf("failed to secure browser profile directory %s: %w", s.userDataDir, err) | |
| } |
nkanf-dev
force-pushed
the
feat/browser-use
branch
3 times, most recently
from
7f25472 to
3dcd699
Compare
nkanf-dev
force-pushed
the
feat/browser-use
branch
from
3dcd699 to
bc645df
Compare
Adds a built-in Browser Use tool that opens and controls a managed visible Chrome for Testing browser.
Includes support for navigation, snapshots, clicking, typing, tab switching, media playback, and browser lifecycle management.