Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -685,6 +685,27 @@ The `chat` command translates natural language instructions into agent-browser c

The Chat tab is always visible in the dashboard. When `AI_GATEWAY_API_KEY` is set, the Rust server proxies requests to the gateway and streams responses back using the Vercel AI SDK's UI Message Stream protocol. Without the key, sending a message shows an error inline.

### Exa Web Search

The `chat` command supports [Exa](https://exa.ai) as a built-in web search tool. When `EXA_API_KEY` is set, the AI can search the web using Exa instead of navigating to a search engine. This is faster and returns cleaner results with titles and text snippets.

```bash
export EXA_API_KEY=your_exa_api_key_here
```

Once configured, the AI will automatically use Exa when it needs to look something up. You can also ask it directly:

```bash
agent-browser chat "search for the latest news about AI agents"
agent-browser chat "find me some good rust CLI libraries"
```

The AI can combine Exa search with browser automation -- for example, searching for pages and then opening the results for deeper exploration.

| Variable | Description |
| ------------- | ------------------------------------ |
| `EXA_API_KEY` | Exa API key for web search in chat |

## Configuration

Create an `agent-browser.json` file to set persistent defaults instead of repeating flags on every command.
Expand Down
34 changes: 25 additions & 9 deletions cli/src/chat.rs
Original file line number Diff line number Diff line change
Expand Up @@ -218,7 +218,7 @@ async fn run_chat_turn(
}
};

let tools: Value = serde_json::from_str(chat::CHAT_TOOLS).unwrap();
let tools: Value = serde_json::from_str(&chat::chat_tools_json()).unwrap();
let url = format!("{}/v1/chat/completions", gateway_url);
let client = chat::http_client();

Expand Down Expand Up @@ -341,21 +341,36 @@ async fn run_chat_turn(
}
}

for (tc_id, _tc_name, tc_args) in &tool_calls {
for (tc_id, tc_name, tc_args) in &tool_calls {
let input: Value = serde_json::from_str(tc_args).unwrap_or(json!({}));
let command = input.get("command").and_then(|c| c.as_str()).unwrap_or("");

if !json_mode && verbosity != Verbosity::Quiet {
eprintln!("{}", color::dim(&format!("> {}", command)));
}
let result = if tc_name == "exa_web_search" {
let query = input.get("query").and_then(|q| q.as_str()).unwrap_or("");
let num = input.get("numResults").and_then(|n| n.as_u64());

if !json_mode && verbosity != Verbosity::Quiet {
eprintln!("{}", color::dim(&format!("> exa search: {}", query)));
}

match tokio::time::timeout(tool_timeout, chat::execute_exa_search(query, num)).await
{
Ok(r) => r,
Err(_) => "Exa search timed out after 60 seconds.".to_string(),
}
} else {
let command = input.get("command").and_then(|c| c.as_str()).unwrap_or("");

if !json_mode && verbosity != Verbosity::Quiet {
eprintln!("{}", color::dim(&format!("> {}", command)));
}

let result =
match tokio::time::timeout(tool_timeout, chat::execute_chat_tool(session, command))
.await
{
Ok(r) => r,
Err(_) => "Tool execution timed out after 60 seconds.".to_string(),
};
}
};

if !json_mode && verbosity == Verbosity::Verbose {
for line in result.lines() {
Expand All @@ -364,7 +379,8 @@ async fn run_chat_turn(
}

all_tool_calls.push(json!({
"command": command,
"tool": tc_name,
"input": input,
"output": result
}));

Expand Down
131 changes: 119 additions & 12 deletions cli/src/native/stream/chat.rs
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,12 @@ pub(crate) fn get_system_prompt() -> &'static str {
sections.push_str(&format!("\n\n<skill name=\"{}\">\n{}\n</skill>", name, body.trim()));
}

let exa_section = if std::env::var("EXA_API_KEY").is_ok() {
"\n\nWEB SEARCH:\n- You have access to the exa_web_search tool for fast web search.\n- Use exa_web_search when the user wants to find information, look something up, or discover pages. It returns clean results with titles and text snippets.\n- After searching, you can use agent_browser to open any of the result URLs for deeper exploration.\n- Do NOT navigate to google.com or other search engines — use exa_web_search instead."
} else {
""
};

format!(
r#"You are an AI assistant that controls a browser through agent-browser. You have an active browser session, but you can also create new sessions.

Expand All @@ -148,12 +154,22 @@ RULES:
- To use a different browser engine: add `--engine <engine>` (e.g. `agent-browser --session lp-session --engine lightpanda open https://example.com`). Supported engines: chrome (default), lightpanda.

The following skill references describe agent-browser capabilities in detail. Use them when deciding which commands to run and how to approach tasks.
{sections}"#,
{exa_section}{sections}"#,
)
})
}

pub(crate) const CHAT_TOOLS: &str = r#"[{"type":"function","function":{"name":"agent_browser","description":"Execute an agent-browser command. Runs against the active session by default. Add --session <name> to target or create a different session, and --engine <engine> to choose a browser engine.","parameters":{"type":"object","properties":{"command":{"type":"string","description":"The command to execute, e.g. 'agent-browser open https://google.com' or 'agent-browser --session new-session open https://example.com' or 'agent-browser snapshot -i' or 'agent-browser click @e3'"}},"required":["command"]}}}]"#;
const AGENT_BROWSER_TOOL: &str = r#"{"type":"function","function":{"name":"agent_browser","description":"Execute an agent-browser command. Runs against the active session by default. Add --session <name> to target or create a different session, and --engine <engine> to choose a browser engine.","parameters":{"type":"object","properties":{"command":{"type":"string","description":"The command to execute, e.g. 'agent-browser open https://google.com' or 'agent-browser --session new-session open https://example.com' or 'agent-browser snapshot -i' or 'agent-browser click @e3'"}},"required":["command"]}}}"#;

const EXA_SEARCH_TOOL: &str = r#"{"type":"function","function":{"name":"exa_web_search","description":"Search the web using Exa AI. Returns relevant URLs with titles and text snippets. Use this instead of navigating to a search engine when the user needs to find information, look something up, or discover relevant pages.","parameters":{"type":"object","properties":{"query":{"type":"string","description":"The search query"},"numResults":{"type":"integer","description":"Number of results to return (default 5, max 20)"}},"required":["query"]}}}"#;

pub(crate) fn chat_tools_json() -> String {
if std::env::var("EXA_API_KEY").is_ok() {
format!("[{},{}]", AGENT_BROWSER_TOOL, EXA_SEARCH_TOOL)
} else {
format!("[{}]", AGENT_BROWSER_TOOL)
}
}

pub(crate) const COMPACT_THRESHOLD_CHARS: usize = 200_000;
pub(crate) const KEEP_RECENT_MESSAGES: usize = 6;
Expand Down Expand Up @@ -452,6 +468,92 @@ const ALLOWED_COMMANDS: &[&str] = &[
"session",
];

const EXA_API_URL: &str = "https://api.exa.ai/search";

/// Execute an Exa web search and return formatted results.
pub(crate) async fn execute_exa_search(query: &str, num_results: Option<u64>) -> String {
let api_key = match std::env::var("EXA_API_KEY") {
Ok(k) => k,
Err(_) => return "EXA_API_KEY not set. Set the EXA_API_KEY environment variable to enable web search.".to_string(),
};

let num = num_results.unwrap_or(5).min(20).max(1);
let body = json!({
"query": query,
"numResults": num,
"type": "auto",
"contents": {
"highlights": true
}
});

let client = http_client();
let result = client
.post(EXA_API_URL)
.header("x-api-key", &api_key)
.header("Content-Type", "application/json")
.header("x-exa-integration", "vercel-agent-browser")
.body(body.to_string())
.send()
.await;

let response = match result {
Ok(r) => r,
Err(e) => return format!("Exa search request failed: {}", e),
};

if !response.status().is_success() {
let status = response.status();
let body_text = response.text().await.unwrap_or_default();
return format!("Exa search failed ({}): {}", status, body_text);
}

let resp_json: Value = match response.json().await {
Ok(v) => v,
Err(e) => return format!("Failed to parse Exa response: {}", e),
};

let results = resp_json.get("results").and_then(|r| r.as_array());
let Some(results) = results else {
return "No results found.".to_string();
};

if results.is_empty() {
return "No results found.".to_string();
}

let mut output = String::new();
for (i, result) in results.iter().enumerate() {
let title = result
.get("title")
.and_then(|t| t.as_str())
.unwrap_or("(no title)");
let url = result.get("url").and_then(|u| u.as_str()).unwrap_or("");
output.push_str(&format!("{}. {} - {}\n", i + 1, title, url));

if let Some(highlights) = result.get("highlights").and_then(|h| h.as_array()) {
for highlight in highlights {
if let Some(text) = highlight.as_str() {
let trimmed = if text.len() > 300 {
let mut end = 300;
while !text.is_char_boundary(end) {
end -= 1;
}
format!("{}...", &text[..end])
} else {
text.to_string()
};
output.push_str(&format!(" {}", trimmed));
output.push('\n');
}
}
}
output.push('\n');
}

output.trim_end().to_string()
}

const ALLOWED_GLOBAL_FLAGS: &[&str] = &["--session", "--engine"];

pub(crate) async fn execute_chat_tool(session: &str, command: &str) -> String {
Expand Down Expand Up @@ -771,7 +873,7 @@ pub(super) async fn handle_chat_request(
}
}

let tools: Value = serde_json::from_str(CHAT_TOOLS).unwrap();
let tools: Value = serde_json::from_str(&chat_tools_json()).unwrap();
let url = format!("{}/v1/chat/completions", gateway_url);
let client = http_client();

Expand Down Expand Up @@ -918,7 +1020,6 @@ pub(super) async fn handle_chat_request(

for (tc_id, tc_name, tc_args) in &tool_calls {
let input: Value = serde_json::from_str(tc_args).unwrap_or(json!({}));
let command = input.get("command").and_then(|c| c.as_str()).unwrap_or("");

let ev = format!(
"data: {}\n\n",
Expand All @@ -931,14 +1032,20 @@ pub(super) async fn handle_chat_request(
);
let _ = stream.write_all(ev.as_bytes()).await;

let result = match tokio::time::timeout(
TOOL_TIMEOUT,
execute_chat_tool(&session, command),
)
.await
{
Ok(r) => r,
Err(_) => "Tool execution timed out after 60 seconds.".to_string(),
let result = if tc_name == "exa_web_search" {
let query = input.get("query").and_then(|q| q.as_str()).unwrap_or("");
let num = input.get("numResults").and_then(|n| n.as_u64());
match tokio::time::timeout(TOOL_TIMEOUT, execute_exa_search(query, num)).await {
Ok(r) => r,
Err(_) => "Exa search timed out after 60 seconds.".to_string(),
}
} else {
let command = input.get("command").and_then(|c| c.as_str()).unwrap_or("");
match tokio::time::timeout(TOOL_TIMEOUT, execute_chat_tool(&session, command)).await
{
Ok(r) => r,
Err(_) => "Tool execution timed out after 60 seconds.".to_string(),
}
};

let frontend_output = enrich_tool_output(&result);
Expand Down
1 change: 1 addition & 0 deletions cli/src/output.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2963,6 +2963,7 @@ Environment:
AI_GATEWAY_URL Vercel AI Gateway base URL (default: https://ai-gateway.vercel.sh)
AI_GATEWAY_API_KEY API key for the AI Gateway (enables chat command and dashboard AI chat)
AI_GATEWAY_MODEL Default AI model (default: anthropic/claude-sonnet-4.6, or --model flag)
EXA_API_KEY Exa API key for web search in chat (https://exa.ai)

Install:
npm install -g agent-browser # npm
Expand Down
1 change: 1 addition & 0 deletions docs/src/app/dashboard/page.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -156,5 +156,6 @@ The Rust server proxies chat requests from the dashboard to the Vercel AI Gatewa
<tr><td><code>AI_GATEWAY_URL</code></td><td>Vercel AI Gateway base URL.</td><td><code>https://ai-gateway.vercel.sh</code></td></tr>
<tr><td><code>AI_GATEWAY_API_KEY</code></td><td>API key for the AI Gateway. Required to enable AI chat responses.</td><td>(none)</td></tr>
<tr><td><code>AI_GATEWAY_MODEL</code></td><td>Default AI model for chat requests.</td><td><code>anthropic/claude-sonnet-4.6</code></td></tr>
<tr><td><code>EXA_API_KEY</code></td><td>Exa API key. When set, the AI can search the web via <a href="https://exa.ai">Exa</a> instead of navigating to a search engine.</td><td>(none)</td></tr>
</tbody>
</table>
8 changes: 8 additions & 0 deletions skills/agent-browser/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -813,6 +813,14 @@ export AI_GATEWAY_URL=https://ai-gateway.vercel.sh # optional default

The Chat tab is always visible in the dashboard. Set `AI_GATEWAY_API_KEY` to enable AI responses.

#### Exa Web Search

Set `EXA_API_KEY` to give the AI chat a built-in web search tool powered by [Exa](https://exa.ai). When available, the AI uses Exa to look things up instead of navigating to a search engine. Results include titles, URLs, and text snippets. The AI can then open any result URL with agent-browser for deeper exploration.

```bash
export EXA_API_KEY=your_exa_api_key_here
```

## Ready-to-Use Templates

| Template | Description |
Expand Down