Skip to content

Commit 04a327e

Browse files
committed
Update README.md
1 parent 5c69d40 commit 04a327e

2 files changed

Lines changed: 0 additions & 356 deletions

File tree

README.md

Lines changed: 0 additions & 356 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,6 @@ Free Alternative to ChatGPT Atlas.
2121
- **Sidebar Chat Interface**: Clean, modern React-based chat UI accessible from any tab
2222
- **Direct Browser Automation**: No backend required - all API calls made directly from extension
2323
- **Visual Feedback**: Blue click indicators and element highlighting during automation
24-
- **Smart Coordinate Scaling**: Automatically scales Gemini's 1000x1000 coordinate system to actual viewport dimensions
2524
- **Safety Features**: Confirmation dialogs for sensitive actions (checkout, payment, etc.)
2625

2726
## Getting Started
@@ -104,358 +103,3 @@ Then reload the extension in Chrome after each change.
104103
- **[FAQ](./FAQ.md)** - Frequently asked questions and quick troubleshooting
105104
- **[TROUBLESHOOTING.md](./TROUBLESHOOTING.md)** - Detailed troubleshooting guide for common issues
106105

107-
## How It Works - Deep Dive
108-
109-
### Architecture Overview
110-
111-
Atlas consists of three main components that work together:
112-
113-
```
114-
┌─────────────────┐
115-
│ sidepanel.tsx │ ← React UI with chat interface
116-
│ (React UI) │
117-
└────────┬────────┘
118-
119-
↓ Messages
120-
┌─────────────────┐
121-
│ background.ts │ ← Service worker (screenshots, navigation)
122-
│ (Service │
123-
│ Worker) │
124-
└────────┬────────┘
125-
126-
↓ Execute actions
127-
┌─────────────────┐
128-
│ content.ts │ ← Content script (DOM manipulation)
129-
│ (Injected │
130-
│ on all tabs) │
131-
└─────────────────┘
132-
```
133-
134-
### Component Details
135-
136-
#### 1. Sidepanel (sidepanel.tsx) - The Brain
137-
138-
The main React component handles:
139-
140-
**Browser Tools Mode (`streamWithGeminiComputerUse`):**
141-
- Takes initial screenshot of current page
142-
- Sends screenshot + conversation history to Gemini 2.5 Computer Use
143-
- Receives function calls (click, type, navigate, scroll, etc.)
144-
- Executes actions via `executeBrowserAction()`
145-
- Re-takes screenshot after each action
146-
- Scales coordinates from Gemini's 1000x1000 grid to actual viewport
147-
- Supports up to 30 turns of action-response loops
148-
- Shows visual feedback in UI
149-
150-
**Tool Router Mode (`streamWithAISDKAndMCP`):**
151-
- Connects to Composio's MCP (Model Context Protocol) server
152-
- Uses Vercel AI SDK for streaming responses
153-
- Auto-discovers and calls tools via MCP
154-
- Manages MCP client lifecycle and session persistence
155-
156-
**Key Functions:**
157-
- `executeTool()` - Sends message to background script to perform browser actions
158-
- `scaleCoordinates()` - Converts Gemini coordinates to viewport coordinates
159-
- `executeBrowserAction()` - Maps Gemini function names to actual browser actions
160-
- `requiresUserConfirmation()` - Safety checks for sensitive actions
161-
- `loadSettings()` - Manages Composio session initialization
162-
163-
#### 2. Background Script (background.ts) - The Bridge
164-
165-
The service worker provides:
166-
167-
**Screenshot Functionality:**
168-
- Captures visible tab using `chrome.tabs.captureVisibleTab()`
169-
- Auto-handles restricted pages (chrome://, about:, etc.) by navigating to Google.com
170-
- Filters for actual visible tabs (not devtools or hidden windows)
171-
- Returns data URL for screenshot
172-
173-
**Action Execution:**
174-
- Relays commands from sidepanel to content script
175-
- Ensures content script is injected before execution
176-
- Handles tab targeting and message passing
177-
178-
**Browser APIs:**
179-
- Gets browser history
180-
- Manages bookmarks
181-
- Navigates tabs
182-
- Tracks recent pages for memory
183-
184-
**Key Message Types:**
185-
- `TAKE_SCREENSHOT` - Capture current tab
186-
- `GET_PAGE_CONTEXT` - Extract page metadata
187-
- `EXECUTE_ACTION` - Run browser action (click, type, etc.)
188-
- `NAVIGATE` - Change page URL
189-
- `GET_HISTORY` - Fetch browsing history
190-
191-
#### 3. Content Script (content.ts) - The Hands
192-
193-
Injected into every webpage to:
194-
195-
**Extract Page Context:**
196-
- URL, title, text content
197-
- Links, images, forms
198-
- Viewport dimensions (width, height, scroll position)
199-
- Metadata (description, keywords, author)
200-
201-
**Execute DOM Actions:**
202-
203-
**Click (`highlightElement()` + `executePageAction('click')`):**
204-
- Can click by CSS selector OR by coordinates
205-
- Dispatches full mouse event sequence (mousedown, mouseup, click)
206-
- Shows blue pulsing animation at click location
207-
- Highlights element with blue outline
208-
- Returns element info for debugging
209-
210-
**Type (`keyboard_type` action):**
211-
- Types character by character to simulate real keyboard input
212-
- Works with regular inputs, textareas, and contenteditable elements
213-
- Dispatches input/change events for React/Vue/Angular compatibility
214-
- For `type_text_at`: clicks coordinates, waits for focus, clears existing text, then types
215-
216-
**Scroll:**
217-
- Supports up/down by pixels
218-
- Can scroll to top/bottom
219-
- Can scroll element into view
220-
221-
**Navigate:**
222-
- Uses `chrome.tabs.update()` to change URLs
223-
224-
**Special Actions:**
225-
- `hover` - Mouse over at coordinates
226-
- `drag_drop` - Drag from one point to another
227-
- `key_combination` - Press keyboard shortcuts
228-
- `clear_input` - Clear focused field
229-
230-
**Visual Feedback:**
231-
- Blue outline on clicked elements
232-
- Blue pulsing circle at click coordinates
233-
- Animation automatically cleans up after 600ms
234-
235-
### Browser Tools Mode - Detailed Flow
236-
237-
When you enable Browser Tools (◉ button):
238-
239-
```
240-
1. User sends message: "Navigate to reddit.com and scroll"
241-
242-
2. sidepanel.tsx → streamWithGeminiComputerUse()
243-
244-
3. Take initial screenshot via executeTool('screenshot')
245-
246-
4. Send to Gemini 2.5 Computer Use with:
247-
- Screenshot as inline_data (base64 PNG)
248-
- Conversation history
249-
- System instruction with available functions
250-
251-
5. Gemini responds with function call: navigate({url: "https://reddit.com"})
252-
253-
6. executeBrowserAction() maps to executeTool('navigate')
254-
255-
7. background.ts receives EXECUTE_ACTION message
256-
257-
8. chrome.tabs.update() navigates to URL
258-
259-
9. Wait 2.5 seconds for page to load
260-
261-
10. Take new screenshot
262-
263-
11. Send function_response back to Gemini:
264-
{ url: "https://reddit.com", success: true, [screenshot] }
265-
266-
12. Gemini calls scroll_down()
267-
268-
13. Execute via content.ts → window.scrollBy()
269-
270-
14. Take another screenshot
271-
272-
15. Continue loop (up to 30 turns) until task complete
273-
```
274-
275-
**Coordinate Scaling Explained:**
276-
277-
Gemini Computer Use uses normalized coordinates (0-1000 on both axes). Atlas automatically scales them:
278-
279-
```javascript
280-
// Gemini returns: x=500, y=300 (in 1000x1000 space)
281-
// Actual viewport: 1920x1080
282-
283-
scaledX = (500 / 1000) * 1920 = 960
284-
scaledY = (300 / 1000) * 1080 = 324
285-
```
286-
287-
This ensures clicks land in the right place regardless of screen size.
288-
289-
### Safety & Confirmation System
290-
291-
The extension has built-in safety checks:
292-
293-
**Always Requires Confirmation:**
294-
- Keyboard combinations (Ctrl+A, Alt+Tab, etc.)
295-
296-
**Context-Aware Confirmation:**
297-
- **Sensitive Pages:** Checkout, payment, login, admin pages
298-
- **Sensitive Data:** Detecting passwords or credit cards being typed
299-
- **Form Submissions:** When typing with `press_enter: true`
300-
301-
Confirmation appears as a browser dialog before executing the action.
302-
303-
### Tool Router Mode - MCP Integration
304-
305-
When Composio API key is provided:
306-
307-
```
308-
1. initializeComposioToolRouter() creates session
309-
310-
2. Gets MCP URLs (chat_session_mcp_url, tool_router_mcp_url)
311-
312-
3. Connects to MCP via StreamableHTTPClientTransport
313-
314-
4. Queries available tools: mcpClient.tools()
315-
316-
5. Merges MCP tools with local tools (getBrowserHistory)
317-
318-
6. Passes all tools to AI SDK: streamText({ tools: allTools })
319-
320-
7. AI SDK orchestrates tool calls via MCP
321-
322-
8. Composio executes integration actions
323-
324-
9. Results streamed back to user
325-
```
326-
327-
**Available Tools in Tool Router Mode:**
328-
- **Composio Tools** - 500+ integrations (Gmail, Slack, GitHub, etc.)
329-
- **getBrowserHistory** - Built-in browser history search
330-
331-
The MCP client persists across messages but is recreated on "New Chat" to refresh available tools.
332-
333-
### Settings & Persistence
334-
335-
**Chrome Storage (`chrome.storage.local`):**
336-
- `atlasSettings` - API keys, model selection
337-
- `composioSessionId` - Active Composio session
338-
- `composioChatMcpUrl` - Chat MCP endpoint
339-
- `composioToolRouterMcpUrl` - Tool Router MCP endpoint
340-
- `extensionUserId` - Unique user ID for rate limiting
341-
- `browserMemory` - Recent pages, preferences
342-
343-
**Session Management:**
344-
- Composio sessions expire after 24 hours
345-
- New chat resets session to refresh tools
346-
- Session automatically recreated if expired
347-
348-
### Browser History Access
349-
350-
When using Tool Router mode (with Composio API key), the agent has access to a built-in `getBrowserHistory` tool that allows it to search through your browsing history.
351-
352-
**Tool Features:**
353-
- **Search by keyword** - Filter history by page title or URL
354-
- **Time range** - Default searches last 7 days, configurable
355-
- **Result limit** - Default returns 20 results, adjustable
356-
357-
**Example Usage:**
358-
- "What GitHub repositories did I visit this week?"
359-
- "Find the Reddit post I looked at yesterday"
360-
- "Show me recent news articles I read"
361-
- "What programming tutorials did I visit last month?"
362-
363-
The tool respects Chrome's history permissions and only accesses data you've already stored in your browser history.
364-
365-
## Project Structure
366-
367-
```
368-
atlas/
369-
├── sidepanel.tsx # Main React component (1426 lines)
370-
│ ├── Browser Tools mode with Gemini Computer Use
371-
│ ├── Tool Router mode with MCP
372-
│ ├── Message parsing with React Markdown
373-
│ └── Coordinate scaling and safety checks
374-
375-
├── content.ts # Content script (714 lines)
376-
│ ├── Page context extraction
377-
│ ├── DOM manipulation (click, type, scroll)
378-
│ ├── Visual feedback (blue click indicators)
379-
│ └── Keyboard simulation
380-
381-
├── background.ts # Service worker (302 lines)
382-
│ ├── Screenshot capture
383-
│ ├── Tab navigation
384-
│ ├── Browser history
385-
│ └── Message routing
386-
387-
├── settings.tsx # Settings page (163 lines)
388-
│ └── API key configuration UI
389-
390-
├── tools.ts # Composio integration (68 lines)
391-
│ └── MCP session management
392-
393-
├── types.ts # TypeScript definitions (271 lines)
394-
│ ├── Zod schemas for validation
395-
│ └── Interface definitions
396-
397-
├── manifest.json # Extension manifest (Manifest V3)
398-
│ ├── Permissions (tabs, history, bookmarks, etc.)
399-
│ └── Content script injection
400-
401-
└── vite.config.ts # Build configuration
402-
```
403-
404-
## Technologies Used
405-
406-
- **React 18** - UI framework with hooks
407-
- **TypeScript** - Type safety
408-
- **Vite** - Build tool and bundler
409-
- **Vercel AI SDK** - Streaming AI responses
410-
- **React Markdown** - Markdown rendering
411-
- **Zod** - Runtime validation
412-
- **Chrome Extension APIs** - Manifest V3
413-
- **Google Gemini API** - AI models (2.5 Pro/Flash/Lite/Computer Use)
414-
- **Composio MCP** - Tool Router integration
415-
- **StreamableHTTPClientTransport** - MCP transport
416-
417-
## Advanced Features
418-
419-
### Visual Feedback System
420-
421-
When clicking on a page:
422-
1. Blue outline appears around element (3px solid #007AFF)
423-
2. Light blue background highlight (rgba(0, 122, 255, 0.1))
424-
3. Pulsing circle animation at click coordinates
425-
4. All effects auto-remove after 600ms
426-
427-
### Keyboard Typing Simulation
428-
429-
The `keyboard_type` action:
430-
- Types character-by-character
431-
- Dispatches keydown, keypress, keyup for each char
432-
- Triggers input/change events for React compatibility
433-
- Works with INPUT, TEXTAREA, and contenteditable elements
434-
435-
### Error Handling & Retries
436-
437-
- Screenshots have retry logic (3 attempts with 1.5s delays)
438-
- Connection errors automatically retry
439-
- Graceful fallbacks for missing elements
440-
- Detailed error messages in UI
441-
442-
## Contributing
443-
444-
Contributions welcome! Please:
445-
1. Open an issue first to discuss changes
446-
2. Fork the repository
447-
3. Create a feature branch
448-
4. Submit a pull request
449-
450-
## References
451-
452-
- [Composio Platform](https://composio.dev/?utm_source=Github&utm_medium=Youtube&utm_campaign=2025-11&utm_content=Atlas) - Intelligent tool routing for AI agents
453-
- [Composio Tool Router Documentation](https://docs.composio.dev/docs/tool-router/quick-start) - Learn how to use Tool Router to route tool calls across 500+ integrations
454-
- [Composio GitHub](https://github.com/composiohq) - Python and TS SDK
455-
- [ChatGPT Atlas](https://openai.com/index/introducing-chatgpt-atlas/) - OpenAI's browser automation AI agent
456-
- [Gemini Computer Use Model](https://blog.google/technology/google-deepmind/gemini-computer-use-model/) - Google's AI model for browser automation
457-
- [Gemini API Documentation](https://ai.google.dev/gemini-api/docs/computer-use) - Official documentation for Gemini Computer Use
458-
459-
## License
460-
461-
MIT

atlas.gif

-50.6 MB
Loading

0 commit comments

Comments
 (0)