@@ -21,7 +21,6 @@ Free Alternative to ChatGPT Atlas.
2121- ** Sidebar Chat Interface** : Clean, modern React-based chat UI accessible from any tab
2222- ** Direct Browser Automation** : No backend required - all API calls made directly from extension
2323- ** Visual Feedback** : Blue click indicators and element highlighting during automation
24- - ** Smart Coordinate Scaling** : Automatically scales Gemini's 1000x1000 coordinate system to actual viewport dimensions
2524- ** Safety Features** : Confirmation dialogs for sensitive actions (checkout, payment, etc.)
2625
2726## Getting Started
@@ -104,358 +103,3 @@ Then reload the extension in Chrome after each change.
104103- ** [ FAQ] ( ./FAQ.md ) ** - Frequently asked questions and quick troubleshooting
105104- ** [ TROUBLESHOOTING.md] ( ./TROUBLESHOOTING.md ) ** - Detailed troubleshooting guide for common issues
106105
107- ## How It Works - Deep Dive
108-
109- ### Architecture Overview
110-
111- Atlas consists of three main components that work together:
112-
113- ```
114- ┌─────────────────┐
115- │ sidepanel.tsx │ ← React UI with chat interface
116- │ (React UI) │
117- └────────┬────────┘
118- │
119- ↓ Messages
120- ┌─────────────────┐
121- │ background.ts │ ← Service worker (screenshots, navigation)
122- │ (Service │
123- │ Worker) │
124- └────────┬────────┘
125- │
126- ↓ Execute actions
127- ┌─────────────────┐
128- │ content.ts │ ← Content script (DOM manipulation)
129- │ (Injected │
130- │ on all tabs) │
131- └─────────────────┘
132- ```
133-
134- ### Component Details
135-
136- #### 1. Sidepanel (sidepanel.tsx) - The Brain
137-
138- The main React component handles:
139-
140- ** Browser Tools Mode (` streamWithGeminiComputerUse ` ):**
141- - Takes initial screenshot of current page
142- - Sends screenshot + conversation history to Gemini 2.5 Computer Use
143- - Receives function calls (click, type, navigate, scroll, etc.)
144- - Executes actions via ` executeBrowserAction() `
145- - Re-takes screenshot after each action
146- - Scales coordinates from Gemini's 1000x1000 grid to actual viewport
147- - Supports up to 30 turns of action-response loops
148- - Shows visual feedback in UI
149-
150- ** Tool Router Mode (` streamWithAISDKAndMCP ` ):**
151- - Connects to Composio's MCP (Model Context Protocol) server
152- - Uses Vercel AI SDK for streaming responses
153- - Auto-discovers and calls tools via MCP
154- - Manages MCP client lifecycle and session persistence
155-
156- ** Key Functions:**
157- - ` executeTool() ` - Sends message to background script to perform browser actions
158- - ` scaleCoordinates() ` - Converts Gemini coordinates to viewport coordinates
159- - ` executeBrowserAction() ` - Maps Gemini function names to actual browser actions
160- - ` requiresUserConfirmation() ` - Safety checks for sensitive actions
161- - ` loadSettings() ` - Manages Composio session initialization
162-
163- #### 2. Background Script (background.ts) - The Bridge
164-
165- The service worker provides:
166-
167- ** Screenshot Functionality:**
168- - Captures visible tab using ` chrome.tabs.captureVisibleTab() `
169- - Auto-handles restricted pages (chrome://, about:, etc.) by navigating to Google.com
170- - Filters for actual visible tabs (not devtools or hidden windows)
171- - Returns data URL for screenshot
172-
173- ** Action Execution:**
174- - Relays commands from sidepanel to content script
175- - Ensures content script is injected before execution
176- - Handles tab targeting and message passing
177-
178- ** Browser APIs:**
179- - Gets browser history
180- - Manages bookmarks
181- - Navigates tabs
182- - Tracks recent pages for memory
183-
184- ** Key Message Types:**
185- - ` TAKE_SCREENSHOT ` - Capture current tab
186- - ` GET_PAGE_CONTEXT ` - Extract page metadata
187- - ` EXECUTE_ACTION ` - Run browser action (click, type, etc.)
188- - ` NAVIGATE ` - Change page URL
189- - ` GET_HISTORY ` - Fetch browsing history
190-
191- #### 3. Content Script (content.ts) - The Hands
192-
193- Injected into every webpage to:
194-
195- ** Extract Page Context:**
196- - URL, title, text content
197- - Links, images, forms
198- - Viewport dimensions (width, height, scroll position)
199- - Metadata (description, keywords, author)
200-
201- ** Execute DOM Actions:**
202-
203- ** Click (` highlightElement() ` + ` executePageAction('click') ` ):**
204- - Can click by CSS selector OR by coordinates
205- - Dispatches full mouse event sequence (mousedown, mouseup, click)
206- - Shows blue pulsing animation at click location
207- - Highlights element with blue outline
208- - Returns element info for debugging
209-
210- ** Type (` keyboard_type ` action):**
211- - Types character by character to simulate real keyboard input
212- - Works with regular inputs, textareas, and contenteditable elements
213- - Dispatches input/change events for React/Vue/Angular compatibility
214- - For ` type_text_at ` : clicks coordinates, waits for focus, clears existing text, then types
215-
216- ** Scroll:**
217- - Supports up/down by pixels
218- - Can scroll to top/bottom
219- - Can scroll element into view
220-
221- ** Navigate:**
222- - Uses ` chrome.tabs.update() ` to change URLs
223-
224- ** Special Actions:**
225- - ` hover ` - Mouse over at coordinates
226- - ` drag_drop ` - Drag from one point to another
227- - ` key_combination ` - Press keyboard shortcuts
228- - ` clear_input ` - Clear focused field
229-
230- ** Visual Feedback:**
231- - Blue outline on clicked elements
232- - Blue pulsing circle at click coordinates
233- - Animation automatically cleans up after 600ms
234-
235- ### Browser Tools Mode - Detailed Flow
236-
237- When you enable Browser Tools (◉ button):
238-
239- ```
240- 1. User sends message: "Navigate to reddit.com and scroll"
241- ↓
242- 2. sidepanel.tsx → streamWithGeminiComputerUse()
243- ↓
244- 3. Take initial screenshot via executeTool('screenshot')
245- ↓
246- 4. Send to Gemini 2.5 Computer Use with:
247- - Screenshot as inline_data (base64 PNG)
248- - Conversation history
249- - System instruction with available functions
250- ↓
251- 5. Gemini responds with function call: navigate({url: "https://reddit.com"})
252- ↓
253- 6. executeBrowserAction() maps to executeTool('navigate')
254- ↓
255- 7. background.ts receives EXECUTE_ACTION message
256- ↓
257- 8. chrome.tabs.update() navigates to URL
258- ↓
259- 9. Wait 2.5 seconds for page to load
260- ↓
261- 10. Take new screenshot
262- ↓
263- 11. Send function_response back to Gemini:
264- { url: "https://reddit.com", success: true, [screenshot] }
265- ↓
266- 12. Gemini calls scroll_down()
267- ↓
268- 13. Execute via content.ts → window.scrollBy()
269- ↓
270- 14. Take another screenshot
271- ↓
272- 15. Continue loop (up to 30 turns) until task complete
273- ```
274-
275- ** Coordinate Scaling Explained:**
276-
277- Gemini Computer Use uses normalized coordinates (0-1000 on both axes). Atlas automatically scales them:
278-
279- ``` javascript
280- // Gemini returns: x=500, y=300 (in 1000x1000 space)
281- // Actual viewport: 1920x1080
282-
283- scaledX = (500 / 1000 ) * 1920 = 960
284- scaledY = (300 / 1000 ) * 1080 = 324
285- ```
286-
287- This ensures clicks land in the right place regardless of screen size.
288-
289- ### Safety & Confirmation System
290-
291- The extension has built-in safety checks:
292-
293- ** Always Requires Confirmation:**
294- - Keyboard combinations (Ctrl+A, Alt+Tab, etc.)
295-
296- ** Context-Aware Confirmation:**
297- - ** Sensitive Pages:** Checkout, payment, login, admin pages
298- - ** Sensitive Data:** Detecting passwords or credit cards being typed
299- - ** Form Submissions:** When typing with ` press_enter: true `
300-
301- Confirmation appears as a browser dialog before executing the action.
302-
303- ### Tool Router Mode - MCP Integration
304-
305- When Composio API key is provided:
306-
307- ```
308- 1. initializeComposioToolRouter() creates session
309- ↓
310- 2. Gets MCP URLs (chat_session_mcp_url, tool_router_mcp_url)
311- ↓
312- 3. Connects to MCP via StreamableHTTPClientTransport
313- ↓
314- 4. Queries available tools: mcpClient.tools()
315- ↓
316- 5. Merges MCP tools with local tools (getBrowserHistory)
317- ↓
318- 6. Passes all tools to AI SDK: streamText({ tools: allTools })
319- ↓
320- 7. AI SDK orchestrates tool calls via MCP
321- ↓
322- 8. Composio executes integration actions
323- ↓
324- 9. Results streamed back to user
325- ```
326-
327- ** Available Tools in Tool Router Mode:**
328- - ** Composio Tools** - 500+ integrations (Gmail, Slack, GitHub, etc.)
329- - ** getBrowserHistory** - Built-in browser history search
330-
331- The MCP client persists across messages but is recreated on "New Chat" to refresh available tools.
332-
333- ### Settings & Persistence
334-
335- ** Chrome Storage (` chrome.storage.local ` ):**
336- - ` atlasSettings ` - API keys, model selection
337- - ` composioSessionId ` - Active Composio session
338- - ` composioChatMcpUrl ` - Chat MCP endpoint
339- - ` composioToolRouterMcpUrl ` - Tool Router MCP endpoint
340- - ` extensionUserId ` - Unique user ID for rate limiting
341- - ` browserMemory ` - Recent pages, preferences
342-
343- ** Session Management:**
344- - Composio sessions expire after 24 hours
345- - New chat resets session to refresh tools
346- - Session automatically recreated if expired
347-
348- ### Browser History Access
349-
350- When using Tool Router mode (with Composio API key), the agent has access to a built-in ` getBrowserHistory ` tool that allows it to search through your browsing history.
351-
352- ** Tool Features:**
353- - ** Search by keyword** - Filter history by page title or URL
354- - ** Time range** - Default searches last 7 days, configurable
355- - ** Result limit** - Default returns 20 results, adjustable
356-
357- ** Example Usage:**
358- - "What GitHub repositories did I visit this week?"
359- - "Find the Reddit post I looked at yesterday"
360- - "Show me recent news articles I read"
361- - "What programming tutorials did I visit last month?"
362-
363- The tool respects Chrome's history permissions and only accesses data you've already stored in your browser history.
364-
365- ## Project Structure
366-
367- ```
368- atlas/
369- ├── sidepanel.tsx # Main React component (1426 lines)
370- │ ├── Browser Tools mode with Gemini Computer Use
371- │ ├── Tool Router mode with MCP
372- │ ├── Message parsing with React Markdown
373- │ └── Coordinate scaling and safety checks
374- │
375- ├── content.ts # Content script (714 lines)
376- │ ├── Page context extraction
377- │ ├── DOM manipulation (click, type, scroll)
378- │ ├── Visual feedback (blue click indicators)
379- │ └── Keyboard simulation
380- │
381- ├── background.ts # Service worker (302 lines)
382- │ ├── Screenshot capture
383- │ ├── Tab navigation
384- │ ├── Browser history
385- │ └── Message routing
386- │
387- ├── settings.tsx # Settings page (163 lines)
388- │ └── API key configuration UI
389- │
390- ├── tools.ts # Composio integration (68 lines)
391- │ └── MCP session management
392- │
393- ├── types.ts # TypeScript definitions (271 lines)
394- │ ├── Zod schemas for validation
395- │ └── Interface definitions
396- │
397- ├── manifest.json # Extension manifest (Manifest V3)
398- │ ├── Permissions (tabs, history, bookmarks, etc.)
399- │ └── Content script injection
400- │
401- └── vite.config.ts # Build configuration
402- ```
403-
404- ## Technologies Used
405-
406- - ** React 18** - UI framework with hooks
407- - ** TypeScript** - Type safety
408- - ** Vite** - Build tool and bundler
409- - ** Vercel AI SDK** - Streaming AI responses
410- - ** React Markdown** - Markdown rendering
411- - ** Zod** - Runtime validation
412- - ** Chrome Extension APIs** - Manifest V3
413- - ** Google Gemini API** - AI models (2.5 Pro/Flash/Lite/Computer Use)
414- - ** Composio MCP** - Tool Router integration
415- - ** StreamableHTTPClientTransport** - MCP transport
416-
417- ## Advanced Features
418-
419- ### Visual Feedback System
420-
421- When clicking on a page:
422- 1 . Blue outline appears around element (3px solid #007AFF)
423- 2 . Light blue background highlight (rgba(0, 122, 255, 0.1))
424- 3 . Pulsing circle animation at click coordinates
425- 4 . All effects auto-remove after 600ms
426-
427- ### Keyboard Typing Simulation
428-
429- The ` keyboard_type ` action:
430- - Types character-by-character
431- - Dispatches keydown, keypress, keyup for each char
432- - Triggers input/change events for React compatibility
433- - Works with INPUT, TEXTAREA, and contenteditable elements
434-
435- ### Error Handling & Retries
436-
437- - Screenshots have retry logic (3 attempts with 1.5s delays)
438- - Connection errors automatically retry
439- - Graceful fallbacks for missing elements
440- - Detailed error messages in UI
441-
442- ## Contributing
443-
444- Contributions welcome! Please:
445- 1 . Open an issue first to discuss changes
446- 2 . Fork the repository
447- 3 . Create a feature branch
448- 4 . Submit a pull request
449-
450- ## References
451-
452- - [ Composio Platform] ( https://composio.dev/?utm_source=Github&utm_medium=Youtube&utm_campaign=2025-11&utm_content=Atlas ) - Intelligent tool routing for AI agents
453- - [ Composio Tool Router Documentation] ( https://docs.composio.dev/docs/tool-router/quick-start ) - Learn how to use Tool Router to route tool calls across 500+ integrations
454- - [ Composio GitHub] ( https://github.com/composiohq ) - Python and TS SDK
455- - [ ChatGPT Atlas] ( https://openai.com/index/introducing-chatgpt-atlas/ ) - OpenAI's browser automation AI agent
456- - [ Gemini Computer Use Model] ( https://blog.google/technology/google-deepmind/gemini-computer-use-model/ ) - Google's AI model for browser automation
457- - [ Gemini API Documentation] ( https://ai.google.dev/gemini-api/docs/computer-use ) - Official documentation for Gemini Computer Use
458-
459- ## License
460-
461- MIT
0 commit comments