Feat: Added browser centric approach on the puppeteer mode (#9)

* Add browser use functionality with routing support

Introduced a new browser use feature, complete with routing and navigation using React Router. Updated the Header component with navigation links, added a new Browser Use component, and integrated its route in the application. Additionally, backend support for browser-related routes was added.

* Refactor and enhance browser automation services

Replaced Puppeteer-centric implementation with Playwright-compatible logic for better flexibility and scalability. Introduced new interfaces to standardize action types and element attributes, improving maintainability and readability. Removed unused code and redundant dependencies, streamlining overall functionality.

* Add BrowserUse component, backend routes, and .gitignore updates

Introduced a new BrowserUse component along with backend routes to handle browser-specific actions and messages. Updated the .gitignore file structure to distinguish between frontend and backend ignored files, improving project organization.

* Refactor system prompt to support marked screenshot analysis.

Introduces conditional logic for generating system prompts based on the availability of marked screenshots or omni parser results. Updated guidelines, action formats, and element interaction methods to leverage markers and element lists for improved task execution without hallucinations. Enhanced scroll and input handling rules for clearer and more precise user flows.

* Remove browserUseRoutes and related endpoints

The browserUseRoutes file and its associated endpoints are no longer needed. This cleanup helps streamline the codebase and removes unused functionality.

* Add new dependencies and update PuppeteerActions initialization

This commit introduces new dependencies in `package-lock.json`, including libraries like `canvas` and `playwright`. Additionally, it modifies the `PuppeteerActions` class to include a new `puppeteerService` parameter in the `initialize` method and updates the type of `ActionResponse`.

* Standardize quotes in server.ts to double quotes

Replaced all single quotes with double quotes for consistency in import statements, route definitions, and error messages. This improves code uniformity and adheres to a single style convention across the file.

* Refactor and optimize browser use and screenshot handling.

Removed unused frontend components for Browser Use and simplified header navigation. Refactored backend code to improve screenshot processing, support structured image data, and enforce consistent style. Enhanced Docker and Puppeteer service logic for robust container and screenshot management.

* Refactor action formatting and enhance image handling logic

Updated the action tag format from `<action>` to `<perform_action>` for better clarity and consistency, including improvements in attribute naming and structure. Enhanced the OpenAIProvider logic to conditionally append image data only when available, ensuring cleaner, error-free handling of messages with or without images.

* Set Puppeteer to headless mode and improve screenshot handling.

The browser is now launched in headless mode for better performance and resource usage. The `takeScreenshot` method has been updated to include error handling and ensure proper context and page extraction for capturing screenshots, reducing potential runtime issues.

* Handle empty imageData gracefully in LLM providers.

Updated GeminiProvider and AnthropicProvider to conditionally include image data only if it is non-empty. This prevents unnecessary processing and ensures stability when no image data is provided.

* Update URL requirements for 'launch' actions

Ensure URL is mandatory for 'launch' actions in prompts and error handling. Also, remove redundant `screenshot` field in error response for cleaner output.

* Initialize Puppeteer browser with success message

Added a return object with success status and a message when the Puppeteer browser initializes. This provides clear feedback on initialization status and improves code clarity.

* Remove unused dependencies from package-lock.json

This commit removes multiple unused dependencies, including Puppeteer-related packages and other redundant libraries, to optimize the project. This cleanup reduces the lock file size and ensures fewer unnecessary dependencies are included in the backend.

* Add new properties to ActionRequest interface

Introduced `elements` and `marker` properties to the `ActionRequest` interface to support enhanced interaction capabilities. This change integrates clickable elements and allows for additional data annotations within action requests.

* Improve action prompt clarity and ensure HTML stability checks

Refined system prompt formatting for better readability and action clarity by repositioning the <marker_number> placeholder. Added a waitTillHTMLStable method call to stabilize the page's DOM in PuppeteerService before proceeding with actions. This enhances reliability and reduces potential errors in page interactions.

* Refactor action handling and improve coord parsing.

Updated action types, replaced `IPlaywrightAction` with `ActionRequest` across services, and centralized coordinate parsing in `getCoordinate()` function. Minor improvements include better code clarity, error handling, and standardized logging.

* Refactor action handling to use ActionRequest type

Updated performAction methods in DockerActions and DockerVNCService to use the ActionRequest type instead of a plain string. This ensures type safety and better structure when handling actions. Adjusted related logic to accommodate the new type.

* Improve streaming controller and pass coordinates for actions

Updated the streaming controller to refine actions with structured `ActionRequest` and ensure better error handling. Enhanced UI interaction service to include last click coordinates when emitting typing actions.

* Remove .idea/.gitignore from the repository

The .idea/.gitignore file has been deleted as it is unnecessary. This change ensures a cleaner repository and removes IDE-specific clutter. Developers can manage ignored files locally if needed.

* Remove package.json and lock file from repository

The package.json and package-lock.json files were deleted, likely as part of a cleanup or migration effort. This ensures these files are no longer tracked in the repository, possibly due to changes in how dependencies are managed.

* Add package.json with project scripts

Introduce a `package.json` file to manage project metadata and scripts. Includes commands for starting frontend/backend, installing dependencies, and building both. This enhances project organization and simplifies workflow management.

* Remove unnecessary debug logging in PuppeteerService

Eliminated a console log that printed an element during evaluation in PuppeteerService. This cleanup reduces noise in the logs and improves code readability.

* Simplify request parameter handling in controllers

Removed unused `folderPath` and `chatId` query parameters from actionController and `imageData` parameter from chatController. This improves code clarity and reduces unnecessary data processing.

* Refactor logging and improve error handling in services

Remove redundant console logs in PuppeteerActions to reduce noise and improve debugging clarity. Add an error log in AnthropicProvider for better error tracking and enhance validation for required fields in action input.

* Refactor keypress action validation and logic.

Replace text validation with key validation to align with action parameters. Simplify coordinate handling and remove unnecessary context evaluation for focused elements. Adjust key handling logic for improved clarity and consistency.

* Add Changesets configuration and setup

Integrated Changesets for versioning and changelog management. Included configuration, commands for versioning and publishing in `package.json`, and initial changelog documentation setup. This enables streamlined release processes and better project maintainability.

* Update puppeteer mode and apply general improvements

    Improved the browser-centric approach in puppeteer mode to enhance functionality. Included various general improvements and resolved minor bugs for better overall stability.

Author: sanju-presidio

.changeset/README.md

@@ -0,0 +1,8 @@
+# Changesets
+
+Hello and welcome! This folder has been automatically generated by `@changesets/cli`, a build tool that works
+with multi-package repos, or single-package repos to help you version and publish your code. You can
+find the full documentation for it [in our repository](https://github.com/changesets/changesets)
+
+We have a quick list of common questions to get you started engaging with this project in
+[our documentation](https://github.com/changesets/changesets/blob/main/docs/common-questions.md)

.changeset/config.json

@@ -0,0 +1,11 @@
+{
+  "$schema": "https://unpkg.com/@changesets/config@3.1.1/schema.json",
+  "changelog": "@changesets/cli/changelog",
+  "commit": false,
+  "fixed": [],
+  "linked": [],
+  "access": "restricted",
+  "baseBranch": "main",
+  "updateInternalDependencies": "patch",
+  "ignore": []
+}

.changeset/giant-carrots-wink.md

@@ -0,0 +1,6 @@
+---
+"Factifai": minor
+---
+
+Added browser centric approach on the puppeteer mode.
+General improvements & bugfixes

.gitignore

@@ -0,0 +1,75 @@
+# Dependencies
+frontend/node_modules/
+frontend/npm-debug.log*
+frontend/yarn-debug.log*
+frontend/yarn-error.log*
+
+# Environment variables
+frontend/.env
+frontend/.env.local
+frontend/.env.development.local
+frontend/.env.test.local
+frontend/.env.production.local
+
+# Build output
+frontend/dist/
+frontend/dist-ssr/
+frontend/build/
+
+# TypeScript
+frontend/*.tsbuildinfo
+
+# Vite
+frontend/.vite/
+
+# IDE
+.idea/
+.vscode/
+*.swp
+*.swo
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Coverage
+frontend/coverage/
+
+# Logs
+frontend/logs/
+*.log
+
+
+
+# Dependencies
+backend/node_modules/
+backend/npm-debug.log*
+backend/yarn-debug.log*
+backend/yarn-error.log*
+
+# Environment variables
+backend/.env
+backend/.env.local
+backend/.env.development.local
+backend/.env.test.local
+backend/.env.production.local
+
+# Build output
+backend/dist/
+backend/build/
+
+# TypeScript
+*.tsbuildinfo
+
+# IDE
+.idea/
+.vscode/
+*.swp
+*.swo
+
+# OS
+.DS_Store
+Thumbs.db
+backend/logs/
+logs
+backend/output

CHANGELOG.md

@@ -0,0 +1,16 @@
+# Changelog
+
+## [1.0.0]
+
+### Built-in support for leading vision-language models:
+- **Claude**: Anthropic's advanced vision and reasoning model
+- **OpenAI**: GPT-4o with visual understanding capabilities
+- **Gemini**: Google's multimodal AI for computer interaction
+- **[OmniParser](https://github.com/microsoft/OmniParser)**: Screen Parsing tool for Pure Vision Based GUI Agent
+
+
+### AI-Powered Computer Control
+- Intelligent element detection and navigation
+- Automated verification and validation
+- Comprehensive test documentation with automated screenshot capture for each step
+- Integrated test case export with visual step-by-step documentation