Performance improvements and simplification of code classes

.github/workflows/python-app.yml

@@ -27,7 +27,10 @@ jobs:
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip
-        pip install flake8 pytest
+        pip install --upgrade setuptools
+        pip install -e .
+        pip install -r test-requirements.txt
+        pip install -r dev-requirements.txt
         if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
     - name: Lint with flake8
       run: |
@@ -37,8 +40,4 @@ jobs:
         flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
     - name: Test with pytest
       run: |
-        pip install --upgrade pip
-        pip install --upgrade setuptools
-        pip install -e .
-        pip install -r test-requirements.txt
-        python -m pytest
+        python -m pytest -n auto

.gitignore

@@ -6,6 +6,8 @@ __pycache__/
 
 # C extensions
 *.so
+*.pyx
+*.c
 
 # Distribution / packaging
 .Python

MANIFEST.in

@@ -0,0 +1,3 @@
+recursive-include caqui *.pyx
+recursive-include caqui *.pxd
+recursive-include caqui *.py

Makefile

@@ -0,0 +1,30 @@
+build:
+	rm -rf build/ dist/
+#	python utils/build-pyx-files.py
+	python setup.py build_ext --inplace
+	python setup.py bdist_wheel
+
+setenv:
+	python3.7 -m venv venv
+	. venv/bin/activate
+	pip install --upgrade pip setuptools wheel
+	pip install -r test-requirements.txt
+	pip install -r dev-requirements.txt
+
+test:
+	pytest -n auto
+
+linter:
+	black -l 100 .
+	isort --profile black --line-length 100 caqui tests
+	flake8 --exclude venv*,.tox,build,*/test_process_data.py --max-line-length 100
+	mypy caqui tests --config=pyproject.toml    
+
+coverage:
+	coverage run --source='caqui' -m pytest -n auto
+	coverage report
+	coverage html
+
+clear:
+	rm -rf build/ dist/ *.egg-info
+	python utils/cleanup-cython-files.py

README.md

@@ -1,131 +1,158 @@
 # Caqui
 
+[![Python application](https://github.com/douglasdcm/caqui/actions/workflows/python-app.yml/badge.svg)](https://github.com/douglasdcm/caqui/actions/workflows/python-app.yml)
 [![PyPI Downloads](https://static.pepy.tech/badge/caqui)](https://pepy.tech/projects/caqui)
 
-**Caqui** executes commands against Drivers synchronously and asynchronously. The intention is that the user does not worry about which Driver they're using. It can be **Web**Drivers like [Selenium](https://www.selenium.dev/), **Mobile**Drivers like [Appium](http://appium.io/docs/en/2.0/), or **Desktop**Drivers like [Winium](https://github.com/2gis/Winium.Desktop). It can also be used in remote calls. The user can start the Driver as a server in any host and provide the URL to **Caqui** clients.
+**Caqui** is a Python library for browser, mobile, and desktop automation that works with any driver that exposes a WebDriver-style REST API. It lets you send commands **synchronously or asynchronously**, and you don’t need to think about which underlying driver you’re using.
 
-# Tested WebDrivers
+Caqui is designed for developers who want a unified automation API that can run:
 
-| WebDriver               | Version       | Remote* | Comment |
-| ----------------------- | ------------- | ------- |-------- |
-| Appium                  | 2.0.0+        | Y       | Accepts remote calls by default. Tested with Appium in Docker container |
-| Firefox (geckodriver)   | 113+          | Y       | Need to add the host ip, e.g. "--host 123.45.6.78" |
-| Google Chrome           | 113+          | Y       | Need to inform the allowed ips to connect, e.g "--allowed-ips=123.45.6.78" |
-| Opera                   | 99+           | Y       | Need to inform the allowed ips to connect, e.g "--allowed-ips=123.45.6.78". Similar to Google Chrome |
-| WinAppDriver            | 1.2.1+        | Y       | Need to define the host ip, e.g. "WinApppage.exe 10.0.0.10 4723" |
-| Winium Desktop          | 1.6.0+        | Y       | Accepts remote calls by default |
+* WebDriver (Chrome, Firefox, Opera, Edge)
+* Appium (Android, iOS)
+* Winium / WinAppDriver (Windows desktop applications)
+* Any remote WebDriver-compatible server
 
-* Accepts remote requests when running as servers
+Caqui runs seamlessly on a local machine or across remote hosts, and supports both **multitasking with asyncio** and **multiprocessing** for high-throughput use cases such as parallel testing, web scraping, or distributed automation.
 
-# Simple start
-Install the lastest version of **Caqui**
+---
+
+# Supported Drivers
+
+| WebDriver             | Version | Remote* | If remote                                                       |
+| --------------------- | ------- | ------- | ------------------------------------------------------------- |
+| Appium                | 2.0.0+  | Y       | Accepts remote calls by default. Tested with Appium in Docker |
+| Firefox (geckodriver) | 113+    | Y       | Requires defining the host IP, e.g. `--host 123.45.6.78`      |
+| Google Chrome         | 113+    | Y       | Requires allowed IPs, e.g. `--allowed-ips=123.45.6.78`        |
+| Opera                 | 99+     | Y       | Same restrictions as Chrome                                   |
+| WinAppDriver          | 1.2.1+  | Y       | Requires host IP, e.g. `WinApppage.exe 10.0.0.10 4723`        |
+| Winium Desktop        | 1.6.0+  | Y       | Accepts remote calls by default                               |
+
+*Remote = can accept REST requests when running as a server.
+
+---
+
+# Installation
 
 ```bash
 pip install caqui
 ```
 
-# Version 2.0.0+
-In version 2.0.0+ it is possible to use Python objects similarly to Selenium. **Read the [API documentation](https://caqui.readthedocs.io/en/latest/caqui.html) for more information.**
+---
+
+# Using Caqui 2.0.0+
+
+From version **2.0.0+**, Caqui includes a high-level API that mirrors Selenium’s object model and exposes async methods for browser, mobile, and desktop automation.
+[Full documentation:](https://caqui.readthedocs.io/en/latest/caqui.html)
 
 Example:
 
 ```python
+from os import getcwd
 from pytest import mark, fixture
-from tests.constants import PAGE_URL
-from caqui.easy import AsyncPage
+from caqui.easy.drivers import AsyncDriver
+from caqui.easy.capabilities import ChromeCapabilitiesBuilder
 from caqui.by import By
-from caqui import synchronous
-from caqui.easy.capabilities import ChromeOptionsBuilder
-from caqui.easy.options import ChromeOptionsBuilder
-from caqui.easy.server import Server
-from time import sleep
+from caqui.easy.server import LocalServer
 
+BASE_DIR = getcwd()
+PAGE_URL = f"file:///{BASE_DIR}/html/playground.html"
 SERVER_PORT = 9999
 SERVER_URL = f"http://localhost:{SERVER_PORT}"
-PAGE_URL = "file:///sample.html"
+
 
 @fixture(autouse=True, scope="session")
 def setup_server():
-    server = Server.get_instance(port=SERVER_PORT)
-    server.start()
+    server = LocalServer(port=SERVER_PORT)
+    server.start_chrome()
     yield
-    sleep(3)
-    server.dispose()
+    server.dispose(delay=3)
+
 
 @fixture
-def setup_environment():
+def caqui_driver():
     server_url = SERVER_URL
-    options = ChromeOptionsBuilder().args(["headless"]).to_dict()
-    capabilities = ChromeCapabilitiesBuilder().accept_insecure_certs(True).add_options(options).to_dict()
-    page = AsyncPage(server_url, capabilities, PAGE_URL)
+    capabilities = (
+        ChromeCapabilitiesBuilder().accept_insecure_certs(True).args(["headless"])
+    )
+    page = AsyncDriver(server_url, capabilities)
     yield page
     page.quit()
 
-@mark.asyncio
-async def test_switch_to_parent_frame_and_click_alert(setup_environment: AsyncPage):
-    page = setup_environment
-    await page.get(PAGE_URL)
-
-    locator_type = "id"
-    locator_value = "my-iframe"
-    locator_value_alert_parent = "alert-button"
-    locator_value_alert_frame = "alert-button-iframe"
 
-    element_frame = await page.find_element(locator_type, locator_value)
-    assert await page.switch_to.frame(element_frame) is True
+@mark.asyncio
+async def test_switch_to_parent_frame_and_click_alert(caqui_driver: AsyncDriver):
+    await caqui_driver.get(PAGE_URL)
+    element_frame = await caqui_driver.find_element(By.ID, "my-iframe")
+    assert await caqui_driver.switch_to.frame(element_frame)
 
-    alert_button_frame = await page.find_element(locator_type, locator_value_alert_frame)
-    assert await alert_button_frame.click() is True
-    assert await page.switch_to.alert.dismiss() is True
+    alert_button_frame = await caqui_driver.find_element(By.ID, "alert-button-iframe")
+    await alert_button_frame.click()
+    await caqui_driver.switch_to.alert.dismiss()
 
-    assert await page.switch_to.default_content() is True
-    alert_button_parent = await page.find_element(locator_type, locator_value_alert_parent)
+    await caqui_driver.switch_to.default_content()
+    alert_button_parent = await caqui_driver.find_element(By.ID, "alert-button")
     assert await alert_button_parent.get_attribute("any") == "any"
-    assert await alert_button_parent.click() is True
+    await alert_button_parent.click()
 
 ```
 
-## Running as multitasking
+---
 
-To execute the test in multiple tasks, use [pytest-async-cooperative](https://github.com/willemt/pytest-asyncio-cooperative). It will speed up the execution considerably.
+# Running Tests with Multitasking
+
+Caqui supports asyncio out of the box.
+To run multiple async tests concurrently, use **pytest-async-cooperative**:
 
 ```python
 @mark.asyncio_cooperative
-async def test_save_screenshot(setup_environment: AsyncPage):
-    page = setup_environment
-    assert await page.save_screenshot("/tmp/test.png") is True
+async def test_save_screenshot(caqui_driver: AsyncDriver):
+    await caqui_driver.get(PAGE_URL)
+    assert await caqui_driver.save_screenshot("/tmp/test.png")
 
-@mark.asyncio_cooperative
-async def test_object_to_string(setup_environment: AsyncPage):
-    page = setup_environment
-    element_string = synchronous.find_element(page.remote, page.session, By.XPATH, "//button")
-    element = await page.find_element(locator=By.XPATH, value="//button")
-    assert str(element) == element_string
 
+@mark.asyncio_cooperative
+async def test_click(caqui_driver: AsyncDriver):
+    await caqui_driver.get(PAGE_URL)
+    element = await caqui_driver.find_element(By.XPATH, "//button")
+    await element.click()
 ```
 
-## Running as multiprocessing
-To run the tests in multiple processes use [pytest-xdist](https://github.com/pytest-dev/pytest-xdist). The execution is even faster than running in multiple tasks. Check this article [Supercharge Your Web Crawlers with Caqui: Boosting Speed with Multi-Processing](https://medium.com/@douglas.dcm/speed-up-your-web-crawlers-at-90-148f3ca97b6) to know how to increase the velocity of the executions in 90%.
+Running tests this way significantly reduces execution time, especially when interacting with multiple drivers or sessions.
+
+---
+
+# Running Tests with Multiprocessing
+
+If your workloads benefit from multiple processes, Caqui also works with **pytest-xdist**.
+This approach is often faster than cooperative multitasking.
+
+A guide to optimizing performance (including a real benchmark):
+[Speed up your web crawlers at 90%](https://medium.com/@douglas.dcm/speed-up-your-web-crawlers-at-90-148f3ca97b6)
+
+Example:
 
 ```python
 @mark.asyncio
-async def test_save_screenshot(setup_environment: AsyncPage):
-    page = setup_environment
-    assert await page.save_screenshot("/tmp/test.png") is True
+async def test_save_screenshot(caqui_driver: AsyncDriver):
+    await caqui_driver.get(PAGE_URL)
+    assert await caqui_driver.save_screenshot("/tmp/test.png")
+
 
 @mark.asyncio
-async def test_object_to_string(setup_environment: AsyncPage):
-    page = setup_environment
-    element_string = synchronous.find_element(page.remote, page.session, By.XPATH, "//button")
-    element = await page.find_element(locator=By.XPATH, value="//button")
-    assert str(element) == element_string
+async def test_click(caqui_driver: AsyncDriver):
+    await caqui_driver.get(PAGE_URL)
+    element = await caqui_driver.find_element(By.XPATH, "//button")
+    await element.click()
 
 ```
 
-# Driver as a server
-In case you are using Appium, Winium or other driver not started by the library, just start the driver as a server.
+---
 
-For example. Download the same [ChromeDriver](https://chromepage.chromium.org/downloads) version as your installed Chrome and start the Driver as a server using the port "9999"
+# Running a Driver as a Server
+
+If you use external drivers such as Appium, Winium, or a standalone ChromeDriver, run them as servers and point Caqui to their URL.
+
+Example for ChromeDriver on port 9999:
 
 ```bash
 $ ./chromedriver --port=9999
@@ -134,11 +161,24 @@ Only local connections are allowed.
 Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
 ChromeDriver was started successfully.
 ```
-# Webdriver Manager
 
-Caqui depends on [Webdriver Manager](https://pypi.org/project/webdriver-manager/) that can be configured independenly and has some limitations. Check the project documentation for more information.
+---
+
+# WebDriver Manager
+
+Caqui’s `LocalServer` class uses [Webdriver Manager](https://pypi.org/project/webdriver-manager/).
+The tool comes with its own constraints.
+Check its documentation for details if you need custom driver handling.
 
+---
 
 # Contributing
-Read the [Code of Conduct](https://github.com/douglasdcm/caqui/blob/main/docs/CODE_OF_CONDUCT.md) before push new Merge Requests.
-Now, follow the steps in [Contributing](https://github.com/douglasdcm/caqui/blob/main/docs/CONTRIBUTING.md) session.
+
+Before submitting a pull request, review the project guidelines:
+Code of Conduct:
+[CODE OF CONDUCT](https://github.com/douglasdcm/caqui/blob/main/docs/CODE_OF_CONDUCT.md)
+
+Contribution Guide:
+[CONTRIBUTING](https://github.com/douglasdcm/caqui/blob/main/docs/CONTRIBUTING.md)
+
+Contributions, issue reports, and performance feedback are welcome.

caqui/__init__.py

@@ -0,0 +1,4 @@
+# Copyright (C) 2023 Caqui - All Rights Reserved
+# You may use, distribute and modify this code under the
+# terms of the MIT license.
+# Visit: https://github.com/douglasdcm/caqui