Skip to content

Commit 6bed3f9

Browse files
douglasdcmdouglasdcm
authored
Merge pull request #114 from douglasdcm/performance-improvements
Performance improvements and simplification of code classes - Update documentation with new classes and methods - Add support to single connection by `aiohttp.ClientSession` - Improve type-hints - Add support to `W3C` and `Jsonwire` specifications - Convert all locators to CSS Selector in runtime to improve performance - Add pool of elements in `AsyncDriver` for better performance - Rename `AsyncPage` class to `AsyncDriver` - Merge `OptionsBuilder` class to `CapabilitiesBuilder` class - Rename `Server` class to `LocalServer` class - Rename internal variables for better code understanding - Add the parameter `executable_path` to `LocalServer` to use existing local drivers - Add specific methods in `LocalServer` to start web drivers - Fix skipped tests - Test all functions with Chrome, Opera, Edge and Firefox
2 parents 82ec0b8 + 820a4db commit 6bed3f9

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

64 files changed

+7029
-3004
lines changed

‎.github/workflows/python-app.yml‎

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,10 @@ jobs:
2727
- name: Install dependencies
2828
run: |
2929
python -m pip install --upgrade pip
30-
pip install flake8 pytest
30+
pip install --upgrade setuptools
31+
pip install -e .
32+
pip install -r test-requirements.txt
33+
pip install -r dev-requirements.txt
3134
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
3235
- name: Lint with flake8
3336
run: |
@@ -37,8 +40,4 @@ jobs:
3740
flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
3841
- name: Test with pytest
3942
run: |
40-
pip install --upgrade pip
41-
pip install --upgrade setuptools
42-
pip install -e .
43-
pip install -r test-requirements.txt
44-
python -m pytest
43+
python -m pytest -n auto

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,8 @@ __pycache__/
66

77
# C extensions
88
*.so
9+
*.pyx
10+
*.c
911

1012
# Distribution / packaging
1113
.Python

‎MANIFEST.in‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
recursive-include caqui *.pyx
2+
recursive-include caqui *.pxd
3+
recursive-include caqui *.py

‎Makefile‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
build:
2+
rm -rf build/ dist/
3+
# python utils/build-pyx-files.py
4+
python setup.py build_ext --inplace
5+
python setup.py bdist_wheel
6+
7+
setenv:
8+
python3.7 -m venv venv
9+
. venv/bin/activate
10+
pip install --upgrade pip setuptools wheel
11+
pip install -r test-requirements.txt
12+
pip install -r dev-requirements.txt
13+
14+
test:
15+
pytest -n auto
16+
17+
linter:
18+
black -l 100 .
19+
isort --profile black --line-length 100 caqui tests
20+
flake8 --exclude venv*,.tox,build,*/test_process_data.py --max-line-length 100
21+
mypy caqui tests --config=pyproject.toml
22+
23+
coverage:
24+
coverage run --source='caqui' -m pytest -n auto
25+
coverage report
26+
coverage html
27+
28+
clear:
29+
rm -rf build/ dist/ *.egg-info
30+
python utils/cleanup-cython-files.py

‎README.md‎

Lines changed: 116 additions & 76 deletions
Original file line numberDiff line numberDiff line change
@@ -1,131 +1,158 @@
11
# Caqui
22

3+
[![Python application](https://github.com/douglasdcm/caqui/actions/workflows/python-app.yml/badge.svg)](https://github.com/douglasdcm/caqui/actions/workflows/python-app.yml)
34
[![PyPI Downloads](https://static.pepy.tech/badge/caqui)](https://pepy.tech/projects/caqui)
45

5-
**Caqui** executes commands against Drivers synchronously and asynchronously. The intention is that the user does not worry about which Driver they're using. It can be **Web**Drivers like [Selenium](https://www.selenium.dev/), **Mobile**Drivers like [Appium](http://appium.io/docs/en/2.0/), or **Desktop**Drivers like [Winium](https://github.com/2gis/Winium.Desktop). It can also be used in remote calls. The user can start the Driver as a server in any host and provide the URL to **Caqui** clients.
6+
**Caqui** is a Python library for browser, mobile, and desktop automation that works with any driver that exposes a WebDriver-style REST API. It lets you send commands **synchronously or asynchronously**, and you don’t need to think about which underlying driver you’re using.
67

7-
# Tested WebDrivers
8+
Caqui is designed for developers who want a unified automation API that can run:
89

9-
| WebDriver | Version | Remote* | Comment |
10-
| ----------------------- | ------------- | ------- |-------- |
11-
| Appium | 2.0.0+ | Y | Accepts remote calls by default. Tested with Appium in Docker container |
12-
| Firefox (geckodriver) | 113+ | Y | Need to add the host ip, e.g. "--host 123.45.6.78" |
13-
| Google Chrome | 113+ | Y | Need to inform the allowed ips to connect, e.g "--allowed-ips=123.45.6.78" |
14-
| Opera | 99+ | Y | Need to inform the allowed ips to connect, e.g "--allowed-ips=123.45.6.78". Similar to Google Chrome |
15-
| WinAppDriver | 1.2.1+ | Y | Need to define the host ip, e.g. "WinApppage.exe 10.0.0.10 4723" |
16-
| Winium Desktop | 1.6.0+ | Y | Accepts remote calls by default |
10+
* WebDriver (Chrome, Firefox, Opera, Edge)
11+
* Appium (Android, iOS)
12+
* Winium / WinAppDriver (Windows desktop applications)
13+
* Any remote WebDriver-compatible server
1714

18-
* Accepts remote requests when running as servers
15+
Caqui runs seamlessly on a local machine or across remote hosts, and supports both **multitasking with asyncio** and **multiprocessing** for high-throughput use cases such as parallel testing, web scraping, or distributed automation.
1916

20-
# Simple start
21-
Install the lastest version of **Caqui**
17+
---
18+
19+
# Supported Drivers
20+
21+
| WebDriver | Version | Remote* | If remote |
22+
| --------------------- | ------- | ------- | ------------------------------------------------------------- |
23+
| Appium | 2.0.0+ | Y | Accepts remote calls by default. Tested with Appium in Docker |
24+
| Firefox (geckodriver) | 113+ | Y | Requires defining the host IP, e.g. `--host 123.45.6.78` |
25+
| Google Chrome | 113+ | Y | Requires allowed IPs, e.g. `--allowed-ips=123.45.6.78` |
26+
| Opera | 99+ | Y | Same restrictions as Chrome |
27+
| WinAppDriver | 1.2.1+ | Y | Requires host IP, e.g. `WinApppage.exe 10.0.0.10 4723` |
28+
| Winium Desktop | 1.6.0+ | Y | Accepts remote calls by default |
29+
30+
*Remote = can accept REST requests when running as a server.
31+
32+
---
33+
34+
# Installation
2235

2336
```bash
2437
pip install caqui
2538
```
2639

27-
# Version 2.0.0+
28-
In version 2.0.0+ it is possible to use Python objects similarly to Selenium. **Read the [API documentation](https://caqui.readthedocs.io/en/latest/caqui.html) for more information.**
40+
---
41+
42+
# Using Caqui 2.0.0+
43+
44+
From version **2.0.0+**, Caqui includes a high-level API that mirrors Selenium’s object model and exposes async methods for browser, mobile, and desktop automation.
45+
[Full documentation:](https://caqui.readthedocs.io/en/latest/caqui.html)
2946

3047
Example:
3148

3249
```python
50+
from os import getcwd
3351
from pytest import mark, fixture
34-
from tests.constants import PAGE_URL
35-
from caqui.easy import AsyncPage
52+
from caqui.easy.drivers import AsyncDriver
53+
from caqui.easy.capabilities import ChromeCapabilitiesBuilder
3654
from caqui.by import By
37-
from caqui import synchronous
38-
from caqui.easy.capabilities import ChromeOptionsBuilder
39-
from caqui.easy.options import ChromeOptionsBuilder
40-
from caqui.easy.server import Server
41-
from time import sleep
55+
from caqui.easy.server import LocalServer
4256

57+
BASE_DIR = getcwd()
58+
PAGE_URL = f"file:///{BASE_DIR}/html/playground.html"
4359
SERVER_PORT = 9999
4460
SERVER_URL = f"http://localhost:{SERVER_PORT}"
45-
PAGE_URL = "file:///sample.html"
61+
4662

4763
@fixture(autouse=True, scope="session")
4864
def setup_server():
49-
server = Server.get_instance(port=SERVER_PORT)
50-
server.start()
65+
server = LocalServer(port=SERVER_PORT)
66+
server.start_chrome()
5167
yield
52-
sleep(3)
53-
server.dispose()
68+
server.dispose(delay=3)
69+
5470

5571
@fixture
56-
def setup_environment():
72+
def caqui_driver():
5773
server_url = SERVER_URL
58-
options = ChromeOptionsBuilder().args(["headless"]).to_dict()
59-
capabilities = ChromeCapabilitiesBuilder().accept_insecure_certs(True).add_options(options).to_dict()
60-
page = AsyncPage(server_url, capabilities, PAGE_URL)
74+
capabilities = (
75+
ChromeCapabilitiesBuilder().accept_insecure_certs(True).args(["headless"])
76+
)
77+
page = AsyncDriver(server_url, capabilities)
6178
yield page
6279
page.quit()
6380

64-
@mark.asyncio
65-
async def test_switch_to_parent_frame_and_click_alert(setup_environment: AsyncPage):
66-
page = setup_environment
67-
await page.get(PAGE_URL)
68-
69-
locator_type = "id"
70-
locator_value = "my-iframe"
71-
locator_value_alert_parent = "alert-button"
72-
locator_value_alert_frame = "alert-button-iframe"
7381

74-
element_frame = await page.find_element(locator_type, locator_value)
75-
assert await page.switch_to.frame(element_frame) is True
82+
@mark.asyncio
83+
async def test_switch_to_parent_frame_and_click_alert(caqui_driver: AsyncDriver):
84+
await caqui_driver.get(PAGE_URL)
85+
element_frame = await caqui_driver.find_element(By.ID, "my-iframe")
86+
assert await caqui_driver.switch_to.frame(element_frame)
7687

77-
alert_button_frame = await page.find_element(locator_type, locator_value_alert_frame)
78-
assert await alert_button_frame.click() is True
79-
assert await page.switch_to.alert.dismiss() is True
88+
alert_button_frame = await caqui_driver.find_element(By.ID, "alert-button-iframe")
89+
await alert_button_frame.click()
90+
await caqui_driver.switch_to.alert.dismiss()
8091

81-
assert await page.switch_to.default_content() is True
82-
alert_button_parent = await page.find_element(locator_type, locator_value_alert_parent)
92+
await caqui_driver.switch_to.default_content()
93+
alert_button_parent = await caqui_driver.find_element(By.ID, "alert-button")
8394
assert await alert_button_parent.get_attribute("any") == "any"
84-
assert await alert_button_parent.click() is True
95+
await alert_button_parent.click()
8596

8697
```
8798

88-
## Running as multitasking
99+
---
89100

90-
To execute the test in multiple tasks, use [pytest-async-cooperative](https://github.com/willemt/pytest-asyncio-cooperative). It will speed up the execution considerably.
101+
# Running Tests with Multitasking
102+
103+
Caqui supports asyncio out of the box.
104+
To run multiple async tests concurrently, use **pytest-async-cooperative**:
91105

92106
```python
93107
@mark.asyncio_cooperative
94-
async def test_save_screenshot(setup_environment: AsyncPage):
95-
page = setup_environment
96-
assert await page.save_screenshot("/tmp/test.png") is True
108+
async def test_save_screenshot(caqui_driver: AsyncDriver):
109+
await caqui_driver.get(PAGE_URL)
110+
assert await caqui_driver.save_screenshot("/tmp/test.png")
97111

98-
@mark.asyncio_cooperative
99-
async def test_object_to_string(setup_environment: AsyncPage):
100-
page = setup_environment
101-
element_string = synchronous.find_element(page.remote, page.session, By.XPATH, "//button")
102-
element = await page.find_element(locator=By.XPATH, value="//button")
103-
assert str(element) == element_string
104112

113+
@mark.asyncio_cooperative
114+
async def test_click(caqui_driver: AsyncDriver):
115+
await caqui_driver.get(PAGE_URL)
116+
element = await caqui_driver.find_element(By.XPATH, "//button")
117+
await element.click()
105118
```
106119

107-
## Running as multiprocessing
108-
To run the tests in multiple processes use [pytest-xdist](https://github.com/pytest-dev/pytest-xdist). The execution is even faster than running in multiple tasks. Check this article [Supercharge Your Web Crawlers with Caqui: Boosting Speed with Multi-Processing](https://medium.com/@douglas.dcm/speed-up-your-web-crawlers-at-90-148f3ca97b6) to know how to increase the velocity of the executions in 90%.
120+
Running tests this way significantly reduces execution time, especially when interacting with multiple drivers or sessions.
121+
122+
---
123+
124+
# Running Tests with Multiprocessing
125+
126+
If your workloads benefit from multiple processes, Caqui also works with **pytest-xdist**.
127+
This approach is often faster than cooperative multitasking.
128+
129+
A guide to optimizing performance (including a real benchmark):
130+
[Speed up your web crawlers at 90%](https://medium.com/@douglas.dcm/speed-up-your-web-crawlers-at-90-148f3ca97b6)
131+
132+
Example:
109133

110134
```python
111135
@mark.asyncio
112-
async def test_save_screenshot(setup_environment: AsyncPage):
113-
page = setup_environment
114-
assert await page.save_screenshot("/tmp/test.png") is True
136+
async def test_save_screenshot(caqui_driver: AsyncDriver):
137+
await caqui_driver.get(PAGE_URL)
138+
assert await caqui_driver.save_screenshot("/tmp/test.png")
139+
115140

116141
@mark.asyncio
117-
async def test_object_to_string(setup_environment: AsyncPage):
118-
page = setup_environment
119-
element_string = synchronous.find_element(page.remote, page.session, By.XPATH, "//button")
120-
element = await page.find_element(locator=By.XPATH, value="//button")
121-
assert str(element) == element_string
142+
async def test_click(caqui_driver: AsyncDriver):
143+
await caqui_driver.get(PAGE_URL)
144+
element = await caqui_driver.find_element(By.XPATH, "//button")
145+
await element.click()
122146

123147
```
124148

125-
# Driver as a server
126-
In case you are using Appium, Winium or other driver not started by the library, just start the driver as a server.
149+
---
127150

128-
For example. Download the same [ChromeDriver](https://chromepage.chromium.org/downloads) version as your installed Chrome and start the Driver as a server using the port "9999"
151+
# Running a Driver as a Server
152+
153+
If you use external drivers such as Appium, Winium, or a standalone ChromeDriver, run them as servers and point Caqui to their URL.
154+
155+
Example for ChromeDriver on port 9999:
129156

130157
```bash
131158
$ ./chromedriver --port=9999
@@ -134,11 +161,24 @@ Only local connections are allowed.
134161
Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
135162
ChromeDriver was started successfully.
136163
```
137-
# Webdriver Manager
138164
139-
Caqui depends on [Webdriver Manager](https://pypi.org/project/webdriver-manager/) that can be configured independenly and has some limitations. Check the project documentation for more information.
165+
---
166+
167+
# WebDriver Manager
168+
169+
Caqui’s `LocalServer` class uses [Webdriver Manager](https://pypi.org/project/webdriver-manager/).
170+
The tool comes with its own constraints.
171+
Check its documentation for details if you need custom driver handling.
140172
173+
---
141174
142175
# Contributing
143-
Read the [Code of Conduct](https://github.com/douglasdcm/caqui/blob/main/docs/CODE_OF_CONDUCT.md) before push new Merge Requests.
144-
Now, follow the steps in [Contributing](https://github.com/douglasdcm/caqui/blob/main/docs/CONTRIBUTING.md) session.
176+
177+
Before submitting a pull request, review the project guidelines:
178+
Code of Conduct:
179+
[CODE OF CONDUCT](https://github.com/douglasdcm/caqui/blob/main/docs/CODE_OF_CONDUCT.md)
180+
181+
Contribution Guide:
182+
[CONTRIBUTING](https://github.com/douglasdcm/caqui/blob/main/docs/CONTRIBUTING.md)
183+
184+
Contributions, issue reports, and performance feedback are welcome.

‎caqui/__init__.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
# Copyright (C) 2023 Caqui - All Rights Reserved
2+
# You may use, distribute and modify this code under the
3+
# terms of the MIT license.
4+
# Visit: https://github.com/douglasdcm/caqui

0 commit comments

Comments
 (0)