You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
description: Use when writing, debugging, documenting, or reviewing Python scraping and browser automation code with WebSkrap. Covers async fetches, persistent sessions, Playwright/Patchright drivers, browser profiles, resource policies, screenshots, proxies, timeouts, CLI usage, MCP server usage, and safe examples that avoid CAPTCHA solving, login-wall bypassing, credential bypassing, or access-control circumvention.
3
+
description: Use when writing, debugging, documenting, or reviewing Python scraping and browser automation code with WebSkrap. Covers async fetches, persistent sessions, Playwright/Patchright drivers, browser profiles, resource policies, screenshots, proxies, timeouts, LLM-friendly CLI output, MCP server usage, and safe examples that avoid CAPTCHA solving, login-wall bypassing, credential bypassing, or access-control circumvention.
4
4
---
5
5
6
6
# WebSkrap
7
7
8
-
WebSkrap is an async-first Python scraping framework built on Playwright for realistic browser sessions, coherent profiles, resource controls, and structured fetch results.
8
+
WebSkrap is an async-first Python scraping package built on Playwright, with
9
+
Patchright support for stealth-oriented browser sessions.
9
10
10
-
## Start Here
11
+
## Read First
11
12
12
-
Before giving detailed guidance, check the local docs when available:
13
+
Prefer current repo sources over memory:
13
14
14
-
-`docs/getting-started/quickstart.md`: first examples for fetches, sessions, clicks, screenshots, and CLI.
15
-
-`docs/user-guide/client.md`: `WebSkrapClient`, one-shot fetches, timeouts, screenshots, and `FetchResult`.
16
-
-`docs/user-guide/sessions.md`: persistent sessions, headed debugging, page workflows, and human-like clicks.
17
-
-`docs/user-guide/profiles.md`: built-in and custom browser profiles.
-`src/webskrap/cli.py`: current `webskrap` command behavior.
20
+
-`src/webskrap/mcp_server.py`: MCP tools and argument shape.
21
+
-`tests/`: behavior contracts when changing parsing, state, CLI, stealth, or safety.
23
22
24
23
## Guardrails
25
24
26
-
- Do not add CAPTCHA solving, login-wall bypassing, credential bypassing, or access-control circumvention.
27
-
- Keep examples limited to public pages, local servers, or targets the user is allowed to access.
28
-
- Do not commit cookies, storage state, proxy credentials, or persistent browser data such as `.webskrap/`.
29
-
- Prefer typed public APIs exported by `webskrap`; avoid reaching into private helpers except in tests.
30
-
- Use `async with WebSkrapClient()` unless you are explicitly demonstrating manual `start()` / `close()`.
25
+
- Do not add CAPTCHA solving, login-wall bypassing, credential bypassing, or
26
+
access-control circumvention.
27
+
- Use public pages, local test servers, or targets the user is allowed to access.
28
+
- Do not commit cookies, storage state, proxy credentials, or persistent browser
29
+
data such as `.webskrap/`.
30
+
- Prefer public exports from `webskrap`; use private helpers only in tests.
31
+
- Keep changes small and typed. Add focused tests for parsing, state transitions,
32
+
CLI output, and tool-safety behavior.
31
33
32
-
## Core Workflow
34
+
## Python API
33
35
34
-
Install WebSkrap and the Playwright browser before running examples:
36
+
Install normal API/browser support:
35
37
36
38
```bash
37
39
pip install webskrap
38
40
python -m playwright install chromium
39
41
```
40
42
41
-
Use `WebSkrapClient` as an async context manager. Use `client.fetch()` for one-shot page fetches. Use `client.session()` when browser state, cookies, local storage, human-like clicks, or a headed browser should persist across actions.
42
-
43
-
## Common Patterns
44
-
45
-
One-shot fetch:
43
+
Use `WebSkrapClient` as an async context manager. Use `client.fetch()` for a
44
+
one-shot fetch. Use `client.session()` when cookies, storage, manual page work,
45
+
human-like clicks, or headed debugging should persist.
46
46
47
47
```python
48
48
import asyncio
@@ -60,123 +60,35 @@ async def main() -> None:
60
60
asyncio.run(main())
61
61
```
62
62
63
-
Fetch with a screenshot and custom navigation wait:
64
-
65
-
```python
66
-
result =await client.fetch(
67
-
"https://example.com",
68
-
wait_until="load",
69
-
screenshot="example.png",
70
-
timeout_ms=60_000,
71
-
)
72
-
print(result.screenshot_path)
73
-
```
74
-
75
-
Persistent session with storage between runs:
76
-
77
-
```python
78
-
import asyncio
79
-
from pathlib import Path
80
-
81
-
from webskrap import SessionConfig, WebSkrapClient
The Python API defaults to Playwright. For Patchright, opt in through
64
+
`SessionConfig`:
144
65
145
66
```bash
146
67
pip install "webskrap[stealth]"
147
68
patchright install chromium
148
69
```
149
70
150
71
```python
72
+
from pathlib import Path
73
+
151
74
from webskrap import SessionConfig
152
75
153
76
config = SessionConfig(
154
77
driver="patchright",
155
78
channel="chrome",
156
-
headless=False,
79
+
headless=True,
80
+
user_data_dir=Path(".webskrap/headless-profile"),
157
81
)
158
82
```
159
83
160
-
Use Patchright for CDP-aware detection surfaces. It uses a persistent context for full stealth; if `user_data_dir` is omitted, WebSkrap creates a temporary persistent profile. WebSkrap does not inject JavaScript stealth patches, so let the real browser fingerprint show through.
161
-
162
-
Headless Patchright is best-effort. Prefer real Chrome, a stable `user_data_dir`, a coherent virtual screen, and the browser-level user-agent mask only when the user asks for headless stealth:
84
+
Headed Patchright is strongest for strict detection surfaces:
These settings do not hide the page's normal remote address and do not normalize unrelated high-entropy surfaces such as fonts, battery, TLS/session metadata, or installed browser features.
0 commit comments