Skip to content

Commit 09a703d

Browse files
committed
feat(cli): add full install command
1 parent d33ef02 commit 09a703d

19 files changed

Lines changed: 273 additions & 151 deletions

File tree

README.md

Lines changed: 11 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ Documentation: [https://kacigaya.github.io/webskrap/](https://kacigaya.github.io
1717

1818
```bash
1919
pip install webskrap
20-
python -m playwright install chromium
20+
webskrap install
2121
```
2222

2323
## Quickstart
@@ -233,10 +233,8 @@ the [`patchright`](https://github.com/Kaliiiiiiiiii-Vinyzu/patchright) driver, a
233233
CDP-leak-free Playwright fork, and let the browser's real fingerprint show through.
234234
WebSkrap does not inject JavaScript stealth patches.
235235

236-
```bash
237-
pip install "webskrap[stealth]"
238-
patchright install chromium
239-
```
236+
`pip install webskrap` includes Patchright. Run `webskrap install` to download
237+
browser binaries.
240238

241239
```python
242240
from pathlib import Path
@@ -362,12 +360,12 @@ flags, but pages that need WebGL or canvas export may not work correctly.
362360

363361
## CLI
364362

365-
`webskrap fetch` always runs headless Patchright stealth mode. Install the
366-
stealth extra before using it. `webskrap doctor` verifies this CLI setup.
363+
`webskrap fetch` always runs headless Patchright stealth mode. `webskrap install`
364+
downloads the browser binaries, and `webskrap doctor` verifies this CLI setup.
367365

368366
```bash
369-
pip install "webskrap[stealth]"
370-
patchright install chromium
367+
pip install webskrap
368+
webskrap install
371369
```
372370

373371
```bash
@@ -398,8 +396,8 @@ Desktop, Claude Code, ...) can drive scraping directly. It exposes three tools
398396
over stdio: `fetch`, `stealth_fetch`, and `doctor`.
399397

400398
```bash
401-
pip install "webskrap[mcp]"
402-
python -m playwright install chromium
399+
pip install webskrap
400+
webskrap install
403401
webskrap-mcp
404402
```
405403

@@ -431,8 +429,8 @@ command = "webskrap-mcp"
431429
args = []
432430
```
433431

434-
`stealth_fetch` also needs the `stealth` extra (`pip install "webskrap[mcp,stealth]"`
435-
and `patchright install chromium`).
432+
The legacy extras `webskrap[mcp]` and `webskrap[stealth]` remain accepted for
433+
older install snippets, but MCP and Patchright are included by default.
436434

437435
## Performance benchmarks
438436

SKILL.md

Lines changed: 11 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ Install normal API/browser support:
3737

3838
```bash
3939
pip install webskrap
40-
python -m playwright install chromium
40+
webskrap install
4141
```
4242

4343
Use `WebSkrapClient` as an async context manager. Use `client.fetch()` for a
@@ -60,13 +60,8 @@ async def main() -> None:
6060
asyncio.run(main())
6161
```
6262

63-
The Python API defaults to Playwright. For Patchright, opt in through
64-
`SessionConfig`:
65-
66-
```bash
67-
pip install "webskrap[stealth]"
68-
patchright install chromium
69-
```
63+
The Python API defaults to Playwright. `pip install webskrap` includes Patchright
64+
and MCP dependencies. For Patchright, opt in through `SessionConfig`:
7065

7166
```python
7267
from pathlib import Path
@@ -104,12 +99,13 @@ config = SessionConfig(
10499

105100
## CLI
106101

107-
The CLI `fetch` command always uses headless Patchright stealth mode. Install
108-
the stealth extra first; `webskrap doctor` checks this CLI setup.
102+
The CLI `fetch` command always uses headless Patchright stealth mode.
103+
`webskrap install` downloads Playwright and Patchright Chromium browsers;
104+
`webskrap doctor` checks this CLI setup.
109105

110106
```bash
111-
pip install "webskrap[stealth]"
112-
patchright install chromium
107+
pip install webskrap
108+
webskrap install
113109

114110
webskrap doctor
115111
webskrap doctor --format json
@@ -133,16 +129,15 @@ readable body text.
133129
Install MCP support when an MCP client should call WebSkrap directly:
134130

135131
```bash
136-
pip install "webskrap[mcp]"
137-
python -m playwright install chromium
132+
pip install webskrap
133+
webskrap install
138134
webskrap-mcp
139135
```
140136

141137
MCP tools:
142138

143139
- `fetch`: standard Playwright fetch.
144-
- `stealth_fetch`: Patchright fetch; requires `pip install "webskrap[mcp,stealth]"`
145-
and `patchright install chromium`.
140+
- `stealth_fetch`: Patchright fetch.
146141
- `doctor`: Playwright/Chromium MCP readiness check.
147142

148143
## Validation

pyproject.toml

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
44

55
[project]
66
name = "webskrap"
7-
version = "0.5.3"
7+
version = "0.5.4"
88
description = "A Playwright-based Python scraping framework with coherent browser profiles and session controls."
99
readme = "README.md"
1010
requires-python = ">=3.11"
@@ -24,6 +24,8 @@ classifiers = [
2424
"Typing :: Typed",
2525
]
2626
dependencies = [
27+
"mcp>=1.2",
28+
"patchright>=1.60.0",
2729
"playwright>=1.49",
2830
"pydantic>=2.8",
2931
"rich>=13.9",
@@ -37,10 +39,8 @@ dev = [
3739
"ruff>=0.8",
3840
]
3941
stealth = [
40-
"patchright>=1.60.0",
4142
]
4243
mcp = [
43-
"mcp>=1.2",
4444
]
4545

4646
[project.scripts]

src/webskrap/cli.py

Lines changed: 55 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,8 @@
22

33
import asyncio
44
import json
5+
import subprocess
6+
import sys
57
from pathlib import Path
68
from typing import Annotated, Literal
79

@@ -26,6 +28,29 @@
2628
app = typer.Typer(help="WebSkrap browser scraping toolkit.")
2729
console = Console()
2830
OutputFormat = Literal["human", "json"]
31+
INSTALL_COMMANDS = (
32+
(sys.executable, "-m", "playwright", "install", "chromium"),
33+
("patchright", "install", "chromium"),
34+
)
35+
36+
37+
@app.command("install")
38+
def install_command(
39+
format: Annotated[
40+
str,
41+
typer.Option("--format", help="Output format: human or json."),
42+
] = "human",
43+
) -> None:
44+
output_format = _parse_output_format(format)
45+
results = [_run_install_command(command) for command in INSTALL_COMMANDS]
46+
ok = all(result["ok"] for result in results)
47+
payload = {"ok": ok, "steps": results}
48+
if output_format == "json":
49+
_print_json(payload)
50+
else:
51+
_print_install_result(results)
52+
if not ok:
53+
raise typer.Exit(code=1)
2954

3055

3156
@app.command("profiles")
@@ -84,7 +109,7 @@ async def _doctor() -> dict[str, object]:
84109
return {
85110
"ok": False,
86111
"message": f"Patchright import failed: {exc}",
87-
"hint": 'Run: pip install "webskrap[stealth]" && patchright install chromium',
112+
"hint": "Run: webskrap install",
88113
}
89114

90115
try:
@@ -97,7 +122,7 @@ async def _doctor() -> dict[str, object]:
97122
return {
98123
"ok": False,
99124
"message": f"Patchright headless Chrome did not launch: {exc}",
100-
"hint": 'Run: pip install "webskrap[stealth]" && patchright install chromium',
125+
"hint": "Run: webskrap install",
101126
}
102127

103128
return {"ok": True, "message": "Patchright headless Chrome is ready."}
@@ -321,6 +346,34 @@ def _print_json(payload: object) -> None:
321346
typer.echo(json.dumps(payload, ensure_ascii=False))
322347

323348

349+
def _run_install_command(command: tuple[str, ...]) -> dict[str, object]:
350+
try:
351+
completed = subprocess.run(command, capture_output=True, text=True, check=False)
352+
except OSError as exc:
353+
return {
354+
"ok": False,
355+
"command": list(command),
356+
"message": str(exc),
357+
}
358+
output = (completed.stdout or completed.stderr).strip()
359+
return {
360+
"ok": completed.returncode == 0,
361+
"command": list(command),
362+
"message": output,
363+
}
364+
365+
366+
def _print_install_result(results: list[dict[str, object]]) -> None:
367+
for result in results:
368+
command = " ".join(str(part) for part in result["command"])
369+
if result["ok"]:
370+
console.print(f"[green]OK:[/green] {command}")
371+
else:
372+
console.print(f"[red]FAILED:[/red] {command}")
373+
if result["message"]:
374+
console.print(str(result["message"]))
375+
376+
324377
def _print_doctor_result(result: dict[str, object]) -> None:
325378
message = str(result["message"])
326379
if result["ok"]:

src/webskrap/client.py

Lines changed: 3 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -21,17 +21,14 @@ def _async_playwright(driver: str):
2121
2222
``patchright`` is a drop-in, API-compatible fork of Playwright that hides the
2323
CDP ``Runtime.enable`` leak used by CDP-aware bot detectors. It requires the
24-
optional ``stealth`` extra (``pip install webskrap[stealth]``) and its own
25-
browser download (``patchright install chromium``).
24+
``patchright`` ships with WebSkrap, but needs its browser download
25+
(``webskrap install``).
2626
"""
2727
if driver == "patchright":
2828
try:
2929
from patchright.async_api import async_playwright
3030
except ImportError as exc: # pragma: no cover - optional dependency
31-
msg = (
32-
"driver='patchright' requires the optional dependency: "
33-
"pip install webskrap[stealth] && patchright install chromium"
34-
)
31+
msg = "driver='patchright' requires patchright. Run: pip install webskrap"
3532
raise WebSkrapError(msg) from exc
3633
return async_playwright()
3734
from playwright.async_api import async_playwright

src/webskrap/mcp_server.py

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
"""Model Context Protocol server exposing WebSkrap over stdio.
22
3-
Run with ``webskrap-mcp`` (after ``pip install webskrap[mcp]``) or
3+
Run with ``webskrap-mcp`` (after ``pip install webskrap``) or
44
``python -m webskrap.mcp_server``. Point an MCP client (Claude Desktop, Claude
55
Code, ...) at that command to drive scraping through the tools below.
66
"""
@@ -22,7 +22,7 @@
2222
try:
2323
from mcp.server.fastmcp import FastMCP
2424
except ImportError as exc: # pragma: no cover - optional dependency
25-
msg = "the MCP server requires the optional dependency: pip install webskrap[mcp]"
25+
msg = "the MCP server requires mcp. Run: pip install webskrap"
2626
raise WebSkrapError(msg) from exc
2727

2828
mcp = FastMCP("webskrap")
@@ -78,9 +78,8 @@ async def stealth_fetch(
7878
) -> dict[str, Any]:
7979
"""Fetch a URL with the Patchright stealth driver (CDP-leak-free).
8080
81-
Requires the optional stealth extra: pip install webskrap[stealth] and
82-
patchright install chromium. Prefer headless=False with channel="chrome"
83-
for the strictest anti-bot path.
81+
Requires Patchright's browser download: webskrap install. Prefer
82+
headless=False with channel="chrome" for the strictest anti-bot path.
8483
8584
Args:
8685
url: The URL to load.
@@ -135,7 +134,7 @@ async def doctor() -> dict[str, Any]:
135134
return {
136135
"ok": False,
137136
"message": f"Chromium did not launch: {exc}",
138-
"hint": "Run: python -m playwright install chromium",
137+
"hint": "Run: webskrap install",
139138
}
140139

141140
return {"ok": True, "message": "Playwright and Chromium are ready."}

tests/test_bot_detection.py

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,8 +25,7 @@
2525
2626
Compare this strict headed suite with ``test_bot_detection_headless.py`` for the
2727
current headless baseline. Requires Google Chrome installed on the host and the
28-
optional stealth extra (``pip install webskrap[stealth]`` plus
29-
``patchright install chromium``).
28+
Patchright browser download (``webskrap install``).
3029
"""
3130

3231
from __future__ import annotations

tests/test_cli.py

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
from __future__ import annotations
22

33
import json
4+
import subprocess
45
from typing import Any
56

67
from typer.testing import CliRunner
@@ -135,3 +136,76 @@ async def fake_doctor() -> dict[str, object]:
135136

136137
assert result.exit_code == 1
137138
assert json.loads(result.output) == {"ok": False, "message": "broken", "hint": "fix it"}
139+
140+
141+
def test_install_json_success(monkeypatch: Any) -> None:
142+
calls: list[tuple[str, ...]] = []
143+
144+
def fake_run(command: tuple[str, ...], **_kwargs: Any) -> subprocess.CompletedProcess[str]:
145+
calls.append(command)
146+
return subprocess.CompletedProcess(command, 0, stdout="installed", stderr="")
147+
148+
monkeypatch.setattr(cli.subprocess, "run", fake_run)
149+
150+
result = runner.invoke(cli.app, ["install", "--format", "json"])
151+
152+
assert result.exit_code == 0, result.output
153+
payload = json.loads(result.output)
154+
assert payload["ok"] is True
155+
assert [tuple(step["command"]) for step in payload["steps"]] == list(cli.INSTALL_COMMANDS)
156+
assert calls == list(cli.INSTALL_COMMANDS)
157+
158+
159+
def test_install_json_failure(monkeypatch: Any) -> None:
160+
def fake_run(command: tuple[str, ...], **_kwargs: Any) -> subprocess.CompletedProcess[str]:
161+
return_code = 1 if command[0] == "patchright" else 0
162+
return subprocess.CompletedProcess(
163+
command,
164+
return_code,
165+
stdout="",
166+
stderr="missing browser",
167+
)
168+
169+
monkeypatch.setattr(cli.subprocess, "run", fake_run)
170+
171+
result = runner.invoke(cli.app, ["install", "--format", "json"])
172+
173+
assert result.exit_code == 1
174+
payload = json.loads(result.output)
175+
assert payload["ok"] is False
176+
assert payload["steps"][0]["ok"] is True
177+
assert payload["steps"][1]["ok"] is False
178+
assert payload["steps"][1]["message"] == "missing browser"
179+
180+
181+
def test_install_json_handles_missing_executable(monkeypatch: Any) -> None:
182+
def fake_run(command: tuple[str, ...], **_kwargs: Any) -> subprocess.CompletedProcess[str]:
183+
if command[0] == "patchright":
184+
raise FileNotFoundError("missing patchright")
185+
return subprocess.CompletedProcess(command, 0, stdout="installed", stderr="")
186+
187+
monkeypatch.setattr(cli.subprocess, "run", fake_run)
188+
189+
result = runner.invoke(cli.app, ["install", "--format", "json"])
190+
191+
assert result.exit_code == 1
192+
payload = json.loads(result.output)
193+
assert payload["ok"] is False
194+
assert payload["steps"][1]["ok"] is False
195+
assert payload["steps"][1]["message"] == "missing patchright"
196+
197+
198+
def test_install_human_output(monkeypatch: Any) -> None:
199+
def fake_run(command: tuple[str, ...], **_kwargs: Any) -> subprocess.CompletedProcess[str]:
200+
return subprocess.CompletedProcess(command, 0, stdout="installed", stderr="")
201+
202+
monkeypatch.setattr(cli.subprocess, "run", fake_run)
203+
204+
result = runner.invoke(cli.app, ["install"])
205+
206+
assert result.exit_code == 0, result.output
207+
assert "OK:" in result.output
208+
assert "playwright" in result.output
209+
assert "install" in result.output
210+
assert "chromium" in result.output
211+
assert "patchright install chromium" in result.output

0 commit comments

Comments
 (0)