feat(mcp): make fetch stealth by default

fetch now runs the Patchright headless-Chrome stealth path and waits for
networkidle, matching the CLI. JS-heavy and anti-bot pages (SPAs, login
walls) that returned empty text under plain Playwright now load. Adds a
channel arg; raises default timeout to 60s. stealth_fetch stays for finer
fingerprint/WebRTC/UA control.

Bump 0.5.8 -> 0.5.9.

Author: kacigaya

README.md

@@ -400,14 +400,15 @@ WebSkrap ships a Model Context Protocol server so MCP clients (Claude Desktop,
 Claude Code, Codex, ...) can drive a real browser directly. It exposes three
 tools over stdio: `fetch`, `stealth_fetch`, and `doctor`.
 
-Built for LLMs: `fetch` and `stealth_fetch` return **clean visible page text by
-default** — no HTML tags, scripts, or style noise — so agents spend tokens on
-content, not markup (typically 5-10x fewer tokens than raw HTML). Pass
-`text_only=False` when you actually need the HTML. `stealth_fetch` gives agents
-the same CDP-leak-free Patchright path the CLI uses, so anti-bot pages that
-block naive scrapers still load. Every result carries `status`, `final_url`,
-`title`, `text_length`, and truncation flags so the model knows exactly what it
-got.
+Built for LLMs: both `fetch` and `stealth_fetch` run the same CDP-leak-free
+Patchright stealth path the CLI uses (headless Chrome, `networkidle` wait), so
+JS-heavy and anti-bot pages that block naive scrapers still load. They return
+**clean visible page text by default** — no HTML tags, scripts, or style noise —
+so agents spend tokens on content, not markup (typically 5-10x fewer tokens than
+raw HTML). Pass `text_only=False` when you actually need the HTML. Use
+`stealth_fetch` for finer control (fingerprint surface, WebRTC, UA masking,
+persistent profile). Every result carries `status`, `final_url`, `title`,
+`text_length`, and truncation flags so the model knows exactly what it got.
 
 ```bash
 pip install webskrap

SKILL.md

@@ -143,8 +143,8 @@ webskrap-mcp
 
 MCP tools:
 
-- `fetch`: standard Playwright fetch.
-- `stealth_fetch`: Patchright fetch.
+- `fetch`: Patchright stealth fetch (headless Chrome, waits for networkidle).
+- `stealth_fetch`: stealth fetch with finer fingerprint/WebRTC/UA controls.
 - `doctor`: Playwright/Chromium MCP readiness check.
 
 ## Validation

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "webskrap"
-version = "0.5.8"
+version = "0.5.9"
 description = "A Playwright-based Python scraping framework with coherent browser profiles and session controls."
 readme = "README.md"
 requires-python = ">=3.11"

src/webskrap/mcp_server.py

@@ -32,27 +32,37 @@
 async def fetch(
     url: str,
     profile: str = "desktop-chrome",
-    wait_until: str = "domcontentloaded",
+    channel: str = "chrome",
+    wait_until: str = "networkidle",
     resource_policy: str = "all",
-    timeout_ms: float = 30_000,
+    timeout_ms: float = 60_000,
     max_chars: int = 20_000,
     text_only: bool = True,
 ) -> dict[str, Any]:
-    """Fetch a URL with a standard Playwright browser and return page data.
+    """Fetch a URL with the Patchright stealth driver and return page data.
 
-    Returns clean visible page text by default (LLM-friendly, no HTML tags).
-    Set text_only=False to get the raw HTML instead.
+    Uses the same CDP-leak-free headless-Chrome stealth path as the CLI, so
+    JS-heavy and anti-bot pages that block naive scrapers still load. Waits for
+    networkidle by default so single-page apps have hydrated before reading.
+    Returns clean visible page text by default (LLM-friendly, no HTML tags); set
+    text_only=False to get the raw HTML instead. For finer stealth control
+    (fingerprint surface, WebRTC, UA masking, persistent profile) use
+    stealth_fetch.
 
     Args:
         url: The URL to load.
         profile: Bundled profile (desktop-chrome, desktop-edge, mobile-chrome).
+        channel: Browser channel, e.g. chrome. Use chromium on Linux ARM64.
         wait_until: commit, domcontentloaded, load, or networkidle.
         resource_policy: all, lite (block images/fonts/media), or documents.
         timeout_ms: Navigation timeout in milliseconds.
         max_chars: Maximum characters of page text to return.
         text_only: Return clean visible text (default) instead of raw HTML.
     """
     config = SessionConfig(
+        driver="patchright",
+        channel=channel,
+        headless=True,
         navigation_timeout_ms=timeout_ms,
         resource_policy=parse_resource_policy(resource_policy),
     )

tests/test_mcp.py

@@ -46,6 +46,18 @@ def test_fetch_defaults_to_clean_text(monkeypatch: Any) -> None:
     assert result["text"] == "Readable body"
 
 
+def test_fetch_uses_stealth_driver_by_default(monkeypatch: Any) -> None:
+    _fake_client(monkeypatch)
+
+    asyncio.run(mcp_server.fetch("https://example.test"))
+
+    config = _FakeClient.calls[0]["config"]
+    assert config.driver == "patchright"
+    assert config.channel == "chrome"
+    assert config.headless is True
+    assert _FakeClient.calls[0]["wait_until"] == "networkidle"
+
+
 def test_fetch_text_only_false_returns_html(monkeypatch: Any) -> None:
     _fake_client(monkeypatch)
 

web/content/user-guide/mcp.md

@@ -4,12 +4,13 @@ WebSkrap ships a [Model Context Protocol](https://modelcontextprotocol.io)
 server. MCP clients such as Claude Desktop, Claude Code, and Codex can call it to
 drive a real browser directly. It runs over stdio and exposes three tools.
 
-**Built for LLMs.** `fetch` and `stealth_fetch` return clean visible page text
-by default — no HTML tags, scripts, or CSS noise — so the model spends tokens on
-content, not markup (typically 5-10x fewer tokens than raw HTML). `stealth_fetch`
-gives agents the same CDP-leak-free Patchright path the CLI uses, so anti-bot
-pages that block naive scrapers still load. Pass `text_only=false` when you
-actually need the HTML.
+**Built for LLMs.** Both `fetch` and `stealth_fetch` run the same CDP-leak-free
+Patchright stealth path the CLI uses (headless Chrome, `networkidle` wait), so
+JS-heavy and anti-bot pages that block naive scrapers still load. They return
+clean visible page text by default — no HTML tags, scripts, or CSS noise — so
+the model spends tokens on content, not markup (typically 5-10x fewer tokens
+than raw HTML). Use `stealth_fetch` for finer fingerprint/WebRTC/UA control.
+Pass `text_only=false` when you actually need the HTML.
 
 ## Install
 
@@ -34,8 +35,8 @@ python -m webskrap.mcp_server
 
 | Tool | Purpose |
 | --- | --- |
-| `fetch` | Fetch a URL with a standard Playwright browser. |
-| `stealth_fetch` | Fetch a URL with the Patchright stealth driver. |
+| `fetch` | Fetch a URL with the Patchright stealth driver (waits for `networkidle`). |
+| `stealth_fetch` | Same stealth driver with finer fingerprint/WebRTC/UA controls. |
 | `doctor` | Check that Playwright and Chromium can launch. |
 
 Both fetch tools return `status`, `final_url`, `title`, `ok`, `headers`, and the
@@ -51,9 +52,10 @@ text; set `text_only` to `false` to get raw HTML.
 | --- | --- | --- |
 | `url` | required | URL to load. |
 | `profile` | `desktop-chrome` | Bundled profile name. |
-| `wait_until` | `domcontentloaded` | `commit`, `domcontentloaded`, `load`, or `networkidle`. |
+| `channel` | `chrome` | Browser channel; use `chromium` on Linux ARM64. |
+| `wait_until` | `networkidle` | `commit`, `domcontentloaded`, `load`, or `networkidle`. |
 | `resource_policy` | `all` | `all`, `lite`, or `documents`. |
-| `timeout_ms` | `30000` | Navigation timeout. |
+| `timeout_ms` | `60000` | Navigation timeout. |
 | `max_chars` | `20000` | Maximum returned text characters. |
 | `text_only` | `true` | Return clean visible text; set `false` for raw HTML. |