fix(cli): fallback to chromium and cap consent waits

Default --channel chrome dumped a Playwright traceback on platforms
with no Chrome build. Cookie decline could run ~12s despite its 2s
option, and its selectors had no coverage against real consent
platforms.

Author: kacigaya

README.md

@@ -152,45 +152,27 @@ asyncio.run(main())
 
 ## Auto-decline cookie banners
 
-Every `session.fetch()` (and therefore the CLI and the MCP tools) clicks the
-reject control of a cookie consent notice before reading the page, so scraped
-text is not buried under a banner and no optional cookies are accepted.
-
-Detection is inspired by Brave's
+Every `fetch()` (and therefore the CLI and the MCP tools) clicks the reject
+control of a cookie consent notice before reading the page, so scraped text is
+not buried under a banner and no optional cookies are accepted. It knows the
+common consent platforms (OneTrust, Cookiebot, Didomi, Usercentrics,
+Sourcepoint, ...) and falls back to matching reject wording inside a
+cookie/consent container, in every frame. Detection is inspired by Brave's
 [cookiecrumbler](https://github.com/brave/cookiecrumbler), which finds consent
-notices; WebSkrap dismisses them. Two strategies, tried in every frame:
-
-1. Reject buttons of the common consent platforms (OneTrust, Cookiebot, Didomi,
-   Usercentrics, Sourcepoint, Quantcast, Osano, Complianz, CookieYes, ...).
-2. A clickable whose label matches a reject phrase ("Reject all", "Refuser
-   tout", "Ablehnen", "Continue without accepting", ...), scoped to a
-   cookie/consent container so unrelated "Decline" buttons are never touched.
-
-`FetchResult.cookie_notice_declined` reports which strategy clicked (`"cmp"`,
-`"text"`, or `None`).
+notices; WebSkrap dismisses them.
 
 ```python
 config = SessionConfig(
-    decline_cookies=True,             # default
-    decline_cookies_timeout_ms=2_000,  # wait for a late-injected notice; 0 = check once
+    decline_cookies=True,              # default
+    decline_cookies_timeout_ms=2_000,  # wait for a late notice; 0 = check once
 )
-
 result = await client.fetch("https://example.com", config=config)
-print(result.cookie_notice_declined)
+print(result.cookie_notice_declined)   # "cmp", "text", or None
 ```
 
-The timeout is the per-fetch cost on pages without a notice. Pages that carry a
-consent-platform iframe are retried for up to 3s while it renders. For pages you
-drive yourself, call it directly:
-
-```python
-page = await session.context.new_page()
-await page.goto("https://example.com", wait_until="domcontentloaded")
-await session.decline_cookies(page)
-```
-
-Consent walls with no reject control on the first layer (pay-or-consent) are
-left alone.
+Full reference, including `session.decline_cookies(page)` for pages you drive
+yourself:
+[docs/user-guide/client](https://kacigaya.github.io/webskrap/docs/user-guide/client/).
 
 ## Custom profile
 
@@ -403,7 +385,11 @@ flags, but pages that need WebGL or canvas export may not work correctly.
 ## CLI
 
 `webskrap fetch` always runs headless Patchright stealth mode. `webskrap install`
-downloads the browser binaries, and `webskrap doctor` verifies this CLI setup.
+downloads the browser binaries, and `webskrap doctor` verifies this CLI setup and
+reports which browser channel launches.
+
+`--channel` defaults to `chrome`. Where Chrome does not exist (Linux ARM64),
+`fetch` warns on stderr and retries with bundled chromium instead of failing.
 
 ```bash
 pip install webskrap

SKILL.md

@@ -129,8 +129,9 @@ webskrap fetch https://example.com --stdout --text-only
 webskrap fetch https://example.com --quiet --output page.html
 ```
 
-On Linux ARM64, the `chrome` channel can be unsupported. Prefer `chrome` where
-it launches, but use Chromium fallback on Linux ARM64:
+On Linux ARM64, the `chrome` channel can be unsupported. `fetch` detects a
+launch failure and retries with bundled chromium automatically; pass the channel
+explicitly to skip the failed first attempt:
 
 ```bash
 webskrap fetch https://example.com --channel chromium --format json
@@ -166,9 +167,14 @@ For non-trivial changes run:
 ```bash
 pytest -q
 ruff check .
+ruff format --check .
 python -m build
 ```
 
+CI (`.github/workflows/ci.yml`) runs the same gate on push and pull request, and
+`Publish` calls it before uploading to PyPI. PyPI versions are immutable, so a
+red gate must never be bypassed.
+
 Use `WEBSKRAP_LIVE=1 pytest -q -m live` only when explicitly checking public
-third-party bot-detection behavior. Those tests are opt-in and can fail when
-external demos change.
+third-party bot-detection behavior, or `tests/test_consent_live.py` for CMP
+selector rot. Those tests are opt-in and can fail when external sites change.

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "webskrap"
-version = "0.7.0"
+version = "0.7.1"
 description = "A Playwright-based Python scraping framework with coherent browser profiles and session controls."
 readme = "README.md"
 requires-python = ">=3.11"

src/webskrap/cli.py

@@ -9,14 +9,15 @@
 import urllib.request
 from importlib import metadata
 from pathlib import Path
-from typing import Annotated, Literal
+from typing import Annotated, Any, Literal, NoReturn
 
 import typer
 from rich.console import Console
 from rich.table import Table
 
 from webskrap.client import WebSkrapClient
 from webskrap.models import (
+    FetchResult,
     ResourcePolicy,
     SessionConfig,
     WaitUntil,
@@ -179,20 +180,32 @@ async def _doctor() -> dict[str, object]:
             "hint": "Run: webskrap install",
         }
 
-    try:
-        manager = async_playwright()
-        playwright = await manager.start()
-        browser = await playwright.chromium.launch(channel="chrome", headless=True)
-        await browser.close()
-        await playwright.stop()
-    except Exception as exc:
+    # The chrome channel is unavailable on some platforms (Linux ARM64), where
+    # bundled chromium still works. Report the best channel that launches
+    # instead of failing the whole check.
+    failure: Exception | None = None
+    for channel in ("chrome", None):
+        try:
+            manager = async_playwright()
+            playwright = await manager.start()
+            browser = await playwright.chromium.launch(channel=channel, headless=True)
+            await browser.close()
+            await playwright.stop()
+        except Exception as exc:  # noqa: BLE001 - try the next channel
+            failure = exc
+            continue
+        label = channel or "chromium"
         return {
-            "ok": False,
-            "message": f"Patchright headless Chrome did not launch: {exc}",
-            "hint": "Run: webskrap install",
+            "ok": True,
+            "message": f"Patchright headless {label} is ready.",
+            "channel": label,
         }
 
-    return {"ok": True, "message": "Patchright headless Chrome is ready."}
+    return {
+        "ok": False,
+        "message": f"Patchright headless Chrome did not launch: {failure}",
+        "hint": "Run: webskrap install",
+    }
 
 
 @app.command("fetch")
@@ -373,16 +386,15 @@ async def _fetch(
         webrtc_ip_handling_policy=_parse_webrtc_ip_handling_policy(webrtc_ip_handling_policy),
     )
 
-    async with WebSkrapClient() as client:
-        result = await client.fetch(
-            url,
-            profile=selected_profile,
-            config=config,
-            wait_until=_parse_wait_until(wait_until),
-            screenshot=screenshot or False,
-            timeout_ms=timeout_ms,
-            text_only=text_only,
-        )
+    result = await _fetch_with_channel_fallback(
+        config,
+        url=url,
+        profile=selected_profile,
+        wait_until=_parse_wait_until(wait_until),
+        screenshot=screenshot or False,
+        timeout_ms=timeout_ms,
+        text_only=text_only,
+    )
 
     if output:
         output.parent.mkdir(parents=True, exist_ok=True)
@@ -410,6 +422,58 @@ async def _fetch(
         console.print(f"[bold]HTML:[/bold] {output}")
 
 
+LAUNCH_FAILURE_MARKERS = (
+    "executable doesn't exist",
+    "is not found at",
+    "playwright install",
+    "failed to launch",
+    "browsertype.launch",
+)
+
+
+def _is_launch_failure(exc: Exception) -> bool:
+    return any(marker in str(exc).lower() for marker in LAUNCH_FAILURE_MARKERS)
+
+
+def _fail_launch(exc: Exception) -> NoReturn:
+    """Report an unlaunchable browser the way `doctor` does, not as a traceback."""
+    detail = str(exc).strip().splitlines()
+    stderr = Console(stderr=True, highlight=False)
+    stderr.print(f"[red]Browser did not launch:[/red] {detail[0] if detail else exc}")
+    stderr.print("Run: [bold]webskrap install[/bold]")
+    raise typer.Exit(code=1)
+
+
+async def _run_fetch(config: SessionConfig, **kwargs: Any) -> FetchResult:
+    async with WebSkrapClient() as client:
+        return await client.fetch(config=config, **kwargs)
+
+
+async def _fetch_with_channel_fallback(config: SessionConfig, **kwargs: Any) -> FetchResult:
+    """Fetch, retrying on bundled chromium when the chosen channel cannot launch.
+
+    The default channel is `chrome`, which does not exist on every platform
+    (Linux ARM64 has no Chrome build). Falling back keeps `webskrap fetch`
+    working there instead of dumping a Playwright traceback.
+    """
+    try:
+        return await _run_fetch(config, **kwargs)
+    except Exception as exc:
+        if not _is_launch_failure(exc):
+            raise
+        if config.channel is None:
+            _fail_launch(exc)
+        Console(stderr=True, highlight=False).print(
+            f"[yellow]channel '{config.channel}' did not launch; retrying with chromium[/yellow]"
+        )
+        try:
+            return await _run_fetch(config.model_copy(update={"channel": None}), **kwargs)
+        except Exception as retry_exc:
+            if not _is_launch_failure(retry_exc):
+                raise
+            _fail_launch(retry_exc)
+
+
 def _parse_wait_until(value: str) -> WaitUntil:
     try:
         return parse_wait_until(value)

src/webskrap/client.py

@@ -12,6 +12,7 @@
 
 from playwright.async_api import Browser, BrowserContext, Page, Playwright
 
+from webskrap.consent import SETTLED_PAGE_TIMEOUT_MS
 from webskrap.consent import decline_cookies as _decline_cookies
 from webskrap.models import BrowserProfile, FetchResult, ResourcePolicy, SessionConfig, WaitUntil
 from webskrap.profiles import get_profile
@@ -86,7 +87,11 @@ async def fetch(
             )
             declined = None
             if self.config.decline_cookies:
-                declined = await self.decline_cookies(page)
+                budget = self.config.decline_cookies_timeout_ms
+                if wait_until == "networkidle":
+                    # The navigation already waited out the CMP script.
+                    budget = min(budget, SETTLED_PAGE_TIMEOUT_MS)
+                declined = await self.decline_cookies(page, timeout_ms=budget)
             title = await page.title()
             text = await page.locator("body").inner_text() if text_only else await page.content()
             screenshot_path = await _maybe_screenshot(page, screenshot)

src/webskrap/consent.py

@@ -18,6 +18,7 @@
 
 import re
 from contextlib import suppress
+from time import monotonic
 from typing import Any
 
 # Reject/deny buttons of common consent management platforms.
@@ -163,8 +164,16 @@
 # A consent platform's iframe is visible before its contents finish rendering,
 # so retry while one is attached. Only pages that actually carry such a frame
 # pay this; everything else gets a single pass.
+# ponytail: fixed 3s poll ceiling for iframe render; wait on the reject locator
+# inside the frame if slow CMPs start slipping through
 POLL_INTERVAL_MS = 250
 POLL_ATTEMPTS = 12
+# Hard cap on everything after detection (polling plus click attempts), so the
+# total cost stays timeout_ms + this instead of growing with candidate count.
+POST_DETECT_BUDGET_MS = 3_000
+# A caller that already waited for networkidle gave CMP scripts their chance to
+# inject, so the detection wait on top of that is mostly dead time.
+SETTLED_PAGE_TIMEOUT_MS = 500
 
 
 async def decline_cookies(page: Any, *, timeout_ms: float = 2_000) -> str | None:
@@ -175,7 +184,16 @@ async def decline_cookies(page: Any, *, timeout_ms: float = 2_000) -> str | None
     the strategy that clicked (``"cmp"`` or ``"text"``), or ``None`` when no
     notice was found. Never raises: a page without a notice, or a notice this
     cannot handle, is not an error.
+
+    Total wall time is bounded by ``timeout_ms + POST_DETECT_BUDGET_MS``.
+
+    ponytail: only the notice's first layer is handled. Walls with no reject
+    control there (zeit.de and other pay-or-consent publishers) are left alone,
+    and walls that answer the reject click with a subscription upsell
+    (theguardian.com) stay up -- the return value reports the click, not a clean
+    page. Walking further needs per-CMP second-layer flows.
     """
+    deadline = monotonic() + (max(timeout_ms, 0.0) + POST_DETECT_BUDGET_MS) / 1000
     if timeout_ms > 0:
         try:
             await page.wait_for_selector(
@@ -186,13 +204,18 @@ async def decline_cookies(page: Any, *, timeout_ms: float = 2_000) -> str | None
 
     for attempt in range(POLL_ATTEMPTS):
         for frame in page.frames:
-            strategy = await _decline_in_frame(frame)
+            strategy = await _decline_in_frame(frame, deadline)
             if strategy is not None:
                 with suppress(Exception):  # settle wait is best-effort
                     await page.wait_for_timeout(SETTLE_MS)
                 return strategy
-        # Nothing left that can still render into a notice.
-        if timeout_ms <= 0 or attempt + 1 == POLL_ATTEMPTS or not _has_consent_frame(page):
+        # Nothing left that can still render into a notice, or out of budget.
+        if (
+            timeout_ms <= 0
+            or attempt + 1 == POLL_ATTEMPTS
+            or monotonic() >= deadline
+            or not _has_consent_frame(page)
+        ):
             break
         with suppress(Exception):
             await page.wait_for_timeout(POLL_INTERVAL_MS)
@@ -210,7 +233,7 @@ def _is_consent_frame(frame: Any) -> bool:
     return bool(CONSENT_FRAME_URL_PATTERN.search(getattr(frame, "url", "") or ""))
 
 
-async def _decline_in_frame(frame: Any) -> str | None:
+async def _decline_in_frame(frame: Any, deadline: float) -> str | None:
     text_scope = frame if _is_consent_frame(frame) else frame.locator(CONSENT_CONTAINER_CSS)
     candidates = (
         ("cmp", frame.locator(CMP_REJECT_CSS)),
@@ -222,11 +245,19 @@ async def _decline_in_frame(frame: Any) -> str | None:
         except Exception:  # noqa: BLE001 - detached frame or bad selector engine
             return None
         for index in range(total):
+            click_timeout = min(CLICK_TIMEOUT_MS, (deadline - monotonic()) * 1000)
+            if click_timeout <= 0:
+                return None
             element = locator.nth(index)
             try:
                 if not await element.is_visible():
                     continue
-                await element.click(timeout=CLICK_TIMEOUT_MS)
+                # Playwright dispatches real browser input events here, not a
+                # JavaScript el.click(), so this is not a synthesized-event
+                # tell. Deliberately not human_click: that takes a Page and
+                # notices live in frames, and consent widgets do not score
+                # cursor trajectory.
+                await element.click(timeout=click_timeout)
             except Exception:  # noqa: BLE001 - covered, detached, or navigating
                 continue
             return strategy