Auto-decline cookie banners during fetch

- Add consent auto-decline to fetch, CLI, and MCP flows
- Expose cookie_notice_declined in fetch results

Author: kacigaya

README.md

@@ -150,6 +150,48 @@ async def main() -> None:
 asyncio.run(main())
 ```
 
+## Auto-decline cookie banners
+
+Every `session.fetch()` (and therefore the CLI and the MCP tools) clicks the
+reject control of a cookie consent notice before reading the page, so scraped
+text is not buried under a banner and no optional cookies are accepted.
+
+Detection is inspired by Brave's
+[cookiecrumbler](https://github.com/brave/cookiecrumbler), which finds consent
+notices; WebSkrap dismisses them. Two strategies, tried in every frame:
+
+1. Reject buttons of the common consent platforms (OneTrust, Cookiebot, Didomi,
+   Usercentrics, Sourcepoint, Quantcast, Osano, Complianz, CookieYes, ...).
+2. A clickable whose label matches a reject phrase ("Reject all", "Refuser
+   tout", "Ablehnen", "Continue without accepting", ...), scoped to a
+   cookie/consent container so unrelated "Decline" buttons are never touched.
+
+`FetchResult.cookie_notice_declined` reports which strategy clicked (`"cmp"`,
+`"text"`, or `None`).
+
+```python
+config = SessionConfig(
+    decline_cookies=True,             # default
+    decline_cookies_timeout_ms=2_000,  # wait for a late-injected notice; 0 = check once
+)
+
+result = await client.fetch("https://example.com", config=config)
+print(result.cookie_notice_declined)
+```
+
+The timeout is the per-fetch cost on pages without a notice. Pages that carry a
+consent-platform iframe are retried for up to 3s while it renders. For pages you
+drive yourself, call it directly:
+
+```python
+page = await session.context.new_page()
+await page.goto("https://example.com", wait_until="domcontentloaded")
+await session.decline_cookies(page)
+```
+
+Consent walls with no reject control on the first layer (pay-or-consent) are
+left alone.
+
 ## Custom profile
 
 ```python
@@ -179,6 +221,8 @@ config = SessionConfig(
     storage_state=None,
     proxy=ProxyConfig(server="http://127.0.0.1:8080"),
     resource_policy=ResourcePolicy.LITE,
+    decline_cookies=True,
+    decline_cookies_timeout_ms=2_000,
     ignore_https_errors=True,
     java_script_enabled=True,
     service_workers="allow",
@@ -375,6 +419,8 @@ webskrap fetch https://example.com --profile desktop-chrome
 webskrap fetch https://example.com --format json --max-chars 12000
 webskrap fetch https://example.com --stdout --text-only
 webskrap fetch https://example.com --screenshot example.png
+webskrap fetch https://example.com --no-decline-cookies
+webskrap fetch https://example.com --decline-cookies-timeout-ms 4000
 webskrap fetch https://example.com --channel chrome \
   --user-data-dir .webskrap/headless-profile
 webskrap fetch https://amiunique.org/fr/fingerprint \
@@ -405,8 +451,10 @@ clean visible page text by default, with no HTML tags, scripts, or style noise,
 so agents spend tokens on content, not markup (typically 5-10x fewer tokens than
 raw HTML). Pass `text_only=False` when you actually need the HTML. Use
 `stealth_fetch` for finer control over fingerprint surface, WebRTC, UA masking,
-and persistent profiles. Every result carries `status`, `final_url`, `title`,
-`text_length`, and truncation flags so the model knows exactly what it got.
+and persistent profiles. Both auto-decline cookie banners (`decline_cookies=False`
+to keep them). Every result carries `status`, `final_url`, `title`,
+`text_length`, `cookie_notice_declined`, and truncation flags so the model knows
+exactly what it got.
 
 ```bash
 pip install webskrap

SKILL.md

@@ -16,6 +16,7 @@ Prefer current repo sources over memory:
 - `src/webskrap/client.py`: `WebSkrapClient`, sessions, fetch flow, screenshots.
 - `src/webskrap/models.py`: public Pydantic models, `SessionConfig`, result shape.
 - `src/webskrap/profiles.py`: bundled browser profiles.
+- `src/webskrap/consent.py`: cookie-banner auto-decline selectors and strategies.
 - `src/webskrap/cli.py`: current `webskrap` command behavior.
 - `src/webskrap/mcp_server.py`: MCP tools and argument shape.
 - `tests/`: behavior contracts when changing parsing, state, CLI, stealth, or safety.
@@ -76,6 +77,17 @@ config = SessionConfig(
 )
 ```
 
+Every `fetch()` auto-declines cookie consent notices before reading the page
+(`src/webskrap/consent.py`). Turn it off or retune the wait per call:
+
+```python
+config = SessionConfig(decline_cookies=False, decline_cookies_timeout_ms=2_000)
+```
+
+`FetchResult.cookie_notice_declined` reports the strategy that clicked (`"cmp"`,
+`"text"`, or `None`). `session.decline_cookies(page)` does the same on a page you
+drive yourself.
+
 Headed Patchright is strongest for strict detection surfaces:
 
 ```python
@@ -126,7 +138,7 @@ webskrap fetch https://example.com --channel chromium --format json
 
 `fetch --format json` prints bounded JSON to stdout using the MCP-compatible
 shape: `url`, `final_url`, `status`, `ok`, `title`, `headers`, `text`,
-`text_length`, `text_truncated`, and `elapsed_ms`.
+`text_length`, `text_truncated`, `elapsed_ms`, and `cookie_notice_declined`.
 
 Use `--stdout` for raw fetched content, and combine it with `--text-only` for
 readable body text.

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "webskrap"
-version = "0.6.1"
+version = "0.7.0"
 description = "A Playwright-based Python scraping framework with coherent browser profiles and session controls."
 readme = "README.md"
 requires-python = ">=3.11"

src/webskrap/__init__.py

@@ -1,4 +1,5 @@
 from webskrap.client import WebSkrapClient, WebSkrapError, WebSkrapSession
+from webskrap.consent import decline_cookies
 from webskrap.models import (
     BrowserProfile,
     FetchResult,
@@ -21,6 +22,7 @@
     "WebSkrapClient",
     "WebSkrapError",
     "WebSkrapSession",
+    "decline_cookies",
     "get_profile",
     "list_profiles",
 ]

src/webskrap/cli.py

@@ -250,6 +250,21 @@ def fetch_command(
         ResourcePolicy,
         typer.Option("--resource-policy", help="Resource routing preset."),
     ] = ResourcePolicy.ALL,
+    decline_cookies: Annotated[
+        bool,
+        typer.Option(
+            "--decline-cookies/--no-decline-cookies",
+            help="Click the reject button of a cookie consent notice after load.",
+        ),
+    ] = True,
+    decline_cookies_timeout_ms: Annotated[
+        float,
+        typer.Option(
+            "--decline-cookies-timeout-ms",
+            min=0,
+            help="How long to wait for a cookie consent notice to appear.",
+        ),
+    ] = 2_000,
     patchright_context_profile: Annotated[
         bool,
         typer.Option(
@@ -305,6 +320,8 @@ def fetch_command(
             wait_until=wait_until,
             timeout_ms=timeout_ms,
             resource_policy=resource_policy,
+            decline_cookies=decline_cookies,
+            decline_cookies_timeout_ms=decline_cookies_timeout_ms,
             patchright_context_profile=patchright_context_profile,
             reduce_fingerprint_surface=reduce_fingerprint_surface,
             mask_headless_user_agent=mask_headless_user_agent,
@@ -330,6 +347,8 @@ async def _fetch(
     wait_until: str,
     timeout_ms: float,
     resource_policy: ResourcePolicy,
+    decline_cookies: bool,
+    decline_cookies_timeout_ms: float,
     patchright_context_profile: bool,
     reduce_fingerprint_surface: bool,
     mask_headless_user_agent: bool,
@@ -345,6 +364,8 @@ async def _fetch(
         user_data_dir=user_data_dir,
         navigation_timeout_ms=timeout_ms,
         resource_policy=resource_policy,
+        decline_cookies=decline_cookies,
+        decline_cookies_timeout_ms=decline_cookies_timeout_ms,
         patchright_context_profile=patchright_context_profile,
         reduce_fingerprint_surface=reduce_fingerprint_surface,
         mask_headless_user_agent=mask_headless_user_agent,
@@ -381,6 +402,8 @@ async def _fetch(
     console.print(f"[bold]Status:[/bold] {result.status}")
     console.print(f"[bold]Final URL:[/bold] {result.final_url}")
     console.print(f"[bold]Title:[/bold] {result.title}")
+    if result.cookie_notice_declined:
+        console.print(f"[bold]Cookie notice:[/bold] declined ({result.cookie_notice_declined})")
     if result.screenshot_path:
         console.print(f"[bold]Screenshot:[/bold] {result.screenshot_path}")
     if output:

src/webskrap/client.py

@@ -12,6 +12,7 @@
 
 from playwright.async_api import Browser, BrowserContext, Page, Playwright
 
+from webskrap.consent import decline_cookies as _decline_cookies
 from webskrap.models import BrowserProfile, FetchResult, ResourcePolicy, SessionConfig, WaitUntil
 from webskrap.profiles import get_profile
 
@@ -83,6 +84,9 @@ async def fetch(
                 wait_until=wait_until,
                 timeout=timeout_ms or self.config.navigation_timeout_ms,
             )
+            declined = None
+            if self.config.decline_cookies:
+                declined = await self.decline_cookies(page)
             title = await page.title()
             text = await page.locator("body").inner_text() if text_only else await page.content()
             screenshot_path = await _maybe_screenshot(page, screenshot)
@@ -101,10 +105,23 @@ async def fetch(
                 cookies=cookies,
                 timings={"elapsed_ms": elapsed_ms},
                 screenshot_path=screenshot_path,
+                cookie_notice_declined=declined,
             )
         finally:
             await page.close()
 
+    async def decline_cookies(self, page: Page, *, timeout_ms: float | None = None) -> str | None:
+        """Click a cookie consent notice's reject control on ``page``.
+
+        Called automatically by :meth:`fetch` unless
+        ``SessionConfig.decline_cookies`` is False. Call it directly for pages
+        you drive yourself. Returns the strategy that clicked, or None.
+        """
+        self._ensure_open()
+        if timeout_ms is None:
+            timeout_ms = self.config.decline_cookies_timeout_ms
+        return await _decline_cookies(page, timeout_ms=timeout_ms)
+
     async def human_click(
         self,
         page: Page,