docs: improve search engine visibility

Author: kacigaya

web/content/api-reference.md

@@ -1,3 +1,8 @@
+---
+title: WebSkrap Python API Reference
+description: Reference for WebSkrapClient, WebSkrapSession, FetchResult, SessionConfig, BrowserProfile, resource policies, and proxy configuration.
+---
+
 # API Reference
 
 > Full typed signatures and docstrings live in the source. Browse them on [GitHub](https://github.com/kacigaya/webskrap/tree/main/src/webskrap).

web/content/comparisons/webskrap-vs-playwright.md

@@ -0,0 +1,58 @@
+---
+title: WebSkrap vs Playwright for Python Web Scraping
+description: Compare WebSkrap and Playwright for Python web scraping, browser automation, persistent sessions, resource routing, Patchright stealth, and MCP agent tools.
+---
+
+# WebSkrap vs Playwright for Python Web Scraping
+
+Playwright is a powerful browser automation library. WebSkrap builds on top of Playwright and Patchright to provide a higher-level Python web scraping toolkit.
+
+## When Playwright is enough
+
+Use Playwright directly when you need full low-level browser control and want to write the whole scraping workflow yourself.
+
+```python
+from playwright.async_api import async_playwright
+```
+
+This is ideal for custom browser automation, testing flows, or highly specific scraping logic.
+
+## What WebSkrap adds
+
+WebSkrap packages common scraping needs into a smaller API:
+
+| Capability | Playwright | WebSkrap |
+| --- | --- | --- |
+| Browser automation | Yes | Yes, via Playwright/Patchright |
+| One-shot fetch result object | Manual | Built in |
+| Persistent scraping sessions | Manual setup | Built in |
+| Browser profiles | Manual setup | Built in |
+| Resource routing presets | Manual setup | Built in |
+| Patchright stealth path | Separate integration | Built in |
+| MCP server for agents | Manual | Built in |
+| CLI JSON fetch output | Manual | Built in |
+
+## Example WebSkrap fetch
+
+```python
+import asyncio
+from webskrap import WebSkrapClient
+
+async def main() -> None:
+    async with WebSkrapClient() as client:
+        result = await client.fetch("https://example.com")
+        print(result.title)
+
+asyncio.run(main())
+```
+
+## Recommendation
+
+Use Playwright directly when you need total browser control. Use WebSkrap when you want a Python web scraping package with sensible defaults, reusable sessions, profiles, resource policy, CLI output, and MCP tools.
+
+## Related docs
+
+- [Python Web Scraping with WebSkrap](/docs/guides/python-web-scraping)
+- [Client API](/docs/user-guide/client)
+- [Stealth](/docs/user-guide/stealth)
+- [MCP Server](/docs/user-guide/mcp)

web/content/development.md

@@ -1,3 +1,8 @@
+---
+title: Developing WebSkrap
+description: Development notes for contributing to WebSkrap, running tests, browser integrations, and maintaining the Python scraping package.
+---
+
 # Development
 
 Clone the repository:

web/content/getting-started/installation.md

@@ -1,3 +1,8 @@
+---
+title: Install WebSkrap for Python Web Scraping
+description: Install WebSkrap, Playwright browsers, Patchright support, and the CLI for Python web scraping and browser automation.
+---
+
 # Installation
 
 Install WebSkrap from PyPI:

web/content/getting-started/quickstart.md

@@ -1,3 +1,8 @@
+---
+title: Python Web Scraping Quickstart
+description: Start scraping web pages with WebSkrap, Playwright, async Python, screenshots, persistent browser state, and clean fetch results.
+---
+
 # Quickstart
 
 ## One-shot fetch

web/content/guides/llm-web-scraping.md

@@ -0,0 +1,41 @@
+---
+title: LLM Web Scraping and MCP Server for AI Agents
+description: Use WebSkrap as an MCP web scraping server for LLM agents that need clean page text from live browser sessions instead of raw HTML.
+---
+
+# LLM Web Scraping and MCP Server for AI Agents
+
+WebSkrap includes an MCP server so AI agents can fetch live web pages with a browser-backed scraper and receive clean page text instead of raw HTML.
+
+This is useful for coding agents, research agents, and automation workflows that need current page content but should avoid flooding the language model with scripts, styles, and markup noise.
+
+## Start the MCP server
+
+```bash
+pip install webskrap
+webskrap install
+webskrap-mcp
+```
+
+The MCP server exposes tools such as `fetch`, `stealth_fetch`, and `doctor`.
+
+## CLI output for agents
+
+The CLI also returns bounded JSON that is easy for agents to parse:
+
+```bash
+webskrap fetch https://example.com --format json --max-chars 12000
+```
+
+The JSON includes `url`, `final_url`, `status`, `ok`, `title`, `headers`, `text`, `text_length`, `text_truncated`, and `elapsed_ms`.
+
+## Why clean text matters
+
+Raw HTML can be thousands of tokens of boilerplate. WebSkrap focuses on browser-backed fetching with output formats that are easier for LLMs and agents to consume.
+
+## Related docs
+
+- [MCP Server](/docs/user-guide/mcp)
+- [CLI](/docs/user-guide/cli)
+- [Stealth](/docs/user-guide/stealth)
+- [API Reference](/docs/api-reference)

web/content/guides/python-web-scraping.md

@@ -0,0 +1,60 @@
+---
+title: Python Web Scraping with WebSkrap
+description: Learn Python web scraping with WebSkrap, Playwright, browser profiles, persistent sessions, and clean text extraction for modern JavaScript pages.
+---
+
+# Python Web Scraping with WebSkrap
+
+WebSkrap is a Python web scraping toolkit for pages that need a real browser. It wraps Playwright with an async API, reusable browser sessions, coherent browser profiles, resource routing, and optional Patchright-powered stealth sessions.
+
+Use it when a simple `requests` or `BeautifulSoup` script cannot see the content because the site renders data with JavaScript, relies on browser storage, or needs a realistic browser context.
+
+## Install
+
+```bash
+pip install webskrap
+webskrap install
+```
+
+## Scrape a page
+
+```python
+import asyncio
+
+from webskrap import WebSkrapClient
+
+
+async def main() -> None:
+    async with WebSkrapClient() as client:
+        result = await client.fetch("https://example.com")
+
+    print(result.status)
+    print(result.final_url)
+    print(result.title)
+    print(result.text[:500])
+
+
+asyncio.run(main())
+```
+
+`FetchResult` includes the final URL, status, headers, cookies, title, HTML/text output, timings, and optional screenshot path.
+
+## Why use a browser scraper?
+
+Modern websites often ship a small HTML shell and fill it with JavaScript. WebSkrap launches Chromium through Playwright so your scraper sees the page after browser navigation, redirects, cookies, and client-side rendering.
+
+Common use cases:
+
+- Scrape JavaScript-heavy pages in Python.
+- Build data collection scripts that need browser state.
+- Reuse login or consent state with persistent sessions on sites you are allowed to access.
+- Return clean page data to LLM agents through MCP.
+- Capture screenshots for debugging.
+
+## Next steps
+
+- [Quickstart](/docs/getting-started/quickstart)
+- [Client API](/docs/user-guide/client)
+- [Persistent sessions](/docs/user-guide/sessions)
+- [Browser profiles](/docs/user-guide/profiles)
+- [Resource policy](/docs/user-guide/resource-policy)

web/content/guides/scrape-javascript-pages.md

@@ -0,0 +1,68 @@
+---
+title: Scrape JavaScript Pages with Playwright and Python
+description: Use WebSkrap to scrape JavaScript-rendered pages in Python with Playwright, Chromium, wait conditions, screenshots, and resource routing.
+---
+
+# Scrape JavaScript Pages with Playwright and Python
+
+Many web pages do not expose their main content in the initial HTML. WebSkrap uses Playwright and Chromium so Python scripts can scrape JavaScript-rendered pages after the browser has loaded them.
+
+## Browser-backed fetch
+
+```python
+import asyncio
+
+from webskrap import WebSkrapClient
+
+
+async def main() -> None:
+    async with WebSkrapClient() as client:
+        result = await client.fetch(
+            "https://example.com",
+            wait_until="domcontentloaded",
+            timeout_ms=30_000,
+            screenshot="debug.png",
+        )
+
+    print(result.ok)
+    print(result.title)
+    print(result.screenshot_path)
+
+
+asyncio.run(main())
+```
+
+## Faster scraping with resource routing
+
+Images, fonts, media, and trackers often slow scraping down. WebSkrap supports resource policies so you can load only what the page needs.
+
+```python
+from webskrap import ResourcePolicy, SessionConfig
+
+config = SessionConfig(resource_policy=ResourcePolicy.LITE)
+```
+
+Use `DOCUMENTS` when you only need top-level document HTML. Use `LITE` when pages need scripts or styles but not heavy media.
+
+## Debug JavaScript pages
+
+For tricky pages, keep a headed browser open:
+
+```python
+from pathlib import Path
+from webskrap import SessionConfig
+
+config = SessionConfig(
+    headless=False,
+    user_data_dir=Path(".webskrap/debug-profile"),
+)
+```
+
+Then inspect what the browser actually sees before changing the scraper.
+
+## Related docs
+
+- [Quickstart](/docs/getting-started/quickstart)
+- [Resource policy](/docs/user-guide/resource-policy)
+- [Sessions](/docs/user-guide/sessions)
+- [CLI](/docs/user-guide/cli)

web/content/guides/web-crawling-with-python.md

@@ -0,0 +1,62 @@
+---
+title: Web Crawling with Python and WebSkrap
+description: Build browser-aware Python web crawlers with WebSkrap, async concurrency, persistent sessions, resource policies, and safe crawling boundaries.
+---
+
+# Web Crawling with Python and WebSkrap
+
+WebSkrap is useful for browser-aware web crawling when each URL may need JavaScript rendering, cookies, redirects, or a realistic browser profile.
+
+A crawler should still be polite and bounded. Respect robots policies where applicable, rate limits, terms of service, and access controls. WebSkrap does not include CAPTCHA solving or access-control bypassing.
+
+## Simple async crawl pattern
+
+```python
+import asyncio
+
+from webskrap import WebSkrapClient
+
+URLS = [
+    "https://example.com",
+    "https://example.com/about",
+]
+
+
+async def main() -> None:
+    async with WebSkrapClient() as client:
+        results = await asyncio.gather(*(client.fetch(url) for url in URLS))
+
+    for result in results:
+        print(result.status, result.final_url, result.title)
+
+
+asyncio.run(main())
+```
+
+## Crawl with sessions
+
+Use sessions when multiple pages belong to the same site and should share browser state.
+
+```python
+from pathlib import Path
+from webskrap import SessionConfig, WebSkrapClient
+
+config = SessionConfig(user_data_dir=Path(".webskrap/crawl-profile"))
+```
+
+Persistent sessions can keep cookies and local storage between fetches for sites you are authorized to access.
+
+## Practical crawler tips
+
+- Keep concurrency low until you understand a site.
+- Use resource policies to avoid downloading heavy assets.
+- Store `final_url` to detect redirects and canonical pages.
+- Capture screenshots only for debugging because they add cost.
+- Log status codes, titles, timings, and failures.
+
+## Related docs
+
+- [Client API](/docs/user-guide/client)
+- [Sessions](/docs/user-guide/sessions)
+- [Profiles](/docs/user-guide/profiles)
+- [Resource policy](/docs/user-guide/resource-policy)

web/content/index.md

@@ -1,3 +1,8 @@
+---
+title: WebSkrap Documentation
+description: Documentation for WebSkrap, a Python web scraping and browser automation toolkit built on Playwright, Patchright, persistent sessions, resource routing, and MCP.
+---
+
 # WebSkrap
 
 Async-first Python scraping on Playwright that also works as a web tool for LLMs and agents. You get coherent browser profiles, persistent sessions, resource routing, and Patchright-powered stealth. The MCP server runs that same stealth path, so agents get live pages back as plain text instead of raw HTML.