fix(web-crawler): remove Playwright rendering andoptimize the robots.txt entry-failure message (#3040)

* refactor(web-crawler): remove Playwright rendering, keep static SPA shells

Drop the Playwright-based fallback rendering path so SPA pages are stored
as their static shell HTML instead of being rendered headlessly. SPA
shells now surface the <noscript> notice (e.g. "You need to enable
JavaScript to run this app.") rather than producing an empty document,
and robots.txt-blocked entry pages get a human-readable error message.

- Delete playwright_renderer.py, render_heuristics.py and their tests
- Strip fallback_playwright/playwright_timeout config, fallback_rendered
  counter, and render-hint plumbing from the crawler and web importer
- HTMLParser: drop the SPA-empty-pattern stripping and fall back to
  <noscript> text when trafilatura extracts nothing
- Humanize the robots.txt entry-failure message
- Migrate the orphaned _convert_to_raw_url tests to HTTPAccessor (the
  method moved there in an earlier reorg) and remove the stale file

* refactor(web-importer): simplify robots.txt-blocked import message

Replace the verbose robots.txt explanation with a short compliance hint
that omits the URL and points users to local-file import instead.

Author: sponge225

docs/en/api/02-resources.md

@@ -53,7 +53,7 @@ OpenViking supports various resource types, categorized by functionality:
 
 | Type | Resource Name | Description |
 |------|---------------|-------------|
-| Single page / recursive crawl | `https://host/path` | By default only the entry page is fetched. Set `args.depth > 0` to crawl same-host links breadth-first; `args.max_pages` only bounds how many pages are collected. Each page is extracted to Markdown with trafilatura, with an automatic Playwright fallback for SPAs whose static HTML carries no body text. Supported `args`: `depth`, `max_pages`, `include_paths`, `exclude_paths`, `allow_external_links`, `skip_download_links`. Download links discovered on pages are skipped by default (`skip_download_links=true`) to avoid importing sidecar files such as `llms.txt`; set it to `false` to download same-host file links and count them toward `max_pages`. `include_paths`/`exclude_paths` use **path-prefix** matching (e.g. `/docs/` matches only paths starting with `/docs/`, never substrings like `/blog/docs-tips`). |
+| Single page / recursive crawl | `https://host/path` | By default only the entry page is fetched. Set `args.depth > 0` to crawl same-host links breadth-first; `args.max_pages` only bounds how many pages are collected. Each page is extracted to Markdown with trafilatura. Supported `args`: `depth`, `max_pages`, `include_paths`, `exclude_paths`, `allow_external_links`, `skip_download_links`. Download links discovered on pages are skipped by default (`skip_download_links=true`) to avoid importing sidecar files such as `llms.txt`; set it to `false` to download same-host file links and count them toward `max_pages`. `include_paths`/`exclude_paths` use **path-prefix** matching (e.g. `/docs/` matches only paths starting with `/docs/`, never substrings like `/blog/docs-tips`). |
 
 > Routing: sitemap-looking URLs (`https://host/sitemap.xml`, `https://host/feed.xml`, `*.atom`, ...) and explicit `args.site=true` are delegated to the whole-website ingestion below; Git hosting URLs such as `https://github.com/{org}/{repo}` are delegated to the Code section above.
 

docs/zh/api/02-resources.md

@@ -47,7 +47,7 @@ OpenViking 支持多种资源类型，按照功能分类如下：
 网页类（递归网页爬虫）
 | 类型 | 资源名 | 说明 |
 |------|--------|------|
-| 单页 / 递归抓取 | `https://host/path` | 默认仅抓入口页；设置 `args.depth > 0` 后，沿同域链接 BFS 递归展开，`args.max_pages` 只限制最多收集的页面数。每页用 trafilatura 抽成 Markdown，对 SPA 等静态 HTML 拿不到正文的站点自动降级到 Playwright 渲染。可选 `args`：`depth`、`max_pages`、`include_paths`、`exclude_paths`、`allow_external_links`、`skip_download_links`。页面中发现的下载链接默认跳过（`skip_download_links=true`），避免导入 `llms.txt` 等 sidecar 文件造成重复；设为 `false` 时会下载同域文件链接，并计入 `max_pages`。`include_paths`/`exclude_paths` 按**路径前缀**匹配（例如 `/docs/` 仅匹配以 `/docs/` 开头的路径，不会误命中 `/blog/docs-tips`）。|
+| 单页 / 递归抓取 | `https://host/path` | 默认仅抓入口页；设置 `args.depth > 0` 后，沿同域链接 BFS 递归展开，`args.max_pages` 只限制最多收集的页面数。每页用 trafilatura 抽成 Markdown。可选 `args`：`depth`、`max_pages`、`include_paths`、`exclude_paths`、`allow_external_links`、`skip_download_links`。页面中发现的下载链接默认跳过（`skip_download_links=true`），避免导入 `llms.txt` 等 sidecar 文件造成重复；设为 `false` 时会下载同域文件链接，并计入 `max_pages`。`include_paths`/`exclude_paths` 按**路径前缀**匹配（例如 `/docs/` 仅匹配以 `/docs/` 开头的路径，不会误命中 `/blog/docs-tips`）。|
 
 > 路由说明：`https://host/sitemap.xml`、`https://host/feed.xml`、`*.atom` 等 sitemap-looking URL 和显式 `args.site=true` 让出给下表的整站导入；`https://github.com/{org}/{repo}` 等 Git 托管平台 URL 让出给上文的代码导入。
 

openviking/parse/accessors/web_crawler/config.py

@@ -22,8 +22,6 @@ class CrawlConfig:
     retry_times: int = 2
     max_links_per_page: int = 500
     max_html_bytes: int = 10 * 1024 * 1024
-    fallback_playwright: bool = True
-    playwright_timeout: float = 30.0
     request_validator: Optional[Callable[[str], None]] = None
 
     def __post_init__(self) -> None:
@@ -43,5 +41,3 @@ def __post_init__(self) -> None:
             raise ValueError("max_links_per_page must be >= 1.")
         if self.max_html_bytes < 1:
             raise ValueError("max_html_bytes must be >= 1.")
-        if self.playwright_timeout <= 0:
-            raise ValueError("playwright_timeout must be > 0 for recursive web crawling.")

openviking/parse/accessors/web_crawler/models.py

@@ -31,4 +31,3 @@ class CrawlResult:
     total_downloads: int = 0
     total_skipped: int = 0
     total_failed: int = 0
-    fallback_rendered: int = 0