docs(adapters): document CLI --explain and Botasaurus OpenAPI 2.0 options

auto/capture --explain print telemetry on stderr. MCP inspect payload includes admission and XHR recon. request.botasaurus matches the scrape-API contract (no extra keys, window_size object).

Author: gildesmarais

src/content/docs/ruby-gem/reference/cli-reference.mdx

@@ -26,6 +26,7 @@ Automatically discovers items from a page and prints the generated RSS or JSONFe
   html2rss auto https://example.com/articles --limit 10 ; \\
   BOTASAURUS_SCRAPER_URL="http://localhost:4010" html2rss auto https://example.com/protected --strategy botasaurus ; \\
   html2rss auto https://example.com/articles --items_selector ".post-card" ; \\
+  html2rss auto https://example.com/articles --explain ; \\
   html2rss auto --input sample.html ; \\
   html2rss auto https://example.com/articles --input sample.html
 `}
@@ -38,11 +39,12 @@ Available options:
 
 - `--strategy`: Optional request strategy (`auto`, `faraday`, `botasaurus`, `local_file`). Defaults to `auto`, which tries `faraday` -> `botasaurus`.
 - `--format`: Output format for the auto-sourced feed (`rss` or `jsonfeed`). Defaults to `rss`.
-- `--limit`: Maximum number of articles to extract during discovery (defaults to `20`).
+- `--limit`: Maximum number of articles to extract during discovery (defaults to `25`).
 - `--items_selector`: Optional CSS selector hint for item extraction.
 - `--max-redirects`: Maximum redirects to follow per request.
 - `--max-requests`: Maximum requests to allow for this feed build (defaults to `4` for auto discovery).
 - `--input`: Local HTML file path to read input from without making network requests. When provided without a `URL`, `html2rss` attempts to detect the base URL from `<link rel="canonical">` or OpenGraph metadata in the HTML.
+- `--explain`: Print `Html2rss::Status` JSON to stderr (`version`, `dedup_dropped`, and when present `scraper_tallies`, `selected_strategy`, `attempt_count`, `strategy_attempts`, `admission_drops`). Stdout stays the RSS or JSON Feed.
 
 #### URL Surface Guidance For `auto`
 
@@ -131,13 +133,14 @@ The CLI keeps `strategy` as a top-level override and writes runtime request limi
 
 ### Capture
 
-Analyzes a URL using auto-source structural analysis and prints a reusable YAML feed configuration with derived CSS selectors to stdout.
+Analyzes a URL using auto-source structural analysis and prints a reusable YAML feed configuration (`items` selector + `enhance: true`) to stdout.
 
 <Code
   code={`
   html2rss capture https://example.com/articles ; \\
   html2rss capture https://example.com/articles --strategy botasaurus ; \\
   html2rss capture https://example.com/articles --items_selector ".post-card" ; \\
+  html2rss capture https://example.com/articles --explain ; \\
   html2rss capture --input sample.html
 `}
   lang="bash"
@@ -153,6 +156,9 @@ Available options:
 - `--max-redirects`: Maximum redirects to follow per request.
 - `--max-requests`: Maximum requests to allow for this feed build.
 - `--input`: Local HTML file path to read input from without making network requests.
+- `--explain`: Print capture quality JSON to stderr (`articles_count`, `channel_title`, `has_selectors`, `segment_strategy`, `selected_strategy`, `admission_drops`). Stdout stays YAML.
+
+When AutoFallback (or a pinned strategy) selects a concrete transport, the printed YAML includes a top-level `strategy:` so later `html2rss feed` uses the same hop.
 
 ### MCP
 
@@ -174,7 +180,7 @@ Available options:
 - `--transport`: MCP transport protocol (`stdio` or `http`). Defaults to `stdio`.
 - `--port`: Port for HTTP transport (default: `8080`). Bound strictly to `127.0.0.1` (loopback only).
 
-HTTP transport requires the `rack`, `rackup`, and `webrick` gems. See the [MCP Server Reference](/ruby-gem/reference/mcp-server/) and [AI Agent Workflows Guide](/ruby-gem/guides/ai-agent-workflows/) for full configuration and tool schemas.
+HTTP transport requires the `rack`, `rackup`, and `webrick` gems. stdio keeps JSON-RPC on stdout; the daemon logs to stderr and defaults to `LOG_LEVEL=info` unless you set `LOG_LEVEL`. See the [MCP Server Reference](/ruby-gem/reference/mcp-server/) and [AI Agent Workflows Guide](/ruby-gem/guides/ai-agent-workflows/) for full configuration and tool schemas.
 
 ### Schema
 

src/content/docs/ruby-gem/reference/mcp-server.mdx

@@ -34,6 +34,7 @@ The MCP server supports two transport mechanisms:
 
 - **Loopback binding:** The HTTP transport only binds to `127.0.0.1`. Never expose it to public interfaces without authentication and reverse proxy origin controls.
 - **Dependencies:** The HTTP transport requires the `rack`, `rackup`, and `webrick` gems. If they are not present, starting the server raises a descriptive `LoadError`.
+- **Logging:** stdio JSON-RPC uses stdout. The daemon logs to stderr and defaults to `LOG_LEVEL=info` unless `LOG_LEVEL` is set (`debug` for more, `warn` to quiet).
 
 ## Strategy Resolution in MCP
 
@@ -47,14 +48,24 @@ Read `html2rss://runtime` for `botasaurus_configured` (boolean only). Set `BOTAS
 
 ## Result envelope
 
-Every tool returns the same JSON object in both the text content and `structuredContent` (there is no `_meta` quality channel):
+Every tool returns the same JSON object in both the text content and `structuredContent`:
 
-```json
-{ "ok": true, "next_step": "validate_config", "guidance": "...", "payload": {} }
-```
+<Code
+  code={`
+  {
+    "ok": true,
+    "next_step": "validate_config",
+    "guidance": "...",
+    "payload": {}
+  }
+`}
+  lang="json"
+/>
 
 `isError` matches `ok: false`. Hosts that parsed scrape text as a raw JSON array must read `payload.items` instead.
 
+`next_step` is one of: `done`, `inspect_url`, `validate_config`, `apply_config`, `scrape_url`, `capture_config`, `read_runtime`.
+
 ## Tools Reference
 
 The server registers 5 primary tools for AI agent orchestration.
@@ -84,6 +95,9 @@ Diagnostic page analysis plus recon from the same fetch. `auto` uses Faraday; pi
   - `scheme_downgrade`: `true` when the fetch went https → http.
   - `alternate_feeds`: `rel=alternate` RSS/Atom links from the document head (no `/feed` path guessing).
   - `content_type` / `html_response` / `scraper_eligibility` / `sst_stats` / `sst.segment_stats`.
+  - `articles_count` and `admission_drops` when the response is HTML (Cleanup drop-reason counts).
+  - `blocked_surface` when an interstitial signature is detected.
+  - `xhr_capture` when the inspect strategy is `botasaurus` (`count`, redacted `sample_endpoints`, `candidate_articles`).
 - **next_step:** `done` when native alternate feeds exist, `capture_config` when articles were found, otherwise `scrape_url`.
 
 ### `capture_config`
@@ -129,6 +143,8 @@ The MCP server exposes reference data under the `html2rss://` URI scheme:
 | **`html2rss://strategies`** | `application/json` | Published MCP request strategies (`auto`, `faraday`, `botasaurus`). Does not list `local_file`.                          |
 | **`html2rss://runtime`**    | `application/json` | Boolean `botasaurus_configured` from process env. Never includes the scraper URL.                                        |
 
+`validate_config` / `apply_config` reject `strategy: local_file` and `request.local_file_path`. Use CLI `--input` for fixtures.
+
 ## Prompts Reference
 
 The server publishes guided prompt workflows for AI assistants:

src/content/docs/ruby-gem/reference/strategy.mdx

@@ -30,6 +30,8 @@ Under `auto`, `Html2rss.feed_result(...).status` exposes scrape telemetry: `sele
 
 `faraday` is the standard static HTTP transport. When no custom `User-Agent` header is supplied in `headers`, `html2rss` sends `html2rss/<VERSION>` by default.
 
+HTML is accepted when `Content-Type` includes `text/html`, or when the body sniffs as HTML (`<!DOCTYPE html` / `<html`) and the response is not JSON. Gzip, deflate, and brotli bodies are decoded even when `Content-Encoding` is missing; unlabeled brotli is tried for `octet-stream`.
+
 ## `botasaurus`
 
 `botasaurus` delegates page fetching to a Botasaurus scrape API endpoint. It runs as the second tier of `auto`, or when you pin `strategy: botasaurus`.
@@ -43,6 +45,8 @@ html2rss enforces local request policy preflight and timeout budget. When a tota
 
 During the **browser** execution tier, the Botasaurus scrape API always captures JSON XHR/fetch response bodies (`xhr_responses`). html2rss forwards them as `Response#captured_responses` so AutoSource `xhr_articles` can extract articles without extra requests. The HTTP-request tier returns an empty capture list. Caps and filtering live in the scrape API (see its README); article-likeness filtering is a client concern.
 
+The client speaks OpenAPI 2.0 `ScrapeSuccess` / `ScrapeError` envelopes. Success requires an `html` string. Extra `request.botasaurus` keys fail YAML validation.
+
 ### Configuration
 
 <Code
@@ -54,29 +58,28 @@ During the **browser** execution tier, the Botasaurus scrape API always captures
   request:
     botasaurus:
       navigation_mode: auto
-      max_retries: 1
+      max_retries: 2
       headless: false
   `}
   lang="yml"
 />
 
-Supported `request.botasaurus` options:
+Supported `request.botasaurus` options (unknown keys are rejected):
 
-- `execution_mode` (`auto`, `request`, `browser`; default `auto`)
+- `execution_mode` (`auto`, `request`, `browser`; scrape-API default `auto` when omitted)
 - `navigation_mode` (`auto`, `get`, `google_get`, `google_get_bypass`, `organic_get`; default `auto`)
-- `max_retries` (`0..3`; default `1`)
+- `max_retries` (`0..3`; omit to use the scrape-API default of `2`)
 - `wait_for_selector` (string)
-- `wait_timeout_seconds` (integer)
+- `wait_timeout_seconds` (integer `1..20`; omit to use the scrape-API default of `15`)
 - `scroll` (boolean)
-- `scroll_to_bottom` (boolean)
 - `block_images` (boolean)
 - `block_images_and_css` (boolean)
 - `block_trackers` (boolean)
 - `wait_for_complete_page_load` (boolean)
 - `headless` (boolean, default `false`)
 - `proxy` (string)
 - `user_agent` (string)
-- `window_size` (two-item integer array, for example `[1920, 1080]`)
+- `window_size` (`{ width: 1920, height: 1080 }` — both positive integers, required together)
 - `lang` (string, for example `en-US`)
 - `headers` (hash, forwarded to the Botasaurus scrape API alongside config-level `headers`)
 - `cookies` (hash)
@@ -90,7 +93,7 @@ Example scrape-API payload shape:
   {
     "url": "https://example.com",
     "navigation_mode": "auto",
-    "max_retries": 1,
+    "max_retries": 2,
     "headless": false
   }
   `}
@@ -103,11 +106,10 @@ Example scrape-API payload shape:
   code={`
   BOTASAURUS_SCRAPER_URL="http://localhost:4010" \\
   html2rss auto https://example.com/updates --strategy botasaurus
-
-BOTASAURUS_SCRAPER_URL="http://localhost:4010" \\
-html2rss feed my_config.yml --strategy botasaurus
+  BOTASAURUS_SCRAPER_URL="http://localhost:4010" \\
+  html2rss feed my_config.yml --strategy botasaurus
 `}
-lang="bash"
+  lang="bash"
 />
 
 ## `local_file`