docs(ruby-gem): anchor 0.27.0 migrations and write forward

Forward-facing guides teach present MCP and capture behavior; breaking contract changes live under backward-compatibility §0.27.0. Fixes inspect_url auto staying on Faraday in MCP prompts.

Author: gildesmarais

src/content/docs/ruby-gem/guides/ai-agent-workflows.mdx

@@ -84,9 +84,9 @@ Add `html2rss` to your Claude Desktop configuration (`~/Library/Application Supp
 
 When an agent needs articles immediately without saving a feed configuration:
 
-1. Call `scrape_url` with `strategy: "auto"`. Auto already runs Faraday then Botasaurus.
-2. An empty item list is still success for articles-now. Read `payload.items`. Follow `next_step` (including `read_runtime` / `html2rss://runtime` if Botasaurus is unset).
-3. Call `inspect_url` only if the result is weak or you need recon. Do not retry `scrape_url` with explicit `faraday` after auto.
+1. Call `scrape_url` with `strategy: "auto"`. Auto runs Faraday then Botasaurus when `BOTASAURUS_SCRAPER_URL` is set on the MCP process.
+2. An empty item list is still success for articles-now. Read `payload.items`. Follow `next_step` (including `read_runtime` / `html2rss://runtime` when Botasaurus is unset).
+3. Call `inspect_url` when the result is weak or you need recon (`final_url`, `status`, `scheme_downgrade`, native RSS/Atom).
 
 ### Pattern B: Iterative Feed Config Authoring
 

src/content/docs/ruby-gem/guides/backward-compatibility.mdx

@@ -5,16 +5,20 @@ description: "Migration guide and backward-compatibility reference for html2rss
 
 import { Code } from "@astrojs/starlight/components";
 
-This page outlines recent breaking changes, purged legacy aliases, and migration steps for older feed configurations.
+This page outlines breaking changes, purged legacy aliases, and migration steps when upgrading the gem.
 
-## Current Botasaurus request schema
+## 0.27.0
 
-`request.botasaurus` matches the scrape-API OpenAPI 2.0 `ScrapeRequest` (unknown keys rejected):
+Upgrade notes for configs and integrators targeting gem **0.27.0** (current `master` after `0.26.0`):
 
-1. **`scroll`** — boolean. There is no `scroll_to_bottom` key; use `scroll: true` for infinite lists.
-2. **`window_size`** — object `{ width: 1920, height: 1080 }`, not a two-item array.
-3. **`wait_timeout_seconds`** — integer `1..20` (omit to use the scrape-API default of `15`).
-4. **`max_retries`** — integer `0..3` (omit to use the scrape-API default of `2`).
+1. **MCP result envelope** — Every MCP tool returns `{ ok, next_step, guidance, payload }` in text and `structuredContent`. Read `payload.items`, `payload.yaml`, and `payload.rss`; scrape text is not a raw item array.
+2. **MCP AutoFallback** — `scrape_url` and `capture_config` with `strategy: "auto"` run Faraday → Botasaurus in one call. `inspect_url` with `auto` stays on Faraday (pin `botasaurus` when you need browser rendering for inspect).
+3. **Capture output** — `html2rss capture` and MCP `capture_config` emit an items selector plus `enhance: true` only (no per-field title/url/description selector soup). Capture stamps `strategy:` when AutoFallback selects a concrete transport.
+4. **CLI `--explain`** — `html2rss auto --explain` and `html2rss capture --explain` print telemetry JSON on stderr; stdout stays the feed or YAML. Status includes `admission_drops` when Cleanup dropped items.
+5. **`Html2rss.auto_feed_result`** — URL-only auto discovery returns `FeedResult` with the same `status` telemetry as config builds.
+6. **`request.botasaurus` OpenAPI 2.0** — Keys must match the scrape-API `ScrapeRequest` (unknown keys rejected). Use `scroll: true` (not `scroll_to_bottom`), `window_size: { width:, height: }` (not a two-item array), and `wait_timeout_seconds` in `1..20`. See [Strategy](/ruby-gem/reference/strategy/#botasaurus).
+7. **Botasaurus wire format** — Client expects OpenAPI 2.0 `ScrapeSuccess` / `ScrapeError` envelopes from the scrape API.
+8. **`--limit` default** — Auto discovery default is `25` (`AutoSource::DEFAULT_LIMIT`).
 
 Run `html2rss validate config.yml` after editing Botasaurus options.
 
@@ -30,7 +34,7 @@ Upgrade notes for configs and integrators targeting gem **0.26.0**:
 Upgrade notes for configs and integrators targeting gem **0.25.0**:
 
 1. **Removed `link` selector alias** — Custom selector configurations must use the canonical `url` key. The legacy `link` selector alias and exclusivity checks have been removed.
-2. **`auto` article limit (`--limit N`)** — Discovery now supports a configurable article limit (`--limit N` CLI flag, default `20`) and `Html2rss.auto_source(url, limit: 20)` in Ruby.
+2. **`auto` article limit (`--limit N`)** — Discovery supports a configurable limit (`--limit N` CLI flag). Default was `20` in 0.25.0; **0.27.0** default is `25` (`Html2rss.auto_source(url, limit: 25)`).
 3. **Channel metadata in `auto`** — `auto_source` automatically extracts channel-level `author` and `image` (favicon, touch-icon, or OpenGraph images) from the host page.
 4. **Actionable `NoFeedItemsExtracted` error** — `auto` mode fails loud with diagnostic surface classifications (blocked surface, app-shell, unsupported extraction surface) when zero items are found.
 
@@ -100,13 +104,17 @@ Update any occurrences of `link`, `pubDate`, or `updated` in your `selectors` bl
 
 ## Migration Checklist
 
-When upgrading to modern `html2rss` releases:
+When upgrading to **0.27.0** or later:
 
-1. **Migrate Browserless to Botasaurus**: Update any feeds using `strategy: browserless` or `request.browserless` to `strategy: botasaurus` or `strategy: auto` with `BOTASAURUS_SCRAPER_URL`.
-2. **Rename URL selectors**: Ensure article URL selectors use `url` rather than `link`.
-3. **Rename date selectors**: Ensure date selectors use `published_at` rather than `updated` or `pubDate`.
-4. **Move channel-level transport keys**: Ensure `strategy` and `headers` are defined at the top level of the YAML file.
-5. **Use `enclosure` for RSS media**: Do not rely on `image` becoming an RSS `<enclosure>`; select podcast/media URLs with `enclosure`.
-6. **Prefer `feed_result` for dual-format / cache**: Integrators that need RSS + JSON Feed from one scrape (or Marshal caching) should use `Html2rss.feed_result` and `status.to_h`.
-7. **Align `request.botasaurus` with OpenAPI 2.0**: Replace `scroll_to_bottom` with `scroll`, use `{ width, height }` for `window_size`, and keep `wait_timeout_seconds` in `1..20`.
-8. **Validate configurations**: Run `html2rss validate config.yml` to ensure your YAML conforms to the current schema.
+1. **MCP integrators**: Read tool results from the envelope `payload`; configure `BOTASAURUS_SCRAPER_URL` on the MCP process env.
+2. **Align `request.botasaurus` with OpenAPI 2.0**: Use `scroll`, `{ width, height }` for `window_size`, and `wait_timeout_seconds` in `1..20` (see [Strategy](/ruby-gem/reference/strategy/#botasaurus)).
+3. **Validate configurations**: Run `html2rss validate config.yml`.
+
+When upgrading from releases before **0.26.0**:
+
+4. **Migrate Browserless to Botasaurus**: Use `strategy: botasaurus` or `strategy: auto` with `BOTASAURUS_SCRAPER_URL`.
+5. **Rename URL selectors**: Use `url` rather than `link`.
+6. **Rename date selectors**: Use `published_at` rather than `updated` or `pubDate`.
+7. **Move channel-level transport keys**: Put `strategy` and `headers` at the top level of the YAML file.
+8. **Use `enclosure` for RSS media**: Select podcast/media URLs with `enclosure`; `image` does not become RSS `<enclosure>`.
+9. **Prefer `feed_result` for dual-format / cache**: Use `Html2rss.feed_result` / `Html2rss.auto_feed_result` and `status.to_h`.

src/content/docs/ruby-gem/guides/capturing-feed-configs.mdx

@@ -5,7 +5,7 @@ description: Derive a reusable YAML feed config from a URL with html2rss capture
 
 import { Aside, Code } from "@astrojs/starlight/components";
 
-`html2rss capture` analyzes a page through the auto-source pipeline and prints a reusable YAML draft with an **items selector plus `enhance: true`** — not title/url/description selector soup. `enhance: true` fills missing article fields at feed-build time.
+`html2rss capture` analyzes a page through the auto-source pipeline and prints a reusable YAML draft: an **items selector plus `enhance: true`**. At feed-build time, `enhance: true` fills missing article fields from each matched card.
 
 <Aside type="tip" title="Treat capture as a draft">
   Selector quality depends on page structure. Catalog feeds still need `directory.topics` and explicit channel

src/content/docs/ruby-gem/reference/cli-reference.mdx

@@ -180,7 +180,7 @@ Available options:
 - `--transport`: MCP transport protocol (`stdio` or `http`). Defaults to `stdio`.
 - `--port`: Port for HTTP transport (default: `8080`). Bound strictly to `127.0.0.1` (loopback only).
 
-HTTP transport requires the `rack`, `rackup`, and `webrick` gems. stdio keeps JSON-RPC on stdout; the daemon logs to stderr and defaults to `LOG_LEVEL=info` unless you set `LOG_LEVEL`. See the [MCP Server Reference](/ruby-gem/reference/mcp-server/) and [AI Agent Workflows Guide](/ruby-gem/guides/ai-agent-workflows/) for full configuration and tool schemas.
+HTTP transport requires the `rack`, `rackup`, and `webrick` gems. MCP daemon logging (stderr, `LOG_LEVEL`) is documented in the [MCP Server Reference](/ruby-gem/reference/mcp-server/). See also the [AI Agent Workflows Guide](/ruby-gem/guides/ai-agent-workflows/) for client setup and tool schemas.
 
 ### Schema
 

src/content/docs/ruby-gem/reference/mcp-server.mdx

@@ -38,9 +38,9 @@ The MCP server supports two transport mechanisms:
 
 ## Strategy Resolution in MCP
 
-`scrape_url` and `capture_config` with `strategy: "auto"` (the default) run Faraday → Botasaurus AutoFallback. One call is enough; do not retry with explicit `faraday` after `auto`.
+`scrape_url` and `capture_config` with `strategy: "auto"` (the default) run Faraday → Botasaurus AutoFallback in one call.
 
-`inspect_url` is the exception: `auto` stays on Faraday for a cheap diagnostic. Pin `strategy: "botasaurus"` on inspect when you need browser rendering.
+`inspect_url` with `auto` uses Faraday for a cheap diagnostic. Pin `strategy: "botasaurus"` when you need browser rendering for inspect.
 
 Read `html2rss://runtime` for `botasaurus_configured` (boolean only). Set `BOTASAURUS_SCRAPER_URL` on the **MCP process** env (see [AI agent workflows](/ruby-gem/guides/ai-agent-workflows/)).
 
@@ -62,7 +62,7 @@ Every tool returns the same JSON object in both the text content and `structured
   lang="json"
 />
 
-`isError` matches `ok: false`. Hosts that parsed scrape text as a raw JSON array must read `payload.items` instead.
+`isError` matches `ok: false`. Scrape results expose articles in `payload.items`.
 
 `next_step` is one of: `done`, `inspect_url`, `validate_config`, `apply_config`, `scrape_url`, `capture_config`, `read_runtime`.
 
@@ -161,7 +161,7 @@ One `scrape_url` call with `strategy: auto` (AutoFallback already hops). Call `i
 Four-step workflow for a durable feed config:
 
 1. Call `capture_config` — YAML is `payload.yaml`. Check `payload.articles_count` and `has_selectors`. Strive `enhance: true`.
-2. If weak or you need recon, call `inspect_url`. Auto already hops to Botasaurus.
+2. If weak or you need recon, call `inspect_url` (`auto` uses Faraday; pin `botasaurus` for browser rendering).
 3. Call `validate_config` with `yaml` (or `config`) — must not be `isError`.
 4. Call `apply_config` — `isError` if zero items. Confirm `payload.item_count` before shipping.
 

src/content/docs/troubleshooting/troubleshooting.mdx

@@ -101,7 +101,7 @@ Common configuration-related errors:
 - **`BOTASAURUS_SCRAPER_URL is invalid`:** Fix the URL format and retry.
 - **`Configuration must include at least 'selectors' or 'auto_source'`:** You need to specify either manual selectors or enable auto-source.
 - **`stylesheet.type invalid`:** Only `text/css` and `text/xsl` are supported for stylesheets.
-- **`request.botasaurus` validation:** Unknown keys fail (`scroll_to_bottom` is not valid — use `scroll: true`). `window_size` must be `{ width:, height: }`. `wait_timeout_seconds` must be `1..20`.
+- **`request.botasaurus` validation:** Unknown keys are rejected. Valid scroll, window size, and wait-timeout shapes are documented in [Strategy — Botasaurus](/ruby-gem/reference/strategy/#botasaurus).
 
 ### Missing Item Parts