|
| 1 | +--- |
| 2 | +title: AI Agent Workflows with MCP |
| 3 | +description: Guide to integrating html2rss with AI agents, Claude Desktop, and Cursor using the Model Context Protocol (MCP). |
| 4 | +--- |
| 5 | + |
| 6 | +import { Code } from "@astrojs/starlight/components"; |
| 7 | + |
| 8 | +`html2rss` includes a native [Model Context Protocol (MCP)](https://modelcontextprotocol.io/) server. This allows AI assistants in tools like Cursor, Claude Desktop, and GitHub Copilot to scrape pages, capture feed configurations, and generate RSS feeds autonomously. |
| 9 | + |
| 10 | +## Why Use MCP? |
| 11 | + |
| 12 | +Without MCP, agents must invoke shell commands, manage output buffers, and parse unstructured text. With MCP: |
| 13 | + |
| 14 | +- The agent automatically discovers available tools, arguments, and return types. |
| 15 | +- Responses are structured (JSON Feed objects, configuration schemas, and RSS XML). |
| 16 | +- Agents can inspect, capture, validate, and verify feeds iteratively in a closed feedback loop. |
| 17 | + |
| 18 | +## Client Setup |
| 19 | + |
| 20 | +### 1. Version Manager Shims (mise, asdf, rbenv, chruby) |
| 21 | + |
| 22 | +Because version managers manage Ruby runtimes and gem paths through environment shims, configure your MCP client to invoke the version manager executable rather than a bare `html2rss` command. |
| 23 | + |
| 24 | +<Code |
| 25 | + code={` |
| 26 | + # Install the gem in your global environment |
| 27 | + mise exec -- gem install html2rss |
| 28 | + # Or add it to your project Gemfile |
| 29 | + mise exec -- bundle add html2rss |
| 30 | +`} |
| 31 | + lang="bash" |
| 32 | +/> |
| 33 | + |
| 34 | +### 2. Cursor Configuration |
| 35 | + |
| 36 | +Add `html2rss` to your Cursor MCP settings (`~/.cursor/mcp.json` or `.cursor/mcp.json`): |
| 37 | + |
| 38 | +<Code |
| 39 | + code={` |
| 40 | + { |
| 41 | + "mcpServers": { |
| 42 | + "html2rss": { |
| 43 | + "command": "mise", |
| 44 | + "args": ["exec", "--", "html2rss", "mcp"] |
| 45 | + } |
| 46 | + } |
| 47 | + } |
| 48 | +`} |
| 49 | + lang="json" |
| 50 | +/> |
| 51 | + |
| 52 | +_(If using `asdf`, replace `"command": "mise"` with `"asdf"` and `"args": ["exec", "html2rss", "mcp"]`.)_ |
| 53 | + |
| 54 | +### 3. Claude Desktop Configuration |
| 55 | + |
| 56 | +Add `html2rss` to your Claude Desktop configuration (`~/Library/Application Support/Claude/claude_desktop_config.json` on macOS or `%APPDATA%\\Claude\\claude_desktop_config.json` on Windows): |
| 57 | + |
| 58 | +<Code |
| 59 | + code={` |
| 60 | + { |
| 61 | + "mcpServers": { |
| 62 | + "html2rss": { |
| 63 | + "command": "mise", |
| 64 | + "args": ["exec", "--", "html2rss", "mcp"] |
| 65 | + } |
| 66 | + } |
| 67 | + } |
| 68 | +`} |
| 69 | + lang="json" |
| 70 | +/> |
| 71 | + |
| 72 | +## Autonomous Workflow Patterns |
| 73 | + |
| 74 | +### Pattern A: One-Shot Content Scraping |
| 75 | + |
| 76 | +When an agent needs articles immediately without saving a feed configuration: |
| 77 | + |
| 78 | +1. The agent calls `scrape_url` with the target URL. |
| 79 | +2. `html2rss` runs auto-source extraction (Schema.org, JSON state, semantic HTML) and returns a JSON Feed items array. |
| 80 | +3. If the page is protected or rendered with JavaScript, the agent calls `inspect_url` to diagnose the structure, then retries `scrape_url` with `strategy: "botasaurus"`. |
| 81 | + |
| 82 | +### Pattern B: Iterative Feed Config Authoring |
| 83 | + |
| 84 | +When an agent is tasked with creating a durable YAML feed configuration: |
| 85 | + |
| 86 | +1. **Inspect:** The agent calls `inspect_url` to check content type, SST node counts, and eligible scrapers. |
| 87 | +2. **Capture:** The agent runs `capture_config` to derive CSS selectors for items, title, link, and description. |
| 88 | +3. **Refine:** The agent reviews the derived selectors or asks the human user for domain-specific adjustments. |
| 89 | +4. **Validate:** The agent passes the configuration to `validate_config` to verify schema conformance. |
| 90 | +5. **Apply:** The agent tests the final configuration with `apply_config` to produce and inspect live RSS XML. |
| 91 | + |
| 92 | +## JavaScript-Rendered Sites (Botasaurus) |
| 93 | + |
| 94 | +For dynamic JavaScript single-page applications or sites protected by anti-bot measures, launch the Botasaurus scrape service: |
| 95 | + |
| 96 | +<Code |
| 97 | + code={` |
| 98 | + docker compose -f docker-compose.botasaurus.yml up -d |
| 99 | +`} |
| 100 | + lang="bash" |
| 101 | +/> |
| 102 | + |
| 103 | +Ensure `BOTASAURUS_SCRAPER_URL` is accessible (typically `http://127.0.0.1:4010`) in the environment where the MCP server runs. Agents can then pass `strategy: "botasaurus"` to `scrape_url`, `inspect_url`, and `capture_config`. |
0 commit comments