Skip to content

webfetch.md

Latest commit

 

History

History
281 lines (214 loc) · 10.1 KB

webfetch.md

File metadata and controls

281 lines (214 loc) · 10.1 KB
Raw

Webfetch (smartfetch)

The webfetch tool fetches remote URLs and returns their content with intelligent extraction designed for documentation, static pages, and structured text. It provides caching, llms.txt probing, binary content handling, and optional secondary-model summarization.

webfetch is already a built-in tool in OpenCode. This plugin replaces it with an enhanced version — the implementation lives in src/tools/smartfetch/, and the tool is registered under the same webfetch name to override the default.

Parameters

Parameter Type Default Description
url URL (string) required The URL to fetch. Must be a valid HTTP/HTTPS URL.
format "text" | "markdown" | "html" "markdown" Output format for the fetched content.
timeout number 30 Timeout in seconds (max 120).
prompt string optional An extraction task for the secondary model to run against the fetched content (see Secondary Model).
extract_main boolean true Extract main content from HTML using Mozilla Readability. When disabled, returns the full page body.
prefer_llms_txt "auto" | "always" | "never" "auto" Prefer /llms.txt or /llms-full.txt over the page itself. "auto" probes only for docs-like domains (readthedocs, gitbook, netlify, vercel, etc.).
include_metadata boolean true Include YAML frontmatter with fetch metadata (status code, content type, charset, redirect chain, cache info, etc.).
save_binary boolean false Save binary payloads (images, PDFs, audio, video) to disk under the system temp dir. When disabled, binary content reports metadata-only.

Output

Text content (HTML, plain text, llms.txt)

Returns the fetched content in the requested format. When include_metadata is enabled (default), the response is prefixed with YAML frontmatter containing metadata about the fetch:

---
requested_url: "https://example.com/docs"
final_url: "https://example.com/docs"
canonical_url: "https://example.com/docs"
status_code: 200
source_content_type: "text/html"
source_kind: "html"
title: "Documentation"
headings:
  - "Getting Started"
  - "API Reference"
used_llms_txt: false
extracted_main: true
redirect_chain: []
upgraded_to_https: true
cache_hit: false
word_count: 1420
quality_signals: []
truncated: false
---

The quality_signals field flags potential issues:

  • very_short_content — fewer than 60 words
  • possible_paywall — content matches paywall/login keywords
  • high_boilerplate_ratio — large HTML-to-text ratio without Readability extraction

Binary content

Binary responses (images, PDFs, audio, video) return metadata about the file:

  • Content type and size
  • Filename (from Content-Disposition or URL path)
  • Binary kind (image, audio, video, pdf, binary)

Two modes:

  1. Metadata-only — content exceeds the download limit (2 MiB without save_binary, 10 MiB with it). Reports size and type without the body.
  2. Saved to disk — when save_binary=true, the binary is written to <tmpdir>/opencode-smartfetch/<filename> and the response includes the filesystem path.

Blocked redirects

When a cross-origin redirect is blocked by policy, the response explains which URL was attempted and provides the redirect URL so you can fetch it directly.

Secondary Model

When a prompt parameter is supplied, webfetch can route the fetched content through a secondary (cheaper) model for focused extraction. This lets you ask questions like "summarize this page" or "extract the code examples" in one step.

How it works:

  1. Content is fetched and cached normally.
  2. A temporary OpenCode session is created with all tools disabled.
  3. The fetched content and your prompt are sent to a secondary model.
  4. The session is cleaned up after the response.

Which model is used (in priority order):

  1. webfetch.model (dedicated — highest priority, supports array for fallback)
  2. small_model from the OpenCode configuration (opencode.json / opencode.jsonc)
  3. The configured explorer agent model
  4. The configured librarian agent model

The secondary model is called only when all of these are true:

  • A prompt parameter is provided
  • A secondary model is configured
  • The fetched content has at least 25 words

If the secondary model fails (timeout, error, empty response), webfetch returns the raw fetched content as a graceful fallback.

Caching

Fetches are cached in memory with an LRU cache (50 MiB max, 15-minute TTL). The cache key includes the URL plus behavior-affecting options (extract_main, prefer_llms_txt, save_binary), so changing these re-fetches the URL.

Revalidation: Cache entries with ETag or Last-Modified headers support conditional revalidation. When a stale entry exists, webfetch sends If-None-Match / If-Modified-Since headers. A 304 Not Modified response refreshes the TTL without re-downloading.

llms.txt validation: Cached llms.txt results are validated — if the cached entry doesn't actually look like an llms.txt response (wrong path, HTML content, login page), it is evicted and re-fetched.

llms.txt Probing

For documentation sites, webfetch probes for /llms-full.txt then /llms.txt before falling back to the page itself.

Probing behavior depends on the prefer_llms_txt parameter:

  • "auto" (default) — probes only when the domain looks documentation-adjacent (suffixes like .readthedocs.io, .gitbook.io, docs.rs; prefixes like docs., developer., dev., wiki.)
  • "always" — always probes; fails with a message if neither llms.txt variant exists
  • "never" — skips probing entirely

The probe respects cross-origin redirect policy (same origin only). If the llms.txt response is HTML or a login page, the probe is rejected.

Redirect Policy

webfetch follows up to 10 redirects per request, but only within same-origin scopes. Cross-origin redirects are blocked and the caller is instructed to fetch the new URL directly.

For URLs entered as http://, webfetch first tries https:// and falls back to http:// if the HTTPS attempt fails (connection error, blocked redirect, or non-2xx status).

Binary Detection

Content type detection follows this flow:

  1. Explicit binary MIME types (image/*, audio/*, video/*, application/pdf, application/zip, application/octet-stream) are treated as binary.
  2. application/octet-stream and known text types are re-examined — the first 2 KiB is scanned for null bytes and non-printable characters to distinguish text from binary.
  3. Content declared as text/plain that looks like HTML is upgraded to text/html for better content extraction.

Tool Timeouts

  • Default timeout: 30 seconds
  • Maximum timeout: 120 seconds
  • llms.txt probe timeout: capped at 8 seconds within the overall timeout
  • Multiple scoped timeouts run in parallel (llms.txt probing and page fetch are independent within a single call)

Configuration

Disabling

Set webfetch.enabled to false to skip registering the enhanced version and use OpenCode's built-in webfetch instead:

{
  "webfetch": {
    "enabled": false
  }
}

Dedicated secondary model

The webfetch.model option sets a dedicated model (or array of fallback models) for secondary-model summarization. Takes priority over all other model resolution sources. Accepts the same format as agent model configs:

{
  "webfetch": {
    "model": "openai/gpt-4o-mini"
  }
}

Multiple fallback models in priority order:

{
  "webfetch": {
    "model": ["openai/gpt-4o-mini", "anthropic/claude-3-haiku"]
  }
}

With optional variant:

{
  "webfetch": {
    "model": [
      "openai/gpt-4o-mini",
      { "id": "anthropic/claude-3-haiku", "variant": "low-latency" }
    ]
  }
}

Each entry is tried in turn; the first to return usable text is used.

Secondary model fallback chain

The secondary model is resolved from these sources (in priority order):

  1. webfetch.model (dedicated — highest priority, supports array for fallback)
  2. small_model in the OpenCode config (opencode.json / opencode.jsonc at project or user level)
  3. The plugin's agents.explorer.model config
  4. The plugin's agents.librarian.model config

Example opencode.jsonc:

{
  "small_model": "openai/gpt-4o-mini"
}

Or in the plugin's opencode.json preset or project config:

{
  "agents": {
    "explorer": { "model": "anthropic/claude-3-haiku" },
    "librarian": { "model": "openai/gpt-4o-mini" }
  }
}

Permissions

The webfetch permission can be configured in the plugin's permission rules. See Configuration for details.

Registration

The tool is registered under the name webfetch in src/index.ts, which overrides OpenCode's built-in webfetch when this plugin is active.

Implementation

The enhanced webfetch lives in src/tools/smartfetch/ (the internal module is named "smartfetch", while the public tool name is webfetch). It is composed of these modules:

Module Responsibility
tool.ts Entry point — permission prompts, cache lookup, llms.txt preference logic, binary-vs-text branching, metadata emission, secondary-model integration
network.ts URL normalization, redirect policy, charset/body decoding, header extraction, llms.txt probing, HTTP fetch with HTTPS upgrade fallback
utils.ts HTML extraction (Mozilla Readability + Turndown), heading cleanup, markdown/text cleaning, frontmatter generation, quality signal detection
cache.ts LRU cache keyed by URL + behavioral options, conditional revalidation, canonical URL aliasing, llms result invalidation
binary.ts Binary content persistence to disk, MIME-to-extension mapping, safe filename allocation
secondary-model.ts Dedicated webfetch/small_model config resolution, temporary session creation, content truncation, model fallback chain
constants.ts Timeouts, size limits, docs domain heuristics, binary MIME prefixes, tool description