Introduction
Would it make sense to expose a simple fetch / plain scrape tool that only fetches and extracts the content, separate from the existing query aware/RAG/chunk selection path?
The current scrape tool has gradually become more than a scraper. In addition to fetching and extracting URLs, it can take a query, chunk the extracted page, score evidence against that query, and select chunks according to evidence budgets. The functionality is useful, but I think it represents a different operation from simply fetching an URL and I think it would be useful to expose both explicitly.
Motivation
- Fetching and relevance have different semantics
When I ask an MCP tool to scrape a URL, my expectation is roughly "fetch this page and give me its contents". Query aware chunk ranking instead means "fetch this page and decide which portions are relevant enough". Both are valid operations, but the latter is inherently a reranking/RAG operation and can be lossy. Keeping them separate would make the tool selection much easier for both users and models: e.g., using scrape/fetch for boring scraping and context_scrape/context_fetch for the reranking/RAG workflow for potentially huge documents/pages.
Technically I can work around this with the following config but it's much cleaner to simply expose them as separate tools.
scrape:
timeout: 45000
textMode: full
maxUrls: 10
concurrency: 5
maxResponseBytes: 33554432
maxEvidenceChars: 10000000
maxCharsPerUrl: 1000000
chunkChars: 1000000
maxChunksPerUrl: 1024
Furthermore, there are many potential ways to implement the RAG/reranking:
lexical scoring
BM25
embeddings
hybrid retrieval
cross encoder reranking
LLM reranking
different chunking strategies
...
That can become a fairly large subsystem on its own.
- It does not require removing the existing feature
I'm not suggesting removing query aware content extraction. For smaller models, context reranking before returning helps keep the context window small and the model focused. The proposal is simply to also offer the boring scrape primitive.
Introduction
Would it make sense to expose a simple
fetch/ plainscrapetool that only fetches and extracts the content, separate from the existing query aware/RAG/chunk selection path?The current
scrapetool has gradually become more than a scraper. In addition to fetching and extracting URLs, it can take aquery, chunk the extracted page, score evidence against that query, and select chunks according to evidence budgets. The functionality is useful, but I think it represents a different operation from simply fetching an URL and I think it would be useful to expose both explicitly.Motivation
- Fetching and relevance have different semantics
When I ask an MCP tool to scrape a URL, my expectation is roughly "fetch this page and give me its contents". Query aware chunk ranking instead means "fetch this page and decide which portions are relevant enough". Both are valid operations, but the latter is inherently a reranking/RAG operation and can be lossy. Keeping them separate would make the tool selection much easier for both users and models: e.g., using
scrape/fetchfor boring scraping andcontext_scrape/context_fetchfor the reranking/RAG workflow for potentially huge documents/pages.Technically I can work around this with the following config but it's much cleaner to simply expose them as separate tools.
Furthermore, there are many potential ways to implement the RAG/reranking:
That can become a fairly large subsystem on its own.
- It does not require removing the existing feature
I'm not suggesting removing query aware content extraction. For smaller models, context reranking before returning helps keep the context window small and the model focused. The proposal is simply to also offer the boring scrape primitive.