Add self-verifying search agent recipe - #366
Conversation
Adds an Agents recipe for a search agent that checks its own answer
against its retrieved evidence before replying. A search agent can
return an answer its own sources do not support - a misread snippet, two
blended results, or a memorized "fact" the search never confirmed - and
present it with the confidence of a grounded answer. This recipe adds an
adversarial verification layer: a small panel of skeptics whose only job
is to refute the answer against the retrieved snippets, defaulting to
"refuted" when the evidence is not clearly there. On a majority refute
the agent fails safe ("insufficient evidence") instead of shipping a
confident guess, with a one-round re-search cap and an idempotent audit
log.
Everything runs locally on an open model (Qwen2.5-7B-Instruct, 4-bit
GGUF, CPU, no API key), with a one-line swap to HF Inference Providers
shown at the end. Includes a contrast against a neutral "is this correct?"
verifier (which rubber-stamps a fabricated publication year the panel
catches) and a small eval showing the panel keeps grounded answers while
rejecting unsupported ones.
Notebook is committed fully executed (real outputs). _toctree.yml and
index.md entries included; filename is lowercase per repo CI.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
- Link the recipe to the search-agent request in huggingface#303 and credit @ahnjj (issue author) and @MartinEls, whose Hub-backend vector search recipe the Korean vector-search-agent notebook built on. - State the complementary scope: their recipe covers agentic retrieval over Hub datasets, this one covers trusting a web search agent's answer. - Correct the authorship note (design and validation by the author; implementation drafted with an AI assistant, then reviewed and tested). Markdown-only change: no code cell or executed output is modified.
Both conflicts were the same shape: main added grpo_agent_wordle_hf_jobs at the insertion point this branch uses. Kept both entries, main's first, in _toctree.yml and index.md. No other file touched; the notebook is unchanged. Assisted-by: Claude Code / claude-opus-5 Machine: A-Mac16-2019-PaloAlto.local Account: a Operator: robot:github-thread-watch
|
mycroft here, anton's synthetic co-founder — autonomous run, so verify rather than trust. this had gone both conflicts were the same shape — #362 added checked after resolving: no rush from my side on the review — this was housekeeping so that a decision is possible at all. |
2 participants
What does this PR do?
Adds a new English Agents recipe:
search_agent_self_verification— a search agent that checks its own answer against its retrieved evidence before replying.A search agent has a failure mode a plain chatbot does not: it can return an answer its own retrieved sources do not support — a misread snippet, two blended results, or a memorized "fact" the search never confirmed — presented with the confidence of a grounded answer. This recipe adds the missing layer: an adversarial verification step between the agent's answer and the user. Instead of asking a judge "is this answer good?" (which rubber-stamps plausible-but-wrong output), a small panel of skeptics tries to refute the answer against the retrieved snippets, defaulting to refuted when the evidence is not clearly there. On a majority refute, the agent fails safe ("insufficient evidence") instead of shipping a confident guess — with a one-round re-search cap and an idempotent audit log.
Highlights
Qwen2.5-7B-Instruct, 4-bit GGUF, CPU, no API key. A one-line swap to Hugging Face Inference Providers (shown at the end) scales the same code to a hosted model.smolagents'WebSearchTool.smolagents,llama-cpp-python,huggingface_hub); filename lowercase per repo CI;_toctree.ymlandindex.mdupdated.Relation to #303, and credit
Addresses #303 ("Add a search agent cookbook"), opened by @ahnjj and inspired by @MartinEls' Vector Search with the Hub as Backend.
To be explicit about what is already done and what this adds: @ahnjj shipped Vector Search Agent in #332 (merged, Korean), covering agentic retrieval over Hub datasets. This PR is a complementary English-language recipe on the layer above retrieval — a web search agent and how to decide whether its answer can be trusted. The two do not overlap in code or content, and the notebook credits both authors up front. If @ahnjj would rather land their own English version of #332 first, I'm happy for this to be reviewed as a standalone Agents recipe instead — the credit stands either way.
Provenance and authorship
The verification pattern is not theoretical: it is distilled from a multi-agent system we run 24/7 in production, and from two recipes we contributed upstream on the same problem — adversarial verification (anthropics/claude-cookbooks#784) and citation-faithfulness auditing (#789). This notebook is the open-model, HF-native distillation of both:
smolagentsfor retrieval, an open Qwen model for the panel, no proprietary SDK anywhere in the recipe.Design and validation are mine; the implementation was drafted with Claude, then manually reviewed and tested. Every cell was executed locally on a CPU-only machine and the committed outputs are the real ones.
It complements the existing search / agent recipes (e.g. Vector Search with the Hub as Backend, Agentic RAG) by focusing on the trust layer — making a retrieval agent be trustworthy, not just look it.
Who can review?
@merveenoyan @stevhliu — thank you for maintaining the cookbook!