Skip to content

feat(reasoners): wire agentfield web_search into 6 high-leverage reasoners#63

Closed
AbirAbbas wants to merge 4 commits into
mainfrom
feat/web-search-on-key-reasoners
Closed

feat(reasoners): wire agentfield web_search into 6 high-leverage reasoners#63
AbirAbbas wants to merge 4 commits into
mainfrom
feat/web-search-on-key-reasoners

Conversation

@AbirAbbas

@AbirAbbas AbirAbbas commented May 6, 2026

Copy link
Copy Markdown
Collaborator

Summary

Wires agentfield.tools.web_search (introduced in Agent-Field/agentfield#543) into the SWE-AF reasoners where external documentation lookup measurably improves output. The selection was made deliberately — most reasoners get nothing from web access and would only pay the latency tax.

Tier 1 — clear value, no caveats

Reasoner Why
run_architect Library / framework / API decisions cascade to every downstream coder. One good architecture decision saves dozens of failed coding loops.
run_ci_fixer CI errors are exactly what Stack Overflow exists for. Diagnosing a build failure without web access is hand-tied.
run_pr_resolver Same as ci_fixer + addresses review comments that may reference external context.

Tier 2 — added with prompt-level scoping

Reasoner Caveat
run_coder Highest loop-risk reasoner. Appends WEB_SEARCH_CODER_GUARDRAIL to the system prompt restricting use to: unfamiliar APIs, opaque error messages, library-version compatibility, deprecation status. Default is to write code; reach for web_search only when blocked by external context.
run_retry_advisor Diagnosing a coder failure often hinges on an error-message lookup. Cheap (advisory-only).
run_product_manager PRDs occasionally need external grounding. Marginal — included since the wiring cost is one line.

Deliberately NOT wired

run_tech_lead, run_sprint_planner, run_issue_writer, run_verifier, run_qa, run_code_reviewer, run_qa_synthesizer, generate_fix_issues, run_merger, run_integration_tester, run_repo_finalize, run_github_pr, run_ci_watcher, run_replanner, run_issue_advisor, and all *workspace* / git_init reasoners. These are pure git/synthesis ops, deterministic flows, or already have full repo context to answer from.

test_run_tech_lead_does_not_attach_web_search and test_run_sprint_planner_does_not_attach_web_search are negative-coverage tests that guard against scope creep silently re-enabling web access on excluded reasoners.

How it works

Each wired call replaces tools=[...] with **with_web_search([...]), which:

  1. Extends the tools list with the namespaced mcp__af_search__web_search tool name
  2. Attaches an in-process MCP server (built via claude_agent_sdk.create_sdk_mcp_server) under mcp_servers["af_search"]

The MCP server is built once per process via lru_cache.

Rollout

This PR ships with the existing agentfield>=0.1.77 floor. The helper is gracefully degrading:

  • When agentfield 0.1.78+ is installed (with the new tools.web_search module), the feature activates automatically.
  • When the older 0.1.77 is installed (current floor), with_web_search silently no-ops and returns just {"tools": [...]}. Reasoners behave identically to today.

A one-line follow-up bumps agentfield>=0.1.78 once that release ships, activating the feature in CI / production. No flag day required.

Test plan

  • 7 unit tests on the helper itself (output shape, mutation safety, both graceful-degradation branches, server-build memoization, guardrail text contract)
  • 8 wiring contract tests proving each of the 6 wired reasoners passes mcp_servers + the namespaced tool name to harness, plus 2 negative-coverage tests for the deliberately-excluded run_tech_lead and run_sprint_planner
  • run_coder test additionally verifies WEB_SEARCH_CODER_GUARDRAIL ends up in the system prompt — the loop-risk mitigation
  • All 15 new tests are sub-second, no API keys required, run on every CI invocation

The end-to-end test that drives a real Claude harness through the full search-and-respond pipeline lives in agentfield as tests/test_harness_web_search_e2e.py (harness_live marker, gated on ANTHROPIC_API_KEY + JINA_API_KEY).

Pre-existing test failures

tests/test_conftest_malformed_planner_execute_nodeids_integration.py::test_agentfield_server_guard_logic_rejects_real_hosts and ::test_agentfield_server_guard_accepts_localhost_url fail on main independently of this PR. Verified by checking out clean main and running both — same 2 failures, no others. Not in scope for this PR; flagging for a separate fix.

Sister PR

Depends on (does not block-merge):

🤖 Generated with Claude Code

AbirAbbas and others added 4 commits May 6, 2026 15:47
@AbirAbbas @claude
feat(tools): add web_search helper that splices into router.harness c…
4542b43 
…alls

Introduces swe_af.tools.web_search.with_web_search(tools), which reasoners
splice into a router.harness call to attach agentfield's in-process MCP
web_search tool::

    result = await router.harness(
        prompt=...,
        schema=...,
        cwd=...,
        max_turns=...,
        permission_mode=...,
        **with_web_search(["Read", "Write", "Glob", "Grep", "Bash"]),
    )

Also exports WEB_SEARCH_CODER_GUARDRAIL — a system-prompt snippet appended
to the coder reasoner's prompt to mitigate the loop-risk of a many-turn
coding agent rabbit-holing on web searches instead of writing code.

Graceful degradation: if the installed agentfield doesn't ship the
tools.web_search helper (i.e. <0.1.78 floor), or claude_agent_sdk isn't
present, with_web_search silently no-ops — returning just the original
tools list. Reasoners stay correct under any deployment; web search is
opportunistic, not required.

The MCP server is built once per process via lru_cache so all reasoners
share a single in-process server instance.

Tests: 7 unit tests covering helper output shape, list-mutation safety,
both graceful-degradation branches, server-build memoization, and the
guardrail text contract.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@AbirAbbas @claude
feat(reasoners): wire web_search into the 6 highest-leverage reasoners
73564b3 
Enables agentfield's web_search MCP tool on the reasoners where external
documentation lookup measurably improves output quality, while leaving
the deterministic / synthesis-only / short-budget reasoners untouched.

Tier 1 (web search adds clear value):
- run_architect       — library/framework/API decisions cascade downstream
- run_ci_fixer        — CI errors are exactly what Stack Overflow exists for
- run_pr_resolver     — same logic as ci_fixer, plus comment context

Tier 2 (with prompt-level scoping for run_coder):
- run_coder           — appends WEB_SEARCH_CODER_GUARDRAIL to system prompt
                        restricting use to API/error/version lookups
- run_retry_advisor   — diagnosing a coder failure often hinges on an error
                        message lookup
- run_product_manager — PRDs occasionally need external grounding

Deliberately NOT wired (would inflate cost without payoff): run_tech_lead,
run_sprint_planner, run_issue_writer, run_verifier, run_qa, run_code_reviewer,
run_qa_synthesizer, generate_fix_issues, run_merger, run_integration_tester,
run_repo_finalize, run_github_pr, run_ci_watcher, run_replanner,
run_issue_advisor, all *workspace* / git_init reasoners.

Each wired call replaces ``tools=[…]`` with ``**with_web_search([…])``,
which extends the tools list with the namespaced web_search tool name and
attaches the in-process MCP server under ``mcp_servers["af_search"]``. The
underlying agentfield change ships in 0.1.78; under the current 0.1.77
floor the helper silently no-ops and reasoners behave identically to today.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@AbirAbbas @claude
test(reasoners): wiring contract tests for the 6 web-search-enabled r…
aa68722 
…easoners

Adds 8 structural tests that prove each chosen reasoner actually attaches
the web_search MCP server when invoked. Approach: stub-agent the singleton
router, AsyncMock the harness call, drive each reasoner with minimum-viable
inputs, then inspect the captured kwargs.

Positive coverage (must attach mcp_servers["af_search"] + the namespaced
web_search tool name):
- run_architect, run_product_manager, run_retry_advisor, run_coder,
  run_ci_fixer, run_pr_resolver

run_coder also asserts WEB_SEARCH_CODER_GUARDRAIL appears in the system
prompt — verifying the loop-risk mitigation is wired.

Negative coverage (must NOT attach web_search — guards against scope creep
silently re-enabling it on excluded reasoners):
- run_tech_lead, run_sprint_planner

These tests are cheap (sub-second) and require no API keys, so they run
in every CI invocation. The proper end-to-end test that drives a real
Claude harness through the full pipeline lives in agentfield as
tests/test_harness_web_search_e2e.py (harness_live marker, gated on
ANTHROPIC_API_KEY + JINA_API_KEY).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@AbirAbbas @claude
fix(tests): mock get_web_search_server so wiring tests pass on the 0.…
2587eac 
…1.77 floor

This PR ships against the existing ``agentfield>=0.1.77`` floor; the new
``agentfield.tools.web_search`` helper that the reasoner wiring depends on
ships in 0.1.78. In CI the floor pin is what gets installed, so
``swe_af.tools.web_search.get_web_search_server`` is None and the helper
correctly falls into its graceful-degradation path — but the wiring
contract tests assert ``mcp_servers`` IS attached, so they failed.

The fix monkeypatches ``get_web_search_server`` to return a deterministic
fake in the captured_harness fixture (and a parallel fake_web_search
fixture in the helper unit tests). This makes the wiring tests verify
the wiring shape independent of which agentfield version is installed.

The dedicated graceful-degradation tests
(test_graceful_degradation_when_agentfield_missing_web_search /
test_graceful_degradation_when_get_server_raises_import_error) continue
to exercise the real None / ImportError paths without the fake.

Verified locally:
- with agentfield 0.1.77 (floor, no tools.web_search): 15/15 pass
- with local /home/abir/gb/agentfield (the 0.1.78 source): 15/15 pass

When the floor pin bumps to 0.1.78, the same tests will continue to pass
against the real helper rather than the fake — same shape, no rewrites.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@AbirAbbas

AbirAbbas commented May 6, 2026

Copy link
Copy Markdown
Collaborator Author

Closing in favor of a much smaller approach using each provider's first-party web search tools.

See Agent-Field/agentfield#543 for the framework rationale. Net plan:

  • claude-code: add WebSearch/WebFetch to the tools allow-list on the chosen reasoners, gated on SWE_AF_ENABLE_WEB_SEARCH=1 (off by default).
  • opencode: nothing — works automatically when OPENCODE_ENABLE_EXA=1 + EXA_API_KEY are in the deployment env. The opencode subprocess inherits parent env via agentfield's run_cli.

Replacement PR coming on a fresh branch. No agentfield change required.

@AbirAbbas AbirAbbas closed this May 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@AbirAbbas