diff --git a/docs/phase-oracle-browser-researcher.md b/docs/phase-oracle-browser-researcher.md new file mode 100644 index 00000000..c6aa38ac --- /dev/null +++ b/docs/phase-oracle-browser-researcher.md @@ -0,0 +1,232 @@ +# Oracle Browser Researcher Adapter + +## Goal + +Give the roster's `researcher` role a working model again on a subscription +cookie lane, with no API key and no new Brigade browser code, by adding +`oracle` as a one-shot CLI adapter. + +## Context + +Three facts set the shape of this phase: + +- `research/llm.py:58` `resolve_backend` already returns a `CliBackend` for a + `researcher` agent that declares `cli`, and `research/engine.py:64` only ever + calls `.complete()`. Backend selection needs no change. +- `research/sources/web.py` already ships `PlaywrightProvider`, a headless + Chromium tier that searches DuckDuckGo and carries `trust = "browser"`, + behind `pip install 'brigade[research]'`. Research already has web grounding. + The gap is a planning and synthesis model, not search. +- Agent Pantry already syncs the encrypted `gemini.google.com` cookie jar + between machines, and `brigade pantry expiry-alert` already warns before it + lapses. Oracle's browser engine reads exactly those cookies from the default + Chrome profile. + +So the whole change is one adapter. Pantry supplies the cookies, oracle drives +the session, Brigade shells out and reads text back. + +## Boundary + +- Require a user-installed `oracle` (MIT, npm `@steipete/oracle`). Brigade does + not install it, does not bundle Node, and does not add it to the component + manifest in this phase. +- Browser engine only (`--engine browser`). The adapter never emits an + API-key path, so the lane cannot silently fall off the subscription rail. +- Read-only is hard by construction: oracle has no filesystem write path, so + no flag and no prompt instruction are needed to enforce it. +- Scope is the `researcher` role. No run transport, no ChatGPT Pro seat, no + multi-model panel, no oracle sessions or `--followup`, no MCP bridge, no cost + reporting. Those are later phases. +- Known side effect: registering the adapter makes `cli = "oracle"` assignable + to any seat, not only the researcher, because `_ADAPTERS` is a global table. + That is accepted rather than gated, but it means the read-only enforcement + contract has to hold for `brigade run` from day one. +- Keep the existing DuckDuckGo web tier as the search provider. Oracle does + planning, query generation, and synthesis only. + +## Design + +Four layers, three of them already built: + +| Layer | Component | Change | +|---|---|---| +| Auth | Agent Pantry cookie sync | none | +| Driver | `oracle --engine browser` | none, external binary | +| Adapter | `src/brigade/agents.py` | new, small | +| Caller | roster `researcher` | config only | +| Timeout | `src/brigade/research/llm.py` | one floor, see below | + +The one non-adapter change: `research/engine.py:138` asks for `timeout=30` on +its planning call and `research/types.py:50` defaults to `timeout=60`, both +hardcoded at the call site. A browser round trip will not meet 30 seconds, so +the lane fails on its first request without a floor. The roster already carries +per-agent `timeout_seconds` and `resolve_backend` currently discards it, so +`CliBackend` takes it as a `min_timeout` that raises short engine timings and +never lowers generous ones. No new config concept, no edits to the engine's +literals, and seats that declare no `timeout_seconds` keep today's behavior. + +Adapter surface in `src/brigade/agents.py`: + +```python +def _oracle_argv(prompt: str, read_only: bool, sandbox: str | None, cwd: Path | None) -> List[str]: + # Oracle has no filesystem write path, so read-only needs no flag and no + # prompt instruction. --engine browser pins the run to the cookie lane so + # the adapter can never fall back to an API key. + return ["oracle", "--engine", "browser", "-p", prompt] +``` + +Registrations: + +- `_ADAPTERS["oracle"] = _oracle_argv` +- `READ_ONLY_ENFORCEMENT["oracle"] = "hard"` +- `_MODEL_PIN["oracle"] = ("--model", _pin_after_cmd)`, producing + `oracle --model gemini-3.1-pro --engine browser -p ` + +`command_for` needs no entry: it falls through to the ref name, and the binary +is already called `oracle`. + +Roster: + +```toml +[agents.researcher] +cli = "oracle" +model = "gemini-3.1-pro" +role = "researcher" +timeout_seconds = 300 +``` + +Data flow: `brigade research run ""` -> `DeepResearcher` -> +`llm.complete()` -> `CliBackend("oracle", "gemini-3.1-pro")` -> `run_agent` -> +`_ADAPTERS` -> subprocess -> stdout -> `validate_final_output` -> engine. + +Models the browser engine accepts: `gemini-3.5-flash`, `gemini-3.1-pro`, and +`gemini-3-deep-think` (browser-only; oracle rejects it in API mode). + +## Failure handling + +- **oracle absent.** `resolve_agent_executable` already returns + `failure_kind="command-not-found"` at `failure_phase="dispatch"`. No work. +- **cookies expired.** This must not surface as a generic nonzero exit. + `_oracle_auth_detail` recognises oracle's login and expired-session messages + and points the operator at `brigade pantry expiry-alert`. It is a sibling of + `_provider_preflight_detail`, not a branch inside it: that function is about + workspace trust, a concept oracle does not have. Both failure paths in + `run_agent` try the auth detail first, since it is the more specific + diagnosis, and an auth hit reports `failure_kind="browser-auth"` rather than + `"workspace-trust"`. Keeping those kinds distinct matters downstream: outcome + capture and the model scorecard read `failure_kind`, and a stale cookie jar + is an operator action while a trust refusal is a workspace problem. The + first implementation shared the preflight branch and mislabelled every + oracle auth failure as `workspace-trust`; the regression tests that caught + it exercise both call sites through `run_agent`, not the detail function + alone. +- **browser too slow for the engine's timings.** Not a hang, the common case. + Fixed by the `min_timeout` floor above, driven by the seat's + `timeout_seconds`. `.brigade/research.toml` is the wrong home for this: + `Caps` has no timeout field, and `Caps.build` silently drops unknown keys, so + a config-only attempt would look applied and do nothing. +- **genuine browser hang.** Covered by `run_agent(timeout=...)` once the floor + raises it to the seat's declared ceiling. +- **partial or scraped garbage.** `validate_final_output` already runs at + `agents.py:1182`. If oracle's stdout wraps the answer in progress chrome, add + an extraction function following the `_parse_grok_final_output` precedent + (`agents.py:360`) rather than loosening validation. +- **no silent fallback.** If the researcher role fails, `research` raises + rather than quietly degrading to another seat, matching the existing + `NoResearcherError` semantics. + +## Verification + +- [x] Unit test `_oracle_argv` argv construction, including model pin position. +- [x] Test that the adapter never emits an API-mode argv. +- [x] Test `READ_ONLY_ENFORCEMENT` reports `hard`, and that + `brigade run --read-only` raises no soft-enforcement warning for an + oracle seat. In scope despite the researcher-only boundary, because + registering the adapter makes oracle dispatchable by `brigade run`. +- [x] Test roster validation accepts a `cli = "oracle"` researcher and that + `resolve_backend` returns a `CliBackend`. +- [x] Test the `min_timeout` floor raises a short engine timeout, never lowers + a generous one, and leaves seats without `timeout_seconds` unchanged. +- [x] Test the expired-cookie preflight detail string. +- [x] Run focused tests and `./scripts/verify` through + `brigade work verify run`. +- [x] Live smoke: one `brigade research run` against real synced cookies, + recording the result or the environmental blocker. + +## Proof coverage + +What is proven, and by what: + +| Claim | Evidence | +|---|---| +| argv shape, model pin position, no `--heartbeat`, no API path | unit, `tests/test_agents_oracle.py` | +| argv survives a real `exec` | stub binary on a narrowed PATH records its own argv | +| stdout is the answer channel at the Brigade layer | stub returns markdown, `run_agent` returns it verbatim | +| auth failure on the nonzero-exit path | `run_agent` returns `failure_kind="browser-auth"` | +| auth failure on the empty-output path | separate branch, same assertion | +| auth beats workspace-trust when both patterns appear | stub emits both, auth wins | +| non-oracle seats keep `workspace-trust` | codex regression guard | +| timeout floor survives the real engine | `DeepResearcher` run asserts 300, not the engine's 30 | +| roster accepts `cli = "oracle"` | clears `is_known` and `limits.allow_models` | + +Still unproven, and only oracle itself can settle it: whether real oracle stdout +carries progress chrome around the answer, and whether the browser session +actually completes against synced cookies. + +## Live result + +The adapter dispatches correctly and fails cleanly when the binary is absent. +`agents.run_agent("oracle", "hello")` returned `ok=False`, +`failure_phase="dispatch"`, `failure_kind="command-not-found"`, detail +`oracle not installed`, and `build_argv` produced exactly +`['oracle', '--model', 'gemini-3.1-pro', '--engine', 'browser', '-p', 'hello']`. + +A full browser round trip could not run, blocked twice on this machine: + +- `oracle` is not installed (`command not found`, and no global npm package). + Brigade does not install it by design. +- `brigade pantry status` reports the agentpantry build **rejected by version + policy** (unreleased or non-semver build; expected released >= 0.5.0), so the + cookie substrate is not healthy here either. + +Consequently the settled stdout decision is **not yet empirically confirmed**. +It rests on reading oracle's `src/cli/renderOutput.ts` (`if (!richTty) return +markdown;`, `richTty` defaulting to `process.stdout.isTTY`) plus never passing +`--heartbeat`. The first real run should check whether stdout carried only the +answer; if it did not, add an extraction function following the +`_parse_grok_final_output` precedent rather than loosening +`validate_final_output`. + +## Resolved during planning + +- **Is stdout clean enough to skip an extraction function? Yes.** Oracle's + `src/cli/renderOutput.ts` returns `markdown` unrendered when `richTty` is + false, and `richTty` defaults to `process.stdout.isTTY`. Brigade captures + subprocess pipes, so that is false and no ANSI reaches stdout. Heartbeat + progress is opt-in behind `--heartbeat`, which the adapter never passes. No + extraction function, and a test asserts the flag is never emitted. + +## Open questions + +- Whether to map Brigade's `reasoning` pin onto + `--browser-thinking-time `. Cheap and a + natural fit, but not needed for research synthesis. Deferred to the ChatGPT + Pro phase, where thinking depth is the whole point. + +## Risks + +- Oracle's browser mode is labelled experimental by its author. It is a + third-party Node tool driving a web UI that can change without notice. The + `cli = "oracle"` seam keeps it replaceable: a future Brigade-owned driver + swaps the adapter without touching `research/`. +- Cookie-driven automation of a consumer web session is a grey area against + provider terms. This is a single-operator machine lane. It should not become + a fleet default or a documented supported install path. + +## Later phases + +1. ChatGPT Pro reviewer seat via a `transport = "browser"` roster entry, with + receipts and outcome capture. acpx is the sizing precedent. +2. `--browser-thinking-time` reasoning pin. +3. Component manifest entry and a station-style doctor, if the lane proves + durable enough to be worth pinning a version against. diff --git a/docs/seat-catalog.md b/docs/seat-catalog.md index ff7c40e5..2c1d8e60 100644 --- a/docs/seat-catalog.md +++ b/docs/seat-catalog.md @@ -168,6 +168,39 @@ role = "Hosted open-weight worker on the ollama cloud free tier." Brigade never auto-pulls ollama models: dispatch fails unless the model is already present, which protects the disk from multi-GB surprise pulls. Hosted models can be retired upstream without notice, so validate before each wiring. +### Gemini web via oracle (browser cookie lane) + +The only lane here that is neither an API nor a coding CLI. [oracle](https://github.com/steipete/oracle) +drives a real `gemini.google.com` session using the Chrome cookies Agent Pantry +syncs, so it costs no API key and no metered quota. It is a consult seat: one +shot, no tools, no file writes. + +Requires a user-installed `oracle` (`npm install -g @steipete/oracle`) and a +pantry-synced cookie jar. Brigade installs neither. + +```toml +[agents.researcher] +cli = "oracle" +model = "gemini-3.1-pro" +role = "researcher" +timeout_seconds = 300 +``` + +Models the browser engine accepts: `gemini-3.5-flash`, `gemini-3.1-pro`, and +`gemini-3-deep-think` (browser-only; oracle rejects it in API mode). + +Read-only enforcement is `hard` by construction, since oracle has no filesystem +write path. When the seat fails with a login or expired-session message, run +`brigade pantry expiry-alert` and re-sync the source before retrying. + +A browser round trip is much slower than a CLI seat. The research engine asks +for a 30 second planning call, which a browser session will not meet, so the +seat must declare its own `timeout_seconds`; `research` takes it as a floor that +raises short engine timings and never lowers generous ones. + +Keep this to a single operator machine. Oracle's browser mode is experimental +and cookie automation of a consumer session is grey against provider terms. + ## Validate before trusting A seat that answers a smoke prompt is wired, not proven. The pattern: diff --git a/src/brigade/agents.py b/src/brigade/agents.py index 71c3c355..7eb6a0dc 100644 --- a/src/brigade/agents.py +++ b/src/brigade/agents.py @@ -276,6 +276,17 @@ def _crush_argv(prompt: str, read_only: bool, sandbox: str | None, cwd: Path | N return ["crush", "run", task] +def _oracle_argv(prompt: str, read_only: bool, sandbox: str | None, cwd: Path | None) -> List[str]: + # Oracle is a one-shot consult CLI with no filesystem write path, so + # read_only needs neither a flag nor a prompt instruction and the argv is + # identical either way. --engine browser pins the run to the cookie lane + # that Agent Pantry keeps fresh, so the adapter can never silently fall + # back to an API key. --heartbeat is deliberately never passed: stdout + # carries the answer, and oracle emits plain unrendered markdown there + # whenever stdout is not a TTY. + return ["oracle", "--engine", "browser", "-p", prompt] + + _ADAPTERS: dict[str, Callable[[str, bool, str | None, Path | None], List[str]]] = { "claude": _claude_argv, "codex": _codex_argv, @@ -294,6 +305,7 @@ def _crush_argv(prompt: str, read_only: bool, sandbox: str | None, cwd: Path | N "grok": _grok_argv, "amp": _amp_argv, "crush": _crush_argv, + "oracle": _oracle_argv, } @@ -321,6 +333,8 @@ def _crush_argv(prompt: str, read_only: bool, sandbox: str | None, cwd: Path | N "crush": "soft", "claude": "hard", "opencode": "none", + # Hard by construction rather than by sandbox: oracle cannot write files. + "oracle": "hard", } @@ -500,6 +514,37 @@ def _provider_preflight_detail(cli_ref: str, stdout: str, stderr: str) -> str | ) +_ORACLE_AUTH_RE = re.compile( + r"(?:" + r"\bnot logged in\b|" + r"\bsign[- ]in\b|" + r"\blogin required\b|" + r"\b(?:browser )?session (?:expired|invalid)\b|" + r"\bcookies? (?:expired|missing|invalid|not found)\b" + r")", + re.IGNORECASE, +) + + +def _oracle_auth_detail(cli_ref: str, stdout: str, stderr: str) -> str | None: + """Turn an oracle browser-session auth failure into a pantry next step. + + Oracle reads its cookies from the Chrome profile Agent Pantry syncs, so a + stale jar is an operator action, not a model failure. Everything else about + oracle failing stays generic. + """ + if cli_ref != "oracle": + return None + combined = "\n".join(part for part in (stderr, stdout) if part).strip() + if not combined or not _ORACLE_AUTH_RE.search(combined): + return None + return ( + "oracle could not use the browser session; the synced cookies are " + "likely stale. Check `brigade pantry expiry-alert`, then re-sync the " + "pantry source before retrying" + ) + + def _pin_after_cmd(argv: List[str], flag: str, model: str) -> List[str]: """Insert `flag model` right after the command (argv[0]).""" return [argv[0], flag, model, *argv[1:]] @@ -527,6 +572,7 @@ def _pin_before_prompt(argv: List[str], flag: str, model: str) -> List[str]: "kimi": ("-m", _pin_after_cmd), # kimi -m X -p "cursor": ("--model", _pin_after_cmd), # cursor-agent --model X -p --output-format text -f "antigravity": ("--model", _pin_after_cmd), # agy --model X [--sandbox] --print + "oracle": ("--model", _pin_after_cmd), # oracle --model X --engine browser -p } _REASONING_ADAPTERS = frozenset({"codex", "opencode", "pi", "grok"}) @@ -1064,14 +1110,18 @@ def scrub_detail(detail: str) -> str: :200 ] if result.code != 0: - provider_preflight = _provider_preflight_detail(cli_ref, safe_stdout, safe_stderr) + oracle_auth = _oracle_auth_detail(cli_ref, safe_stdout, safe_stderr) + provider_preflight = oracle_auth or _provider_preflight_detail(cli_ref, safe_stdout, safe_stderr) if provider_preflight is not None: return AgentResult( text=safe_text, ok=False, detail=provider_preflight, failure_phase="provider-preflight", - failure_kind="workspace-trust", + # A stale browser cookie jar is an auth problem, not a + # workspace-trust one; the kinds must stay distinguishable + # downstream in outcome capture and the scorecard. + failure_kind="browser-auth" if oracle_auth else "workspace-trust", stdout=safe_stdout, stderr=safe_stderr, exit_code=result.code, @@ -1155,11 +1205,12 @@ def scrub_detail(detail: str) -> str: detail = "empty output" empty_failure_phase: str | None = None empty_failure_kind: str | None = None - provider_preflight = _provider_preflight_detail(cli_ref, safe_stdout, safe_stderr) + oracle_auth = _oracle_auth_detail(cli_ref, safe_stdout, safe_stderr) + provider_preflight = oracle_auth or _provider_preflight_detail(cli_ref, safe_stdout, safe_stderr) if provider_preflight is not None: detail = provider_preflight empty_failure_phase = "provider-preflight" - empty_failure_kind = "workspace-trust" + empty_failure_kind = "browser-auth" if oracle_auth else "workspace-trust" elif cursor_limitation is not None: detail = cursor_limitation elif cli_ref in {"cursor", "grok"}: diff --git a/src/brigade/research/llm.py b/src/brigade/research/llm.py index e7f5a745..6ab2ff00 100644 --- a/src/brigade/research/llm.py +++ b/src/brigade/research/llm.py @@ -33,12 +33,24 @@ def _messages_to_prompt(messages: List[Dict[str, str]]) -> str: class CliBackend: - def __init__(self, cli: str, model: Optional[str] = None, env: Optional[Dict[str, str]] = None) -> None: + def __init__( + self, + cli: str, + model: Optional[str] = None, + env: Optional[Dict[str, str]] = None, + min_timeout: Optional[int] = None, + ) -> None: self.cli = cli self.model = model self.env = env + # The engine hardcodes short per-call timeouts (30s to plan, 60s + # default). A browser-driven seat cannot meet those, so the roster's + # timeout_seconds raises the floor without lowering anything. + self.min_timeout = min_timeout def complete(self, messages, *, max_tokens=2048, temperature=0.3, timeout=60) -> str: + if self.min_timeout is not None: + timeout = max(timeout, self.min_timeout) return _run_cli(self.cli, _messages_to_prompt(messages), timeout, model=self.model, env=self.env) @@ -62,5 +74,11 @@ def resolve_backend(roster: Any): if getattr(agent, "endpoint", None) and getattr(agent, "model", None): return HttpBackend(agent.endpoint, agent.model, getattr(agent, "headers", None)) if getattr(agent, "cli", None): - return CliBackend(agent.cli, getattr(agent, "model", None), getattr(agent, "env", None)) + raw_timeout = getattr(agent, "timeout_seconds", None) + return CliBackend( + agent.cli, + getattr(agent, "model", None), + getattr(agent, "env", None), + min_timeout=int(raw_timeout) if raw_timeout else None, + ) raise NoResearcherError("researcher agent needs either cli or endpoint+model") diff --git a/tests/test_agents_oracle.py b/tests/test_agents_oracle.py new file mode 100644 index 00000000..475ce602 --- /dev/null +++ b/tests/test_agents_oracle.py @@ -0,0 +1,169 @@ +"""Oracle browser adapter: exact argv assertions, no CLI execution. + +Oracle is not installed on CI machines, so these argv shapes are confirmed +against oracle's published docs/cli-reference.md and docs/gemini.md rather than +a local `oracle --help`. That is why they live here and not in +tests/test_agents_model_pin.py, whose stated invariant is that every adapter it +covers has a confirmed model flag on an installed CLI. +""" + +import json +import shutil +import sys + +from brigade import agents + + +def _stub_proc(monkeypatch, code, stdout, stderr): + monkeypatch.setattr(agents.proc, "which", lambda c: "/x/" + c) + monkeypatch.setattr(agents.proc, "run", lambda argv, **kw: agents.proc.Result(code, stdout, stderr)) + + +def test_oracle_argv_pins_the_browser_engine(): + assert agents.build_argv("oracle", "P") == [ + "oracle", + "--engine", + "browser", + "-p", + "P", + ] + + +def test_oracle_argv_is_identical_under_read_only(): + # Oracle has no filesystem write path, so read-only needs no flag and no + # prompt instruction. The argv must not change at all. + assert agents.build_argv("oracle", "P", read_only=True) == agents.build_argv("oracle", "P") + assert agents.build_argv("oracle", "P", sandbox="read-only") == agents.build_argv("oracle", "P") + + +def test_oracle_argv_pins_model_after_the_command(): + assert agents.build_argv("oracle", "P", model="gemini-3.1-pro") == [ + "oracle", + "--model", + "gemini-3.1-pro", + "--engine", + "browser", + "-p", + "P", + ] + + +def test_oracle_argv_never_emits_heartbeat_or_an_api_path(): + # stdout is the answer channel: --heartbeat would interleave progress lines + # into it, and dropping --engine browser would fall back to an API key. + for argv in ( + agents.build_argv("oracle", "P"), + agents.build_argv("oracle", "P", read_only=True), + agents.build_argv("oracle", "P", model="gemini-3.1-pro"), + ): + assert "--heartbeat" not in argv + assert argv[argv.index("--engine") + 1] == "browser" + + +def test_oracle_read_only_enforcement_is_hard(): + assert agents.read_only_enforcement("oracle") == "hard" + assert agents.read_only_enforcement("oracle", sandbox="read-only") == "hard" + + +def test_oracle_supports_model_pinning_but_not_reasoning(): + assert agents.supports_model_pinning("oracle") is True + # --browser-thinking-time is deferred to the ChatGPT Pro phase. + assert agents.supports_reasoning("oracle") is False + + +def test_oracle_auth_detail_points_at_pantry(): + detail = agents._oracle_auth_detail("oracle", "", "Error: not logged in to gemini.google.com") + assert detail is not None + assert "brigade pantry expiry-alert" in detail + + +def test_oracle_auth_detail_matches_expired_cookies(): + detail = agents._oracle_auth_detail("oracle", "browser session expired", "") + assert detail is not None + + +def test_oracle_auth_detail_ignores_other_clis(): + # A claude seat saying "sign in" is not a pantry problem. + assert agents._oracle_auth_detail("claude", "", "please sign in") is None + + +def test_oracle_auth_detail_ignores_unrelated_oracle_failures(): + assert agents._oracle_auth_detail("oracle", "", "TypeError: bad flag") is None + + +def test_run_agent_surfaces_oracle_auth_failure_as_browser_auth(monkeypatch): + # Call site one: nonzero exit. A stale cookie jar is not workspace trust, + # and mislabelling it as such poisons outcome capture and the scorecard. + _stub_proc(monkeypatch, 1, "", "Error: not logged in to gemini.google.com") + res = agents.run_agent("oracle", "hello") + assert res.ok is False + assert res.failure_phase == "provider-preflight" + assert res.failure_kind == "browser-auth" + assert "brigade pantry expiry-alert" in res.detail + + +def test_run_agent_surfaces_oracle_auth_failure_on_empty_output(monkeypatch): + # Call site two is a separate branch: exit 0, no answer, login on stderr. + _stub_proc(monkeypatch, 0, "", "browser session expired") + res = agents.run_agent("oracle", "hello") + assert res.ok is False + assert res.failure_kind == "browser-auth" + assert "brigade pantry expiry-alert" in res.detail + + +def test_run_agent_prefers_auth_diagnosis_over_workspace_trust(monkeypatch): + # Both patterns present: auth is the more specific diagnosis and wins. + _stub_proc(monkeypatch, 1, "", "not logged in; also workspace is not trusted") + res = agents.run_agent("oracle", "hello") + assert res.failure_kind == "browser-auth" + + +def test_run_agent_keeps_workspace_trust_for_non_oracle_seats(monkeypatch): + # Regression guard: the oracle branch must not swallow the preflight path. + _stub_proc(monkeypatch, 1, "", "error: workspace is not trusted; run from a Git repository") + res = agents.run_agent("codex", "hello") + assert res.ok is False + assert res.failure_kind == "workspace-trust" + + +def test_run_agent_returns_oracle_markdown_from_stdout(monkeypatch): + # The settled decision: stdout IS the answer channel, no extraction step. + _stub_proc(monkeypatch, 0, "## Answer\n\nPlants convert light.\n", "") + res = agents.run_agent("oracle", "hello") + assert res.ok is True + assert res.text == "## Answer\n\nPlants convert light." + + +def test_oracle_argv_survives_a_real_exec(tmp_path, monkeypatch): + # Mocked proc.run cannot prove the argv reaches a real process. A stub + # binary on PATH records what it actually received. The shebang uses an + # absolute interpreter because PATH is narrowed to tmp_path. + argv_log = tmp_path / "argv.json" + stub = tmp_path / "oracle" + stub.write_text( + f"#!{sys.executable}\n" + "import json, sys\n" + f"json.dump(sys.argv[1:], open({str(argv_log)!r}, 'w'))\n" + "print('## Answer')\n" + "print()\n" + "print('real exec ok')\n" + ) + stub.chmod(0o755) + monkeypatch.setenv("PATH", str(tmp_path)) + # conftest's _no_managed_tools_on_path pins proc.which to None so the suite + # behaves like a bare host. Restore real resolution, confined to tmp_path, + # so no host binary can leak in either. + monkeypatch.setattr(agents.proc, "which", lambda cmd, path=None: shutil.which(cmd, path=str(tmp_path))) + + res = agents.run_agent("oracle", "hello", model="gemini-3.1-pro") + + assert res.ok is True + assert res.text == "## Answer\n\nreal exec ok" + assert json.loads(argv_log.read_text()) == [ + "--model", + "gemini-3.1-pro", + "--engine", + "browser", + "-p", + "hello", + ] diff --git a/tests/test_research_engine.py b/tests/test_research_engine.py index c6fc5106..4a7e5915 100644 --- a/tests/test_research_engine.py +++ b/tests/test_research_engine.py @@ -74,3 +74,32 @@ def test_browser_provider_trust_is_preserved(): ) result = eng.research("how do plants make energy?") assert any(f.trust == "browser" for f in result.findings) + + +def test_planning_call_receives_the_seat_timeout_floor(monkeypatch): + """The floor must survive the real engine path, not just a direct complete(). + + engine._plan() hardcodes timeout=30. A browser-driven seat cannot answer in + 30 seconds, so a roster seat declaring timeout_seconds must lift every call. + """ + from brigade.research import llm as llm_mod + + stub = StubLlm() + seen = [] + + def fake_run_cli(cli, prompt, timeout, model=None, env=None): + seen.append(timeout) + return stub.complete([{"role": "user", "content": prompt}]) + + monkeypatch.setattr(llm_mod, "_run_cli", fake_run_cli) + backend = llm_mod.CliBackend("oracle", "gemini-3.1-pro", min_timeout=300) + eng = DeepResearcher( + llm=backend, local_index=StubIndex(), web=None, caps=Caps.build(max_rounds=2, min_rounds=1, max_time=30) + ) + + result = eng.research("how do plants make energy?") + + assert "Plants convert light" in result.report + assert seen, "the engine made no LLM calls" + # 30 is the engine's planning literal; without the floor this would be 30. + assert min(seen) == 300 diff --git a/tests/test_research_llm.py b/tests/test_research_llm.py index b30cf97b..eef6e4e1 100644 --- a/tests/test_research_llm.py +++ b/tests/test_research_llm.py @@ -3,7 +3,9 @@ class FakeAgent: - def __init__(self, name, cli=None, endpoint=None, model=None, role="researcher", headers=None): + def __init__( + self, name, cli=None, endpoint=None, model=None, role="researcher", headers=None, timeout_seconds=None + ): self.name, self.cli, self.endpoint, self.model, self.role, self.headers = ( name, cli, @@ -12,6 +14,7 @@ def __init__(self, name, cli=None, endpoint=None, model=None, role="researcher", role, headers, ) + self.timeout_seconds = timeout_seconds class FakeRoster: @@ -87,3 +90,65 @@ def fake_run_agent(cli, prompt, **kwargs): backend = llm.resolve_backend(roster) assert backend.complete([{"role": "user", "content": "q"}]) == "answer" assert captured["env"] == {"ANTHROPIC_BASE_URL": "https://api.example.com/anthropic"} + + +def test_resolve_cli_backend_for_oracle_researcher(monkeypatch): + # The whole oracle lane rests on this: a researcher declaring cli="oracle" + # must reach CliBackend with its model intact and no research/ changes. + r = FakeRoster([FakeAgent("scribe", cli="oracle", model="gemini-3.1-pro", role="researcher")]) + captured = {} + + def fake_run_cli(cli, prompt, timeout, model=None, env=None): + captured["cli"] = cli + captured["model"] = model + return "report" + + monkeypatch.setattr(llm, "_run_cli", fake_run_cli) + backend = llm.resolve_backend(r) + assert backend.complete([{"role": "user", "content": "hi"}]) == "report" + assert captured == {"cli": "oracle", "model": "gemini-3.1-pro"} + + +def test_cli_backend_floors_short_engine_timeouts(monkeypatch): + # engine.py asks for timeout=30 on the planning call. A browser seat needs + # far longer, so the roster's timeout_seconds acts as a floor. + r = FakeRoster([FakeAgent("scribe", cli="oracle", model="gemini-3.1-pro", timeout_seconds=300)]) + captured = {} + + def fake_run_cli(cli, prompt, timeout, model=None, env=None): + captured["timeout"] = timeout + return "report" + + monkeypatch.setattr(llm, "_run_cli", fake_run_cli) + backend = llm.resolve_backend(r) + backend.complete([{"role": "user", "content": "hi"}], timeout=30) + assert captured["timeout"] == 300 + + +def test_cli_backend_never_lowers_a_generous_timeout(monkeypatch): + r = FakeRoster([FakeAgent("scribe", cli="oracle", timeout_seconds=120)]) + captured = {} + + def fake_run_cli(cli, prompt, timeout, model=None, env=None): + captured["timeout"] = timeout + return "report" + + monkeypatch.setattr(llm, "_run_cli", fake_run_cli) + backend = llm.resolve_backend(r) + backend.complete([{"role": "user", "content": "hi"}], timeout=180) + assert captured["timeout"] == 180 + + +def test_cli_backend_without_a_floor_is_unchanged(monkeypatch): + # Existing seats declare no timeout_seconds and must keep engine timings. + r = FakeRoster([FakeAgent("chef", cli="codex")]) + captured = {} + + def fake_run_cli(cli, prompt, timeout, model=None, env=None): + captured["timeout"] = timeout + return "ok" + + monkeypatch.setattr(llm, "_run_cli", fake_run_cli) + backend = llm.resolve_backend(r) + backend.complete([{"role": "user", "content": "hi"}], timeout=30) + assert captured["timeout"] == 30 diff --git a/tests/test_roster.py b/tests/test_roster.py index 6130c259..0003a0a2 100644 --- a/tests/test_roster.py +++ b/tests/test_roster.py @@ -1405,3 +1405,18 @@ def test_load_rejects_invalid_scheduler_limit(tmp_path): text = VALID.replace("[limits]\n", '[limits]\nscheduler = "ready-queue"\n') with pytest.raises(ValueError, match="limits.scheduler"): roster_mod.load_roster(_write(tmp_path, text)) + + +def test_load_accepts_oracle_researcher(tmp_path): + # Clears both gates: agent_adapters.is_known (from _ADAPTERS) and the + # limits.allow_models allowlist the VALID fixture declares. The role is + # retargeted too, so find_role("researcher") -- the lookup resolve_backend + # actually performs -- returns this seat rather than nothing. + text = ( + VALID.replace('cli = "ollama:llama3.3"', 'cli = "oracle"') + .replace('role = "write code"', 'role = "researcher"') + .replace('allow_models = ["codex", "ollama:*"]', 'allow_models = ["codex", "oracle"]') + ) + loaded = roster_mod.load_roster(_write(tmp_path, text)) + assert loaded.agents["coder"].cli == "oracle" + assert loaded.find_role("researcher").cli == "oracle"