Deepscholar-Bench

[donovanr]

Description

Adds DeepScholar-Bench as a sandboxed external evaluation (deepscholar_bench). The benchmark retrieves prior academic work, generates a related-work section, and evaluates it using seven metrics covering organization, coverage, importance, and citation quality.

What changed

• Adds the DeepScholar external-eval implementation, argument handling, result parsing, and registration.
• Pins the upstream DeepScholar and LOTUS revisions used during validation.
• Supports the upstream arXiv retrieval path plus optional Semantic Scholar and Tavily backends.
• Connects local vLLM providers to the benchmark through a generated LOTUS configuration.
• Raises the upstream stage-model token limit and bounds Semantic Scholar retries and wall-clock time.
• Persists generation and evaluation artifacts through bind mounts so partial work survives failures.
• Reports both upstream aggregates and fixed-request-denominator metrics, where missing or failed queries contribute zero.
• Reports query counts and generation/scoring completeness in result metadata.
• Makes sandbox streaming control independent of the in-container swe-rex HTTP server for local Docker/Podman runs.
• Detects container-runtime poll failures using the expected poll marker and aborts after three consecutive failures, while preserving the normal exit code 1 from an in-progress poll.

Validation

Successful full 63-query-range Beaker runs:

• OLMo-3-7B-Instruct: 01KXEDME5H6SPB432WBM7XE5ES
• Qwen3.5-9B: 01KXEVZVCJHQMCK336MDZHD5HX

Local validation:

• uv run pytest -q tests/core/harness/test_sandbox_executor.py tests/evals/external/benchmarks/test_deepscholar_external.py — 18
  passed
• Ruff — passed
• ty — passed
• Post-review Beaker smoke: <add experiment ID>

The streaming regression test verifies both sides of the poll behavior: an in-progress command may return exit code 1 with a valid marker, while a container failure without the marker reaches the existing three-strike abort.

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation update
• Refactoring

Testing

• Full local unit-test suite
• Integration testing on Beaker
• New tests added for new functionality

Checklist

• Code follows the project’s style guidelines
• Self-review performed
• Changes generate no new warnings
• No unpublished dependent changes

[donovanr]

@undfined this is ready for a review when you get a chance

[undfined]

Looks good! A couple considerations.

[undfined]

This fallback makes API-backed providers without a base_url look like local providers. For example, LiteLLMProvider("gpt-4o") gets provider_url = "http://localhost:8000/v1", so _is_local_provider() returns true and the run fails the local sandbox health check before generation. I think that contradicts the comment below that external API models should keep upstream behavior. Can we distinguish “missing base_url” from “local default URL” before calling _is_local_provider()?

[donovanr]

Fixed by deciding locality from the provider's actual base_url before the local fallback, so a provider without one stays external:

base_url = getattr(provider, "base_url", None)
is_local = self._is_local_provider(provider, base_url or "")
provider_url = base_url or "http://localhost:8000/v1"

Now LiteLLMProvider("gpt-4o") is treated as external (agentic search, no health check) while local vLLM still resolves as local. VLLMServerProvider always assigns self.base_url (passed URL or its managed server's), and passing "" rather than None keeps the _server check intact for a managed provider that hasn't set one yet.

Added a regression test covering all three states. Note the same pattern lives in asta/eval.py and tau2/eval.py; leaving those for a separate follow-up to keep this PR scoped and on track to merge quickly.

[undfined]

Nit: _SEC is a bit better

[donovanr]

Renamed to S2_SEARCH_BUDGET_SEC, and the budget_s param to budget_sec for consistency.

[donovanr]

found a small formatting error for one of the sub-benchmarks, holding off on merging until that's added