Proposal: add retrieval metrics — NDCG@k (URL) and Recall@k (LLM-judged, UMBRELA)

[potoos]

The current eval framework grades generation accuracy and latency, but doesn't directly measure retrieval quality — i.e. whether the engine actually surfaced the right sources, independent of the synthesis step. Two metrics that fill this gap on top of the existing per-query pipeline:

1. NDCG@k (URL-binary) — always-on, free

Score each predicted URL list against the dataset's metadata.urls (SimpleQA already ships this) with binary relevance and the canonical TREC NDCG@k formulation: IDCG runs over all gold URLs, so missed gold reduces the score. Adds ndcg_at_5 / ndcg_at_10 columns next to accuracy_score and the latency p50s. No new dependencies, no extra API calls.

2. Recall@k (LLM-judged, UMBRELA) — opt-in library

When ground-truth URLs are unavailable or too sparse (FRAMES, internal benchmarks, etc.), grade retrieved chunks with an LLM-as-a-judge using the canonical UMBRELA qrel_zeroshot_bing prompt + parser (Apache-2.0, castorini/umbrela, basis of the TREC 2024 RAG track). Per-query Recall@k = 1 if any chunk in top-k has graded relevance ≥ threshold (default 2, TREC DL convention), else 0.

Library only — not wired into eval_runner.py so cost is opt-in and providers without extract_chunks support are unaffected. Upstream UMBRELA is vendored (prompt + parser, ~50 LOC) rather than installed because the [cloud] extra pulls ~500MB of unrelated deps and requires Python ≥ 3.12, which would tighten this repo's >=3.10,<3.14 support range.

Implementation shape

• Adds a BaseSampler.extract_chunks(raw_results) hook returning a common {url, title, description, extra_snippets, position} schema. Five providers override (you_search, exa, tavily, parallel, google/SerpAPI). The per-row CSV gains a chunks_json column the LLM judge consumes.
• Plumbs gold_urls through eval_runner.py; eval_results_analyzer.py aggregates the new columns when present.
• Default judge model is configurable via LLM_JUDGE_MODEL in constants.py. UMBRELA's published methodology uses gpt-4o; a smaller alternative (gpt-5-mini) works for routine eval runs.
• URL-extraction regex is anchored on )\n / end-of-string so Wikipedia disambiguation slugs (Bug_(Breaking_Bad)) and similar parenthesised paths stop being silently truncated. Locked in by tests.

Diff

Two semantic commits (URL metric, then LLM-judged library), 13 files, +1111 / -32, branched from main:

https://github.com/Nimbleway/web-search-api-evals-fork/compare/main...feat/url-ndcg-llm-recall-metrics

Tests pass (pytest tests/test_retrieval_metrics.py), and a smoke eval_runner run on SimpleQA n=10 populates the new columns end-to-end.

Question for maintainers

Would a PR along these lines be welcome here, or would you prefer a different shape (e.g. URL-NDCG only, LLM-judge as a separate optional package, different judge default)? Happy to open a draft PR for inline review if there's interest.

[zairahms]

Thanks for opening this discussion. Adding retrieval metrics has been on our roadmap, so we welcome PRs that push us in this direction.

[zairahms]

@potoos please go ahead and create a PR for this in our repo, we will fast follow with a review there.