bench(tau2): add category-aware trajectory memory rerank

[huangruiteng]

Description

Follow-up to #2017. This PR adds an opt-in TAU-2 benchmark treatment for category-aware reranking on top of OpenViking trajectory-view memory.

It keeps the benchmark path OpenViking-native:

• train with train_memory_mode: experience_only;
• retrieve from memories/trajectories;
• add sidecar-based query / memory category annotations;
• before write-like tool calls, retrieve 6 candidates, category-rerank them, and inject at most 2 memories;
• emit runtime trace summaries and mark cells diagnostic when category/runtime evidence is missing.

The benchmark trains/evaluates its own OpenViking corpus through the config; it does not require a pre-existing external corpus.

Status

Local TAU-2 evidence is from reasoning-high, fixed-first-user, 8-repeat runs. The headline is that trajectory-view + category rerank gives a clear airline lift, while the no-category pre-write route is currently stronger on retail.

Route	retail avg	airline avg	domain avg	Read
No memory baseline	0.84688	0.75000	0.79844	Current paired OV-native baseline
Trajectory-view + pre-write/scope	0.85938	0.74375	0.80157	Best current retail route
Trajectory-view + pre-write + category rerank	0.83437	0.81250	0.82344	Clearest category positive signal, mainly airline
Best by domain	0.85938	0.81250	0.83594	Retail uses pre-write/scope; airline uses category rerank

Ablation context:

Treatment	retail avg	airline avg	aggregate	Read
Old trajectory prompt -> experiences, first-user only	0.83125	0.66875	0.77708 task-weighted	Negative control; not positive evidence
PR-B trajectory prompt, first-user only	0.82182	0.76875	0.79528 domain avg	Prompt route alone is close to baseline but not enough
PR-B trajectory prompt + pre-write/scope	0.85938	0.74375	0.80157 domain avg	Retail benefits from pre-write/scope
PR-C category rerank	0.83437	0.81250	0.82344 domain avg	Airline benefits from category selection

This PR is still a draft because category-sidecar generation and the final product boundary are intentionally kept under review. The benchmark code path itself is now scoped to the OpenViking-native trajectory route.

Type of Change

• Bug fix
• New feature / benchmark treatment
• Breaking change
• Documentation / benchmark tooling

Verification

• .venv/bin/python -m pytest -o addopts= tests/benchmark/test_tau2_category_rerank.py
• python3 benchmark/tau2/scripts/run_eval.py --config benchmark/tau2/config/category_rerank.yaml --plan-only --run-id prc_cleanup_plan_check2
• python3 -m py_compile benchmark/tau2/scripts/category_rerank.py benchmark/tau2/scripts/run_eval.py benchmark/tau2/scripts/run_memory_v2_eval.py benchmark/tau2/scripts/build_category_catalog.py benchmark/tau2/scripts/build_category_requests.py benchmark/tau2/scripts/generate_category_annotations.py benchmark/tau2/scripts/run_category_annotation_batches.py

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

**🎫 Ticket compliance analysis **

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪

🏅 Score: 85

🧪 PR contains tests

🔒 No security concerns identified

✅ No TODO sections

🔀 No multiple PR themes

⚡ No major issues detected

[github-actions]

Failed to generate code suggestions for PR

[huangruiteng]

@qin-ctx 我按当前 main 复核了这个既有 draft，建议作为陈旧实验关闭、保留 PR 正文中的结果作历史参考，而不是继续大 rebase：

• 当前 main 已领先该分支 678 个提交；PR 自身含 53 个提交、19 个文件、约 5,643 行新增。
• 原 benchmark/tau2/{config,scripts} 已重构为 benchmark/tau2/{llm,train,vikingbot}，旧路径在 current main 不再存在。
• 其基础能力 feat(memory): upgrade trajectory extraction to beat no-memory baseline #2017 已合入，后续经验记忆框架重构 Feat/自进化（经验记忆）框架重构 #2503 也已合入；若仍要验证 category-aware rerank，应基于当前 benchmark/experience contract 重新设计，不能把旧 sidecar/runtime 路径直接搬回。
• 该 PR 自 2026-05-15 后没有新的人工 review，仍为 Draft、CONFLICTING，旧 CI 失败也不再能代表 current-main 可合并性。

因此我不会在这个大分支上做高风险历史重放，也不会新开 issue/PR。请确认是否按上述判断关闭 #2079；如维护者认为其中某个最小实验资产仍应保留，我会只在这条既有 PR 内按明确范围收敛。

[huangruiteng]

Closing this stale benchmark experiment to reduce noise in the OpenViking review queue. The branch remains available if the experiment is resumed with current evidence.