Repository navigation
fix: distinguish requested operations in tool routing - #49
Merged
Merged
Conversation
Preserve versioned routing bodies and historical catalog replay while retaining fixture host revision 1, proposal-review v4, and routing development evidence. New Arena runs use setup revision 5. Verified frozen install, typecheck, 264 offline tests, secret scan, and production build. All 57 candidate request bodies match the measured artifact; 13 historical experiment artifacts replay unchanged. Full CLI measurement of the integrated candidate remains pending. Proposal-review verdicts do not change.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
The broad routing, Arena compatibility, and evaluation changes warrant final human review.
Review effort: Lite
Findings: None
What changed in this PR
Updates tool routing to question-set v4 while preserving v1–v3 compatibility, advancing Arena setup handling, and documenting synthetic evaluation evidence.
Changes:
- Adds version-aware routing serialization, replay, validation, and tests.
- Updates inspector behavior and Arena history/setup compatibility.
- Adds benchmark, provenance, audit, and documentation artifacts.
| File | Summary |
|---|---|
tests/routing-versions.test.ts |
Tests version compatibility and replay. |
tests/routing-experiment.test.ts |
Verifies v4 artifact metadata. |
tests/routing-evidence-replay.test.ts |
Adds v4 evidence replay coverage. |
tests/fixtures/routing-v3-prerequisites.json |
Preserves v3 fixture evidence. |
tests/fixtures/routing-v2-prerequisites.json |
Preserves v2 fixture evidence. |
tests/fixtures/routing-v1-prerequisites.json |
Preserves v1 fixture evidence. |
tests/arena.test.ts |
Tests inspector grounding output. |
tests/arena-history.test.ts |
Tests versioned receipts and setup filtering. |
src/routing/types.ts |
Defines supported routing versions. |
src/routing/catalog.ts |
Advances the current routing version to v4. |
scripts/verify-arena-browser.mjs |
Verifies setup revision 5. |
scripts/arena-mcp.mjs |
Updates inspector grounding text. |
README.md |
Documents v4 evaluation and limitations. |
examples/routing/scenarios.ts |
Defines historical and current catalogs. |
examples/routing/runs/2026-09-25-routing-package-v4-integrated-repeat-provenance.json |
Records repeat-run provenance. |
examples/routing/runs/2026-09-25-routing-package-v4-integrated-repeat-completion.json |
Records repeat-run completion. |
examples/routing/runs/2026-09-25-routing-package-v4-integrated-provenance.json |
Records integrated-run provenance. |
examples/routing/runs/2026-09-25-routing-package-v4-integrated-completion.json |
Records integrated-run completion. |
examples/routing/runs/2026-09-25-routing-package-v4-control-provenance.json |
Records control provenance. |
examples/routing/runs/2026-09-25-routing-package-v4-control-completion.json |
Records control completion. |
examples/routing/experiment.ts |
Adds version-aware artifact replay. |
examples/routing/experiment-tasks.ts |
Defines historical catalog mappings. |
examples/host/jev-choice.ts |
Serializes and rejects unsupported versions. |
examples/arena/history.ts |
Supports receipt versions and setup 5. |
docs/superpowers/plans/2026-09-24-benchmark-optimization.md |
Records v4 experiment status. |
docs/routing.md |
Documents routing semantics and replay. |
docs/routing-evaluation/runs/2026-09-25-routing-v4-integrated-cli-smoke.json |
Records candidate preflight. |
docs/routing-evaluation/runs/2026-09-25-routing-v4-full-repeat-plan.json |
Records the repeat protocol. |
docs/routing-evaluation/runs/2026-09-25-routing-v4-control-cli-smoke.json |
Records control preflight. |
docs/routing-evaluation/runs/2026-09-25-routing-package-v4-integrated-repeat-audit.json |
Audits repeat artifacts. |
docs/routing-evaluation/runs/2026-09-25-routing-package-v4-integrated-audit.json |
Audits candidate artifacts. |
docs/routing-evaluation/runs/2026-09-25-routing-package-v4-control-audit.json |
Audits control artifacts. |
docs/routing-evaluation/runs/2026-09-25-routing-evidence-v4-confirmation-provenance.json |
Preserves confirmation provenance. |
docs/routing-evaluation/runs/2026-09-25-routing-evidence-v4-attempt3-provenance.json |
Preserves diagnostic provenance. |
docs/routing-evaluation/runs/2026-09-25-jev-routing-v4-package-original.mts |
Archives the package runner. |
docs/routing-evaluation/runs/2026-09-25-jev-routing-v4-integration-audit.ts |
Replays integration evidence. |
docs/routing-evaluation/runs/2026-09-25-jev-routing-v4-integration-audit.json |
Records replay audit results. |
docs/routing-evaluation/runs/2026-09-25-blind-G-assessment-audit.json |
Audits repeat assessment. |
docs/routing-evaluation/runs/2026-09-25-blind-F-assessment-audit.json |
Audits candidate assessment. |
docs/routing-evaluation/runs/2026-09-25-blind-E-assessment-audit.json |
Audits control assessment. |
docs/routing-evaluation/2026-09-25-routing-v4.md |
Documents measurements and limitations. |
docs/routing-evaluation/2026-09-25-development.md |
Updates the experiment ledger. |
docs/roadmap.md |
Records completed v4 milestones. |
docs/architecture.md |
Updates routing architecture and provenance. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Clear explanation requests could stop below the routing confidence floor, while broader useful-step wording made preliminary reads compete with requested edits. Routing question set v4 distinguishes the requested operation from its prerequisite read. The deterministic inspector descriptor and response note now describe its actual source-context behavior.
Preserve exact v1/v2/v3 wire bodies and matching catalog replay, reject unsupported versions, and advance new Arena runs to setup 5 while retaining historical runs. Proposal-review v4, fixture host revision 1, model pins, policies, thresholds, synthetic tasks and labels stay fixed.
The complete measured comparison retains a current-main control and two frozen v4 batches, each with 57 paired cases. The routed legacy score moves from 45/57 to 54/57 and 55/57. Both v4 batches deliver rubric-meeting grounded outputs on all 39 clear tasks, including all nine inspections, versus 31 clear-task answers in the control. All 342 observations, blinded annotations, exact request/source provenance, completion records and paired usage are retained.
Five of the 114 v4 routing cases remain unavailable because provider distributions sum to 0.99; no normalization, threshold change or harness retry hides them. Appropriate route-only clarification remains unassessable for answer quality. These repeated synthetic development runs establish neither calibration nor full saturation. Lower paired input alongside higher output and different cache reuse does not establish dollar savings; overlapping batches do not isolate latency.
Verdict impact
None for proposal-review fixtures or decision-table rows. Routing semantics move to question set 4, which can change selected tools and holds. No verdict, permission or execution authority is added. Proposed source is never applied or executed.
Verification
pnpm typecheck;pnpm test(267 tests);pnpm build.Exact-head CI and final review must pass before squash merge.