Skip to content

fix: distinguish requested operations in tool routing - #49

Merged
BunsDev merged 3 commits into
mainfrom
perf/routing-operation-evidence
Sep 25, 2026
Merged

BunsDev merged 3 commits into
mainfrom
perf/routing-operation-evidence

Conversation

@BunsDev

@BunsDev BunsDev commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

What changed

Clear explanation requests could stop below the routing confidence floor, while broader useful-step wording made preliminary reads compete with requested edits. Routing question set v4 distinguishes the requested operation from its prerequisite read. The deterministic inspector descriptor and response note now describe its actual source-context behavior.

Preserve exact v1/v2/v3 wire bodies and matching catalog replay, reject unsupported versions, and advance new Arena runs to setup 5 while retaining historical runs. Proposal-review v4, fixture host revision 1, model pins, policies, thresholds, synthetic tasks and labels stay fixed.

The complete measured comparison retains a current-main control and two frozen v4 batches, each with 57 paired cases. The routed legacy score moves from 45/57 to 54/57 and 55/57. Both v4 batches deliver rubric-meeting grounded outputs on all 39 clear tasks, including all nine inspections, versus 31 clear-task answers in the control. All 342 observations, blinded annotations, exact request/source provenance, completion records and paired usage are retained.

Five of the 114 v4 routing cases remain unavailable because provider distributions sum to 0.99; no normalization, threshold change or harness retry hides them. Appropriate route-only clarification remains unassessable for answer quality. These repeated synthetic development runs establish neither calibration nor full saturation. Lower paired input alongside higher output and different cache reuse does not establish dollar savings; overlapping batches do not isolate latency.

Verdict impact

None for proposal-review fixtures or decision-table rows. Routing semantics move to question set 4, which can change selected tools and holds. No verdict, permission or execution authority is added. Proposed source is never applied or executed.

Verification

  • Frozen dependency install; pnpm typecheck; pnpm test (267 tests); pnpm build.
  • Signed commits, repository secret guard and staged gitleaks pass.
  • Separate implementation spec/quality review and independent full evidence audit pass.
  • Exact integration request parity, historical receipt replay and eight portable blinded-input replay checks pass.
  • Sixteen fake-provider browser checks cover responsive tool menus, history and export; no human accessibility acceptance claimed.
  • Manual synthetic live benchmarks are separate from automated tests; no credentials or proposed-code execution added.

Exact-head CI and final review must pass before squash merge.

Preserve versioned routing bodies and historical catalog replay while retaining fixture host revision 1, proposal-review v4, and routing development evidence. New Arena runs use setup revision 5.

Verified frozen install, typecheck, 264 offline tests, secret scan, and production build. All 57 candidate request bodies match the measured artifact; 13 historical experiment artifacts replay unchanged. Full CLI measurement of the integrated candidate remains pending. Proposal-review verdicts do not change.
@vercel

vercel Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
jev-harness Ready Ready Preview Sep 25, 2026 9:29am UTC

Request Review

@BunsDev BunsDev self-assigned this Sep 25, 2026
@BunsDev
BunsDev marked this pull request as ready for review September 25, 2026 09:30
Copilot AI lite review requested due to automatic review settings September 25, 2026 09:30

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

The broad routing, Arena compatibility, and evaluation changes warrant final human review.

Review effort: Lite
Findings: None

What changed in this PR

Updates tool routing to question-set v4 while preserving v1–v3 compatibility, advancing Arena setup handling, and documenting synthetic evaluation evidence.

Changes:

  • Adds version-aware routing serialization, replay, validation, and tests.
  • Updates inspector behavior and Arena history/setup compatibility.
  • Adds benchmark, provenance, audit, and documentation artifacts.
File Summary
tests/​routing-versions.test.ts Tests version compatibility and replay.
tests/​routing-experiment.test.ts Verifies v4 artifact metadata.
tests/​routing-evidence-replay.test.ts Adds v4 evidence replay coverage.
tests/​fixtures/​routing-v3-prerequisites.json Preserves v3 fixture evidence.
tests/​fixtures/​routing-v2-prerequisites.json Preserves v2 fixture evidence.
tests/​fixtures/​routing-v1-prerequisites.json Preserves v1 fixture evidence.
tests/​arena.test.ts Tests inspector grounding output.
tests/​arena-history.test.ts Tests versioned receipts and setup filtering.
src/​routing/​types.ts Defines supported routing versions.
src/​routing/​catalog.ts Advances the current routing version to v4.
scripts/​verify-arena-browser.mjs Verifies setup revision 5.
scripts/​arena-mcp.mjs Updates inspector grounding text.
README.md Documents v4 evaluation and limitations.
examples/​routing/​scenarios.ts Defines historical and current catalogs.
examples/​routing/​runs/​2026-09-25-routing-package-v4-integrated-repeat-provenance.json Records repeat-run provenance.
examples/​routing/​runs/​2026-09-25-routing-package-v4-integrated-repeat-completion.json Records repeat-run completion.
examples/​routing/​runs/​2026-09-25-routing-package-v4-integrated-provenance.json Records integrated-run provenance.
examples/​routing/​runs/​2026-09-25-routing-package-v4-integrated-completion.json Records integrated-run completion.
examples/​routing/​runs/​2026-09-25-routing-package-v4-control-provenance.json Records control provenance.
examples/​routing/​runs/​2026-09-25-routing-package-v4-control-completion.json Records control completion.
examples/​routing/​experiment.ts Adds version-aware artifact replay.
examples/​routing/​experiment-tasks.ts Defines historical catalog mappings.
examples/​host/​jev-choice.ts Serializes and rejects unsupported versions.
examples/​arena/​history.ts Supports receipt versions and setup 5.
docs/​superpowers/​plans/​2026-09-24-benchmark-optimization.md Records v4 experiment status.
docs/​routing.md Documents routing semantics and replay.
docs/​routing-evaluation/​runs/​2026-09-25-routing-v4-integrated-cli-smoke.json Records candidate preflight.
docs/​routing-evaluation/​runs/​2026-09-25-routing-v4-full-repeat-plan.json Records the repeat protocol.
docs/​routing-evaluation/​runs/​2026-09-25-routing-v4-control-cli-smoke.json Records control preflight.
docs/​routing-evaluation/​runs/​2026-09-25-routing-package-v4-integrated-repeat-audit.json Audits repeat artifacts.
docs/​routing-evaluation/​runs/​2026-09-25-routing-package-v4-integrated-audit.json Audits candidate artifacts.
docs/​routing-evaluation/​runs/​2026-09-25-routing-package-v4-control-audit.json Audits control artifacts.
docs/​routing-evaluation/​runs/​2026-09-25-routing-evidence-v4-confirmation-provenance.json Preserves confirmation provenance.
docs/​routing-evaluation/​runs/​2026-09-25-routing-evidence-v4-attempt3-provenance.json Preserves diagnostic provenance.
docs/​routing-evaluation/​runs/​2026-09-25-jev-routing-v4-package-original.mts Archives the package runner.
docs/​routing-evaluation/​runs/​2026-09-25-jev-routing-v4-integration-audit.ts Replays integration evidence.
docs/​routing-evaluation/​runs/​2026-09-25-jev-routing-v4-integration-audit.json Records replay audit results.
docs/​routing-evaluation/​runs/​2026-09-25-blind-G-assessment-audit.json Audits repeat assessment.
docs/​routing-evaluation/​runs/​2026-09-25-blind-F-assessment-audit.json Audits candidate assessment.
docs/​routing-evaluation/​runs/​2026-09-25-blind-E-assessment-audit.json Audits control assessment.
docs/​routing-evaluation/​2026-09-25-routing-v4.md Documents measurements and limitations.
docs/​routing-evaluation/​2026-09-25-development.md Updates the experiment ledger.
docs/​roadmap.md Records completed v4 milestones.
docs/​architecture.md Updates routing architecture and provenance.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@BunsDev
BunsDev merged commit dee6b57 into main Sep 25, 2026
4 checks passed
@BunsDev
BunsDev deleted the perf/routing-operation-evidence branch September 25, 2026 09:35

This branch was successfully deployed

1 active deployment
Preview — 86bb5b01 Deployed Sep 25, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants