Repository navigation
fix: clarify desired outcomes before selecting tools - #50
Merged
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Reconcile the published test count and align the diagnostic runner’s accepted --attempts value with its implementation.
Review effort: Lite
Findings: 1
Open (1)
What changed in this PR
Updates routing question set v5 to clarify desired outcomes while preserving historical compatibility and strict validation.
Changes:
- Adds v5 routing support and Arena setup 6.
- Expands compatibility, replay, and evidence tests.
- Adds documentation, diagnostics, and benchmark provenance artifacts.
| File | Change |
|---|---|
tests/routing-versions.test.ts |
Tests v5 versioning and serialization. |
tests/routing-experiment.test.ts |
Updates v5 artifact expectations. |
tests/routing-evidence-replay.test.ts |
Adds v5 evidence replay coverage. |
tests/arena-history.test.ts |
Tests setup 6 compatibility. |
src/routing/types.ts |
Adds routing version 5. |
src/routing/catalog.ts |
Makes v5 current. |
README.md |
Documents the v5 experiment. |
examples/routing/runs/2026-09-26-routing-package-v5-integrated-provenance.json |
Stores v5 provenance. |
examples/routing/runs/2026-09-26-routing-package-v5-integrated-completion.json |
Records completion status. |
examples/routing/experiment.ts |
Accepts v5 artifacts. |
examples/routing/experiment-tasks.ts |
Maps v5 to the catalog. |
examples/host/jev-choice.ts |
Serializes v5 instructions. |
examples/arena/history.ts |
Supports v5 receipts and setup 6. |
docs/superpowers/plans/2026-09-26-routing-clarification-v5.md |
Records the implementation plan. |
docs/routing.md |
Documents v5 semantics and compatibility. |
docs/routing-evaluation/runs/2026-09-26-routing-v5-summary.json |
Stores v5 routing summaries. |
docs/routing-evaluation/runs/2026-09-26-routing-v5-diagnostic.mts |
Provides the v5 diagnostic runner. |
docs/routing-evaluation/runs/2026-09-26-jev-routing-v5-package-original.mts |
Records CLI provenance. |
docs/routing-evaluation/runs/2026-09-26-distribution-wire-v5-single-provenance.json |
Stores initial v5 wire provenance. |
docs/routing-evaluation/runs/2026-09-26-distribution-wire-v5-repeat-provenance.json |
Stores repeated v5 wire provenance. |
docs/routing-evaluation/runs/2026-09-26-distribution-wire-single-provenance.json |
Retains diagnostic provenance. |
docs/routing-evaluation/runs/2026-09-26-distribution-wire-funded-single-provenance.json |
Stores funded diagnostic provenance. |
docs/routing-evaluation/runs/2026-09-26-distribution-wire-funded-recovery-repeat-provenance.json |
Stores repeated recovery provenance. |
docs/routing-evaluation/runs/2026-09-26-distribution-wire-funded-recovery-provenance.json |
Stores recovery provenance. |
docs/routing-evaluation/runs/2026-09-26-distribution-summary.json |
Stores diagnostic accounting. |
docs/routing-evaluation/runs/2026-09-26-distribution-diagnostic-v3.mts |
Retains recovery diagnostics. |
docs/routing-evaluation/runs/2026-09-26-distribution-diagnostic-v2.mts |
Retains earlier diagnostics. |
docs/routing-evaluation/runs/2026-09-26-blind-H-assessment-audit.json |
Records assessment integrity. |
docs/routing-evaluation/2026-09-26-routing-v5.md |
Reports v5 measurements. |
docs/routing-evaluation/2026-09-26-distribution-diagnostic.md |
Documents distribution diagnostics. |
docs/roadmap.md |
Records v5 evaluation completion. |
docs/architecture.md |
Records the v5 architecture change. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

What changed
Routing question set v5 clarifies that naming a target does not specify the desired outcome. Ambiguous improvement requests require clarification instead of an invented edit. It preserves historical v1–v4 wire instructions, catalog, model pin, thresholds, strict validation, tasks, labels and prerequisites. Arena setup 6 separates the new cohort.
The development report retains two routing-only batches (57/57 each) and the full CLI comparison (56/57). One full-run distribution sums to 0.99 and remains unavailable. No automatic retry or normalization is added; this is an incremental measured improvement, not full saturation.
All 96 dispatched CLI turns completed. Blinded assessment found all 39 delivered Jev-routed outputs meet the frozen rubric; 18 absent outputs remain unassessable. The all-tools arm has 54 meets, one partial and two failures. On 39 matched grounded outputs, mean reported input is 36,265 all-tools versus 34,838 routed, including Jev. These repeated synthetic development cases do not establish calibration, dollar savings or isolated speed improvements.
The wire diagnostic and recovery record retains every failed attempt and added request; recovery remains experimental and unintegrated.
Verdict impact
None for proposal-review verdicts or offline fixture labels. The pure routing policy is unchanged. The full v5 observed routing score is 56/57, following 54/57 and 55/57 on full v4 batches. Every clear-task routed output still meets the frozen rubric. Correct routing clarification remains separate from an authored answer.
Verification
pnpm typecheckpnpm test— 269 passed, including exact relocated evidence replay for all eight retained batches.pnpm buildFrozen runtime source:
fda2774667872968e1ef48271faa040e435ffdfc. Later delivery changes are retained evidence, documentation and an offline replay test. No proposed source is applied or executed.