Skip to content

fix: clarify desired outcomes before selecting tools - #50

Merged
BunsDev merged 4 commits into
mainfrom
perf/routing-clarification-delivery
Sep 26, 2026
Merged

BunsDev merged 4 commits into
mainfrom
perf/routing-clarification-delivery

Conversation

@BunsDev

@BunsDev BunsDev commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

What changed

Routing question set v5 clarifies that naming a target does not specify the desired outcome. Ambiguous improvement requests require clarification instead of an invented edit. It preserves historical v1–v4 wire instructions, catalog, model pin, thresholds, strict validation, tasks, labels and prerequisites. Arena setup 6 separates the new cohort.

The development report retains two routing-only batches (57/57 each) and the full CLI comparison (56/57). One full-run distribution sums to 0.99 and remains unavailable. No automatic retry or normalization is added; this is an incremental measured improvement, not full saturation.

All 96 dispatched CLI turns completed. Blinded assessment found all 39 delivered Jev-routed outputs meet the frozen rubric; 18 absent outputs remain unassessable. The all-tools arm has 54 meets, one partial and two failures. On 39 matched grounded outputs, mean reported input is 36,265 all-tools versus 34,838 routed, including Jev. These repeated synthetic development cases do not establish calibration, dollar savings or isolated speed improvements.

The wire diagnostic and recovery record retains every failed attempt and added request; recovery remains experimental and unintegrated.

Verdict impact

None for proposal-review verdicts or offline fixture labels. The pure routing policy is unchanged. The full v5 observed routing score is 56/57, following 54/57 and 55/57 on full v4 batches. Every clear-task routed output still meets the frozen rubric. Correct routing clarification remains separate from an authored answer.

Verification

  • pnpm typecheck
  • pnpm test — 269 passed, including exact relocated evidence replay for all eight retained batches.
  • pnpm build
  • Signed commits, secret checks and pre-commit gitleaks.
  • No new provider integration, credential, execution path or verdict; live calls use the existing explicitly enabled synthetic benchmark.
  • Architecture, README, tests and question-set version updated.
  • Independent source and historical compatibility review.
  • Full artifact replay: 114 rows, 57 routing/prerequisite decisions, 82 source files and 42 pinned CLI files verified.
  • Blinded assessment integrity: all 114 case IDs, exact evidence excerpts and source/input/rubric hashes checked; original annotations and joined usage retained.

Frozen runtime source: fda2774667872968e1ef48271faa040e435ffdfc. Later delivery changes are retained evidence, documentation and an offline replay test. No proposed source is applied or executed.

@vercel

vercel Bot commented Sep 26, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
jev-harness Ready Ready Preview Sep 26, 2026 8:07am UTC

Request Review

@BunsDev
BunsDev marked this pull request as ready for review September 26, 2026 08:02
Copilot AI lite review requested due to automatic review settings September 26, 2026 08:02

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Reconcile the published test count and align the diagnostic runner’s accepted --attempts value with its implementation.

Review effort: Lite
Findings: 1 Low severity

Open (1)
What changed in this PR

Updates routing question set v5 to clarify desired outcomes while preserving historical compatibility and strict validation.

Changes:

  • Adds v5 routing support and Arena setup 6.
  • Expands compatibility, replay, and evidence tests.
  • Adds documentation, diagnostics, and benchmark provenance artifacts.
File Change
tests/​routing-versions.test.ts Tests v5 versioning and serialization.
tests/​routing-experiment.test.ts Updates v5 artifact expectations.
tests/​routing-evidence-replay.test.ts Adds v5 evidence replay coverage.
tests/​arena-history.test.ts Tests setup 6 compatibility.
src/​routing/​types.ts Adds routing version 5.
src/​routing/​catalog.ts Makes v5 current.
README.md Documents the v5 experiment.
examples/​routing/​runs/​2026-09-26-routing-package-v5-integrated-provenance.json Stores v5 provenance.
examples/​routing/​runs/​2026-09-26-routing-package-v5-integrated-completion.json Records completion status.
examples/​routing/​experiment.ts Accepts v5 artifacts.
examples/​routing/​experiment-tasks.ts Maps v5 to the catalog.
examples/​host/​jev-choice.ts Serializes v5 instructions.
examples/​arena/​history.ts Supports v5 receipts and setup 6.
docs/​superpowers/​plans/​2026-09-26-routing-clarification-v5.md Records the implementation plan.
docs/​routing.md Documents v5 semantics and compatibility.
docs/​routing-evaluation/​runs/​2026-09-26-routing-v5-summary.json Stores v5 routing summaries.
docs/​routing-evaluation/​runs/​2026-09-26-routing-v5-diagnostic.mts Provides the v5 diagnostic runner.
docs/​routing-evaluation/​runs/​2026-09-26-jev-routing-v5-package-original.mts Records CLI provenance.
docs/​routing-evaluation/​runs/​2026-09-26-distribution-wire-v5-single-provenance.json Stores initial v5 wire provenance.
docs/​routing-evaluation/​runs/​2026-09-26-distribution-wire-v5-repeat-provenance.json Stores repeated v5 wire provenance.
docs/​routing-evaluation/​runs/​2026-09-26-distribution-wire-single-provenance.json Retains diagnostic provenance.
docs/​routing-evaluation/​runs/​2026-09-26-distribution-wire-funded-single-provenance.json Stores funded diagnostic provenance.
docs/​routing-evaluation/​runs/​2026-09-26-distribution-wire-funded-recovery-repeat-provenance.json Stores repeated recovery provenance.
docs/​routing-evaluation/​runs/​2026-09-26-distribution-wire-funded-recovery-provenance.json Stores recovery provenance.
docs/​routing-evaluation/​runs/​2026-09-26-distribution-summary.json Stores diagnostic accounting.
docs/​routing-evaluation/​runs/​2026-09-26-distribution-diagnostic-v3.mts Retains recovery diagnostics.
docs/​routing-evaluation/​runs/​2026-09-26-distribution-diagnostic-v2.mts Retains earlier diagnostics.
docs/​routing-evaluation/​runs/​2026-09-26-blind-H-assessment-audit.json Records assessment integrity.
docs/​routing-evaluation/​2026-09-26-routing-v5.md Reports v5 measurements.
docs/​routing-evaluation/​2026-09-26-distribution-diagnostic.md Documents distribution diagnostics.
docs/​roadmap.md Records v5 evaluation completion.
docs/​architecture.md Records the v5 architecture change.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@BunsDev
BunsDev merged commit 48fbf7f into main Sep 26, 2026
3 checks passed
@BunsDev
BunsDev deleted the perf/routing-clarification-delivery branch September 26, 2026 08:08

This branch was successfully deployed

1 active deployment
Preview — 7f776667 Deployed Sep 26, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants