Repository navigation
feat: recover sum-invalid routing responses with complete attempt accounting - #51
Merged
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Resolve the inconsistent offline-test count in the frozen-source verification artifact.
Review effort: Lite
Findings: 1
Open (1)
What changed in this PR
Adds opt-in host recovery for sum-invalid routing responses, with bounded retries and complete physical-attempt accounting across the host, CLI, Arena, history, and evaluation artifacts.
Changes:
- Retries only sum-invalid responses, capped at three requests under one deadline.
- Records attempt ledgers, usage totals, and partial-stream state.
- Updates UI, history, CLI, documentation, tests, and retained evidence.
- Moderate finding: the frozen-source gate reports 293 offline tests while verification reports 295; the artifact needs regeneration or clarification.
| File | Reviewed scope |
|---|---|
tests/usage.test.ts |
Usage summary expectations |
tests/routing-versions.test.ts |
Routing version coverage |
tests/routing-recovery-integration.test.ts |
Recovery integration and persistence |
tests/routing-evidence-replay.test.ts |
Retained evidence replay |
tests/arena-history.test.ts |
Arena history behavior |
scripts/verify-arena-browser.mjs |
Arena setup verification |
README.md |
Recovery usage documentation |
examples/routing/usage.ts |
Logical and physical usage summaries |
examples/routing/live-client.ts |
Ledger telemetry parsing |
examples/routing/host-policy.ts |
Demo recovery policy |
examples/routing/experiment-table.ts |
Attempt and retry reporting |
examples/routing/experiment-cli.ts |
Recovery CLI option |
examples/host/runtime.ts |
Host runtime integration |
examples/host/live.ts |
Recovery configuration and admission limits |
examples/host/jev-choice.ts |
Bounded recovery and attempt ledgers |
examples/arena/history.ts |
Transport provenance persistence |
docs/superpowers/plans/2026-09-26-routing-host-recovery.md |
Implementation and verification plan |
docs/routing.md |
Recovery and accounting rules |
docs/routing-evaluation/runs/2026-09-26-recovery-wrapper-verification.json |
Wrapper verification artifact |
docs/routing-evaluation/runs/2026-09-26-recovery-wrapper-probes.py |
Wrapper probes |
docs/routing-evaluation/runs/2026-09-26-recovery-wrapper-probe-results.json |
Wrapper probe results |
docs/routing-evaluation/runs/2026-09-26-recovery-real-main-preflight-result.json |
Main preflight result |
docs/routing-evaluation/runs/2026-09-26-recovery-preflight-archive.json |
Preflight archive |
docs/routing-evaluation/runs/2026-09-26-recovery-package-probes.py |
Package probes |
docs/routing-evaluation/runs/2026-09-26-recovery-package-original.mts |
Package source snapshot |
docs/routing-evaluation/runs/2026-09-26-recovery-live-plan.json |
Live evaluation plan |
docs/routing-evaluation/runs/2026-09-26-recovery-full-audit-probe-results.json |
Full audit results |
docs/routing-evaluation/runs/2026-09-26-recovery-frozen-source-gates.json |
Frozen-source verification gates |
docs/routing-evaluation/runs/2026-09-26-recovery-diagnostic-original.mts |
Diagnostic source snapshot |
docs/routing-evaluation/runs/2026-09-26-recovery-diagnostic-audit-probes-green.json |
Diagnostic audit results |
docs/routing-evaluation/runs/2026-09-26-recovery-diagnostic-audit-original.mts |
Diagnostic audit source |
docs/routing-evaluation/runs/2026-09-26-recovery-cancellation-red.json |
Cancellation failure case |
docs/routing-evaluation/runs/2026-09-26-recovery-cancellation-probes.py |
Cancellation probes |
docs/routing-evaluation/runs/2026-09-26-recovery-cancellation-green.json |
Cancellation verification |
docs/routing-evaluation/runs/2026-09-26-recovery-browser-result.json |
Browser verification result |
docs/routing-evaluation/runs/2026-09-26-recovery-browser-original.mts |
Browser verification source |
docs/routing-evaluation/runs/2026-09-26-blind-I-assessment-audit.json |
Blind assessment audit |
docs/routing-evaluation/runs/2026-09-26-blind-H-assessment-audit-v2-regression.json |
Blind regression audit |
docs/routing-evaluation/runs/2026-09-26-blind-assessment-audit-v2-probes.py |
Assessment audit probes |
docs/routing-evaluation/runs/2026-09-26-blind-assessment-audit-v2-probe-results.json |
Assessment probe results |
docs/routing-evaluation/runs/2026-09-26-audit-blind-assessments-v2.py |
Blind assessment audit script |
docs/routing-evaluation/2026-09-26-host-recovery.md |
Retained evaluation report |
docs/routing-demo.md |
Demo behavior and UI documentation |
docs/roadmap.md |
Recovery milestone documentation |
docs/architecture.md |
Host-only recovery architecture |
components/routing-attempts.tsx |
Attempt telemetry cards |
components/header.tsx |
Usage dashboard accounting |
components/arena.tsx |
Streamed routing telemetry |
components/arena-results.tsx |
Routing overhead and attempts |
components/arena-lessons.tsx |
Attempt measurement descriptions |
components/arena-history.tsx |
Recovery cohort separation |
app/globals.css |
Attempt-card styling |
app/api/route/route.ts |
Route-handler compatibility |
app/api/arena/route.ts |
Partial routing ledger streaming |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

What changed
A routing response with an invalid probability total previously ended as unavailable. The example host can now retry that sole structural defect with an identical request, capped at three physical requests under one deadline. All valid answers and every other error stop recovery. The Arena shows labeled attempt cards, complete usage totals, and partial ledgers when streams end early; history, downloads, and benchmark artifacts retain the same accounting.
The demo uses
probability_sum_only_v1(Arena setup 7). The reusable adapter and experiment CLI still default to one request; the CLI requires--live --sum-recovery. Pure routing, proposal review, pinned models, question semantics, fixtures, prerequisites, and CLI isolation are unchanged.Measured results
The retained report links the predeclared protocol, every live artifact, source/package audits, frozen blinded grades, and exact-reference checks:
These are adaptive measurements on the fixed synthetic suite, not held-out calibration. No proposed source was applied or executed.
Verdict impact
None for proposal review. The decision table, questions, threshold, fixtures, and receipt verdicts are unchanged. Routing may obtain valid evidence after a sole sum defect; exhausted recovery remains unavailable. No probability normalization, execution authority, or fallback tool is added. Bounded additional provider requests are introduced only in the optional host.
Verification
pnpm typecheckpnpm test— 295 delivery tests: 293 at the frozen runtime plus two post-measurement evidence-replay checkspnpm build, secret checks, andgit diff --checkCredentials remain outside receipts and model state. Human keyboard-only and VoiceOver acceptance was not performed.