Skip to content

feat: recover sum-invalid routing responses with complete attempt accounting - #51

Merged
BunsDev merged 7 commits into
mainfrom
perf/routing-host-recovery
Sep 26, 2026
Merged

BunsDev merged 7 commits into
mainfrom
perf/routing-host-recovery

Conversation

@BunsDev

@BunsDev BunsDev commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

What changed

A routing response with an invalid probability total previously ended as unavailable. The example host can now retry that sole structural defect with an identical request, capped at three physical requests under one deadline. All valid answers and every other error stop recovery. The Arena shows labeled attempt cards, complete usage totals, and partial ledgers when streams end early; history, downloads, and benchmark artifacts retain the same accounting.

The demo uses probability_sum_only_v1 (Arena setup 7). The reusable adapter and experiment CLI still default to one request; the CLI requires --live --sum-recovery. Pure routing, proposal review, pinned models, question semantics, fixtures, prerequisites, and CLI isolation are unchanged.

Measured results

The retained report links the predeclared protocol, every live artifact, source/package audits, frozen blinded grades, and exact-reference checks:

  • 171/171 expected routing outcomes across two 57-case routing batches and one 57-pair CLI comparison.
  • 39/39 assessable Jev-routed outputs meet the unchanged blinded rubric. All 18 expected routing holds remain unassessable as authored output. Baseline: 56 meets, 1 partial.
  • Every first response was valid; no live recovery request occurred and no recovery benefit is demonstrated. The earlier v5 56/57 result remains retained.
  • Across 39 comparable rubric-meeting pairs, mean input was 36,264 / 34,860 and output 323 / 456 (all tools / Jev including routing). These are token counts, not monetary or causal speed savings.
  • Proposal review remains unchanged at 200/200 retained expected verdicts.

These are adaptive measurements on the fixed synthetic suite, not held-out calibration. No proposed source was applied or executed.

Verdict impact

None for proposal review. The decision table, questions, threshold, fixtures, and receipt verdicts are unchanged. Routing may obtain valid evidence after a sole sum defect; exhausted recovery remains unavailable. No probability normalization, execution authority, or fallback tool is added. Bounded additional provider requests are introduced only in the optional host.

Verification

  • pnpm typecheck
  • pnpm test — 295 delivery tests: 293 at the frozen runtime plus two post-measurement evidence-replay checks
  • pnpm build, secret checks, and git diff --check
  • Signed source freeze; independent specification and code-quality review
  • 29 intercepted browser checks at 320–1920px; desktop/mobile screenshots inspected
  • Fake-transport and fake-CLI preflights, including recovered/exhausted/interrupted accounting; automated tests use no shared credits
  • Fixed live batches, full CLI comparison, unchanged-source and pinned-package audits
  • Frozen blinded output assessment; all 114 exact references matched, with 33 checker fault probes and previous-batch regression coverage

Credentials remain outside receipts and model state. Human keyboard-only and VoiceOver acceptance was not performed.

@vercel

vercel Bot commented Sep 26, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
jev-harness Ready Ready Preview Sep 26, 2026 9:42am UTC

Request Review

@BunsDev
BunsDev marked this pull request as ready for review September 26, 2026 09:35
Copilot AI lite review requested due to automatic review settings September 26, 2026 09:35

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Resolve the inconsistent offline-test count in the frozen-source verification artifact.

Review effort: Lite
Findings: 1 Medium severity

Open (1)
What changed in this PR

Adds opt-in host recovery for sum-invalid routing responses, with bounded retries and complete physical-attempt accounting across the host, CLI, Arena, history, and evaluation artifacts.

Changes:

  • Retries only sum-invalid responses, capped at three requests under one deadline.
  • Records attempt ledgers, usage totals, and partial-stream state.
  • Updates UI, history, CLI, documentation, tests, and retained evidence.
  • Moderate finding: the frozen-source gate reports 293 offline tests while verification reports 295; the artifact needs regeneration or clarification.
File Reviewed scope
tests/​usage.test.ts Usage summary expectations
tests/​routing-versions.test.ts Routing version coverage
tests/​routing-recovery-integration.test.ts Recovery integration and persistence
tests/​routing-evidence-replay.test.ts Retained evidence replay
tests/​arena-history.test.ts Arena history behavior
scripts/​verify-arena-browser.mjs Arena setup verification
README.md Recovery usage documentation
examples/​routing/​usage.ts Logical and physical usage summaries
examples/​routing/​live-client.ts Ledger telemetry parsing
examples/​routing/​host-policy.ts Demo recovery policy
examples/​routing/​experiment-table.ts Attempt and retry reporting
examples/​routing/​experiment-cli.ts Recovery CLI option
examples/​host/​runtime.ts Host runtime integration
examples/​host/​live.ts Recovery configuration and admission limits
examples/​host/​jev-choice.ts Bounded recovery and attempt ledgers
examples/​arena/​history.ts Transport provenance persistence
docs/​superpowers/​plans/​2026-09-26-routing-host-recovery.md Implementation and verification plan
docs/​routing.md Recovery and accounting rules
docs/​routing-evaluation/​runs/​2026-09-26-recovery-wrapper-verification.json Wrapper verification artifact
docs/​routing-evaluation/​runs/​2026-09-26-recovery-wrapper-probes.py Wrapper probes
docs/​routing-evaluation/​runs/​2026-09-26-recovery-wrapper-probe-results.json Wrapper probe results
docs/​routing-evaluation/​runs/​2026-09-26-recovery-real-main-preflight-result.json Main preflight result
docs/​routing-evaluation/​runs/​2026-09-26-recovery-preflight-archive.json Preflight archive
docs/​routing-evaluation/​runs/​2026-09-26-recovery-package-probes.py Package probes
docs/​routing-evaluation/​runs/​2026-09-26-recovery-package-original.mts Package source snapshot
docs/​routing-evaluation/​runs/​2026-09-26-recovery-live-plan.json Live evaluation plan
docs/​routing-evaluation/​runs/​2026-09-26-recovery-full-audit-probe-results.json Full audit results
docs/​routing-evaluation/​runs/​2026-09-26-recovery-frozen-source-gates.json Frozen-source verification gates
docs/​routing-evaluation/​runs/​2026-09-26-recovery-diagnostic-original.mts Diagnostic source snapshot
docs/​routing-evaluation/​runs/​2026-09-26-recovery-diagnostic-audit-probes-green.json Diagnostic audit results
docs/​routing-evaluation/​runs/​2026-09-26-recovery-diagnostic-audit-original.mts Diagnostic audit source
docs/​routing-evaluation/​runs/​2026-09-26-recovery-cancellation-red.json Cancellation failure case
docs/​routing-evaluation/​runs/​2026-09-26-recovery-cancellation-probes.py Cancellation probes
docs/​routing-evaluation/​runs/​2026-09-26-recovery-cancellation-green.json Cancellation verification
docs/​routing-evaluation/​runs/​2026-09-26-recovery-browser-result.json Browser verification result
docs/​routing-evaluation/​runs/​2026-09-26-recovery-browser-original.mts Browser verification source
docs/​routing-evaluation/​runs/​2026-09-26-blind-I-assessment-audit.json Blind assessment audit
docs/​routing-evaluation/​runs/​2026-09-26-blind-H-assessment-audit-v2-regression.json Blind regression audit
docs/​routing-evaluation/​runs/​2026-09-26-blind-assessment-audit-v2-probes.py Assessment audit probes
docs/​routing-evaluation/​runs/​2026-09-26-blind-assessment-audit-v2-probe-results.json Assessment probe results
docs/​routing-evaluation/​runs/​2026-09-26-audit-blind-assessments-v2.py Blind assessment audit script
docs/​routing-evaluation/​2026-09-26-host-recovery.md Retained evaluation report
docs/​routing-demo.md Demo behavior and UI documentation
docs/​roadmap.md Recovery milestone documentation
docs/​architecture.md Host-only recovery architecture
components/​routing-attempts.tsx Attempt telemetry cards
components/​header.tsx Usage dashboard accounting
components/​arena.tsx Streamed routing telemetry
components/​arena-results.tsx Routing overhead and attempts
components/​arena-lessons.tsx Attempt measurement descriptions
components/​arena-history.tsx Recovery cohort separation
app/​globals.css Attempt-card styling
app/​api/​route/​route.ts Route-handler compatibility
app/​api/​arena/​route.ts Partial routing ledger streaming

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@BunsDev
BunsDev merged commit cc1c4dc into main Sep 26, 2026
3 checks passed
@BunsDev
BunsDev deleted the perf/routing-host-recovery branch September 26, 2026 09:44

This branch was successfully deployed

1 active deployment
Preview — 99f5f2e7 Deployed Sep 26, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants