Skip to content

feat(outcome): build verifier-bound skill scorecards - #591

Merged
solomonneas merged 7 commits into
mainfrom
feat/502-verifier-scorecards
Jul 27, 2026
Merged

feat(outcome): build verifier-bound skill scorecards#591
solomonneas merged 7 commits into
mainfrom
feat/502-verifier-scorecards

Conversation

@solomonneas

@solomonneas solomonneas commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator

Summary

Implements the scorecard design from #570 and completes #502 through its five ordered slices:

  • accepts score inputs only from tracked, verifier-authored manifests with bound subjects, check roles, patch or fixture identity, receipt digests, and failure taxonomy
  • projects current-fingerprint effectiveness, verifier cost, retry stability, evidence integrity, and per-check utility retention from eligible verify receipts
  • gives unseen and low-data skills deterministic, route-scoped exploration with 7-day, 30-day, per-run, and optional budget caps
  • promotes only when effectiveness and every required utility check pass, then records scorecard.v1 routing authority; one trusted hurt removes that authority immediately
  • adds read-only scorecard backfill, work-brief fields, and operator checkup --surface outcome

Why

The previous loop counted distinct ledger artifact ids as scored, relied on optional caller-attributed capture, exposed rank as a read-only view, and left reconcile dry-run by default. Receipt volume could therefore grow without producing promotion-ready evidence. Caller labels and no-work receipts could also contaminate routing if treated as score inputs.

This change keeps legacy records.jsonl rows audit-only for skill promotion. It does not infer subjects from those rows or convert them during backfill.

Compatibility

Cards remain on the existing ledger decision path. Generated-patch evidence must satisfy verifier independence, and required utility checks remain verifier-manifest declarations, so this composes with #507 and #499 without making their audits blocking.

Verification

  • brigade work verify run --target . --command "./scripts/verify" --capture brigade-work
  • rebased receipt: .brigade/work/verify-runs/20260727-002601-work-verify-66ea63/receipt.json
  • 4,598 passed, 3 skipped
  • coverage: 82.91%

Closes #502
Closes #571
Closes #572
Closes #573
Closes #503
Closes #574

solomonneas and others added 7 commits July 26, 2026 20:25
@solomonneas @cursoragent
feat(outcome): bind scoreable verifier receipts
dd375ff 
Co-Authored-By: Cursor <cursoragent@cursor.com>
@solomonneas @cursoragent
feat(outcome): project verifier receipt scorecards
8daf0f2 
Co-Authored-By: Cursor <cursoragent@cursor.com>
@solomonneas @cursoragent
feat(router): cap verifier skill exploration
b5f16c6 
Co-Authored-By: Cursor <cursoragent@cursor.com>
@solomonneas @cursoragent
feat(outcome): gate promotion on utility
e7098f7 
Co-Authored-By: Cursor <cursoragent@cursor.com>
@solomonneas @cursoragent
feat(outcome): expose scorecard audit health
09f1082 
Co-Authored-By: Cursor <cursoragent@cursor.com>
@solomonneas @cursoragent
fix(router): satisfy route policy types
53e56fe 
Co-Authored-By: Cursor <cursoragent@cursor.com>
@solomonneas @cursoragent
test(outcome): seed verifier promotion receipt
bcc57f8 
Co-Authored-By: Cursor <cursoragent@cursor.com>
@coderabbitai

coderabbitai Bot commented Jul 27, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are limited based on label configuration.

🏷️ Required labels (at least one) (1)
  • coderabbit-review

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: escoffier-labs/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 8b2bc37c-ffa8-4c5f-8f79-1e37b28acfcf

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/502-verifier-scorecards

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@solomonneas solomonneas self-assigned this Jul 27, 2026
@solomonneas
solomonneas merged commit c783d89 into main Jul 27, 2026
28 checks passed
@solomonneas
solomonneas deleted the feat/502-verifier-scorecards branch July 27, 2026 00:48
solomonneas added a commit that referenced this pull request Aug 5, 2026
@solomonneas @claude @cursoragent
docs(changelog): correct citations and fill gaps in the 0.26.0 notes (#…
1c68455 
…700)

The Unreleased section carried five entries whose citation or scope did not
match the diff, and was missing 17 merged PRs. Every change below is backed by
the merge SHA named in .brigade/salvage/assess-0260/CHANGELOG-AUDIT-REPORT.md
and re-checked against `git show`.

Citations: #565 -> #598 (021c078), two #536 -> #537 (433d285). The Fixed
transport-redaction bullet moves off #671 to the upstream agent-notify #8
(7c97909) that arrived through the #433 history import; #671's real changes,
timeout validation and the Stop-hook opt-in, were already cited correctly in
Changed (92257d2).

Two more wrong-cited entries turned out to have a correctly-cited twin already
in the file, so renumbering them would have produced duplicate bullets. The
#349 entry duplicates the #425 one and the #518 entry duplicates the #520 one;
each pair is collapsed to a single bullet carrying the merging PR. Two
pre-existing duplicates are removed the same way: the uncited passive-update
notice against #426, and the shorter of the two #679 bullets.

Scope: the #685 bullet no longer claims #590's probe mechanics (b966152 owns
the parallel cap, fingerprint cache, and transport smoke; 265639b only wires
admission). The #643 bullet names `agent-notify init` rather than `brigade
init`, matching 0c781e1. The #433 bullet gains #439, which is where the
setup-install wiring actually landed (e524e80). The redundant combined
(#685, #688) bullet is dropped.

Adds entries for #447, #591, #547, #563, #560, #569, #699 under Added; #544 and
#559 under Changed; and #440, #539, #552, #558, #551, #549, #548, #545 under
Fixed. Four of these descriptions correct the audit report where the diff
disagreed with it, most notably #545, which changes the PYTHONPATH import root
and never touches subprocess cwd.

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment