Merge pull request #591 from escoffier-labs/feat/502-verifier-scorecards

feat(outcome): build verifier-bound skill scorecards

Author: solomonneas

docs/command-inventory.md

@@ -42,7 +42,7 @@ enabled: run `brigade extras on` once, or set `BRIGADE_EXTRAS=1`.
 - `brigade notifications` (extras): 4 command path(s)
 - `brigade openclaw-fragments` (extras): 1 command path(s)
 - `brigade operator`: 24 command path(s)
-- `brigade outcome`: 10 command path(s)
+- `brigade outcome`: 11 command path(s)
 - `brigade pantry` (extras): 5 command path(s)
 - `brigade profiles`: 2 command path(s)
 - `brigade projects` (extras): 10 command path(s)
@@ -282,6 +282,7 @@ enabled: run `brigade extras on` once, or set `BRIGADE_EXTRAS=1`.
 - `brigade operator sync-mcp`
 - `brigade operator sync-tools`
 - `brigade operator verify-harness`
+- `brigade outcome backfill scorecard`
 - `brigade outcome capture`
 - `brigade outcome diff`
 - `brigade outcome doctor`

docs/proposals/skill-scorecards.md

@@ -390,7 +390,7 @@ authoritative outcome is routing authority:
 Read-only projection:
 
 ```
-brigade outcome backfill scorecard [--target PATH] [--dry-run] [--json]
+brigade outcome backfill scorecard [--target PATH] [--json]
 ```
 
 1. Walk `.brigade/work/verify-runs/*/receipt.json`.

docs/receipt-schemas.md

@@ -62,6 +62,33 @@ JSON Schema files.
 | `digests` | object | no | `{algorithm, logs, receipt_sha256, signature?, key_id?}` |
 | `reused_from` | string | no | Prior run id when reused |
 | `interruption` | object | no | Cancel metadata |
+| `verify_manifest_id` | string | no | Registered manifest id when the run was manifest-selected |
+| `required_utility_check_ids` | array of string | no | Manifest-owned utility guardrail ids required for scoring (#503) |
+| `subject_binding` | object | no | Verifier-authored scoreable subject metadata (manifest runs only) |
+| `failure_class` | string | no | Receipt-level #474-style failure class when status is not completed |
+| `failure_kind` | string | no | Receipt-level failure kind paired with `failure_class` |
+
+**`subject_binding` object** (additive, manifest-selected runs)
+
+| Field | Type | Notes |
+| --- | --- | --- |
+| `binding_mode` | string | `patch_backed` or `fixture_eval` |
+| `artifact_kind` | string | `skill` or `card` |
+| `artifact_id` | string | Verifier-owned subject id |
+| `content_fingerprint` | string | Subject content fingerprint at verify time |
+| `manifest_binding` | object | `{manifest_id, payload_sha256, source_path?}` for the exact tracked verifier manifest |
+| `patch_source` | string | `worktree` or `generated` (patch-backed only) |
+| `producer_binding` | object | `{work_session_id, owned_delta_sha256, subject_clean_at_start, start_git}` for patch-backed runs |
+| `verifier_identity` | object | `{verifier_id, session_id}` independent verifier session |
+| `patch_binding` | object | Patch-backed tuple plus `subject_path` and `subject_hash` |
+| `fixture_binding` | object | `{manifest_id, case_id, check_id}` for fixture evaluation runs |
+
+Ad hoc `--command` / `--argv-json` runs omit `subject_binding` and remain audit-only (non-scoreable).
+
+Tracked workspace verifier manifests live under `verify/manifests/*.json`. A manifest owns its
+subject, ordered checks, required utility ids, optional scoped-write globs, and optional route
+opt-in (`route_paths` or exact `route_classes`). Untracked manifests cannot produce scoreable
+receipts or routing authority.
 
 **Command object**
 
@@ -76,6 +103,11 @@ JSON Schema files.
 | `duration_seconds` | number | |
 | `stdout_summary`, `stderr_summary` | string | |
 | `stdout_log_path`, `stderr_log_path` | string | Paths under run dir |
+| `check_role` | string | `effectiveness` or `utility_guardrail` (manifest-selected runs) |
+| `check_id` | string | Stable verifier-owned check id (manifest-selected runs) |
+| `obligation_id` | string | Optional obligation id from the manifest |
+| `failure_class` | string | #474-style class when the command did not succeed |
+| `failure_kind` | string | Typed failure kind paired with `failure_class` |
 
 ---
 
@@ -153,6 +185,7 @@ original file is missing, corrupt, or not an object.
 | `roster` | object | no | Resolution metadata |
 | `lock_workspace` | string | no | |
 | `route` | object | no | Routing brief |
+| `skill_route_policy` | object | no | Frozen pre-plan score inputs, assignments, quota counters, and acceptance reasons |
 | `worker` | string | no | Direct-worker seat |
 | `git` | object | no | |
 | `pre_run_snapshot` | object | no | Run-guard snapshot |
@@ -253,6 +286,27 @@ and patch-ref binding may rewrite worker/synthesis artifacts).
 
 ---
 
+## `brigade.route-decision.v1`
+
+**Path:** `.brigade/runs/<run-id>/route-decision.json`
+
+| Field | Type | Required | Notes |
+| --- | --- | --- | --- |
+| `schema_version` | string | yes | `brigade.route-decision.v1` |
+| `chosen_route` | array of string \| null | yes | Route stages selected for the run |
+| `confidence`, `template_version` | string \| null | yes | Route metadata |
+| `admissible_seats` | array of string | yes | Non-orchestrator seats |
+| `decided_at` | string | no | Pre-plan policy timestamp |
+| `policy_version` | string | no | Skill route-policy version |
+| `score_inputs` | object | no | Receipt-only score inputs keyed by artifact id |
+| `skill_assignments` | array of object | no | Band, authority, manifest, scope, and exploration selection |
+| `exploration` | object | no | Route class, 7/30-day counters, quota, and accept/reject reasons |
+
+When skill routing applies, this receipt preserves the decision made before planning. Finalization
+must not recompute it from post-run state.
+
+---
+
 ## `brigade.roster_snapshot.v1`: `schema_version: 1`
 
 **Path:** `.brigade/runs/<run-id>/roster.json`
@@ -303,6 +357,7 @@ and patch-ref binding may rewrite worker/synthesis artifacts).
 | `worker` | string | Assigned seat name |
 | `task` | string | Task text for the worker |
 | `covers` | array of string | Optional covered artifact ids |
+| `selected_skill_ids` | array of string | Optional pre-plan exploratory skill binding |
 
 ---
 

docs/technical-guide.md

@@ -1197,9 +1197,11 @@ An edited skill therefore earns its score back: once signals for the new text st
 
 The fingerprint sees the artifact's own files and the cards a card links, not the runtime harness around them, the same caveat CocoIndex documents for undecorated helpers: a hash cannot see a skill reaching into the wider workspace at run time, only the files it is made of.
 
-`brigade outcome reconcile` and `brigade outcome fork` score the same current cohort when they decide. The promote/rollback ratchet no longer folds over the full lifetime ledger: it scores the current-fingerprint cohort, so an edited skill must re-earn `install_min_helped` against the text that now ships instead of coasting on signals for text that no longer exists. The decision *rules* are untouched (thresholds, cooldown, and the forward-only ratchet are the same functions PR #163 left in place); only the score fed in narrows. Because grandfathering leaves a never-edited artifact with no proven-stale records, its `current` equals lifetime and its decision, receipt, and one-line output are byte-identical to the pre-fingerprint ratchet. When a decision does drop stale evidence, the decision receipt and `reconcile` output carry the audit fields (`content_fingerprint`, `lifetime_*`, `stale_records`, `legacy_records`) and a `scored current text only` tail.
+`brigade outcome rank` and `brigade outcome explain` retain those ledger cohorts for historical audit. Skill promotion does not use them. `brigade outcome reconcile` and `brigade outcome fork` project skill decisions only from current-fingerprint verifier receipts that pass the subject, patch or fixture identity, verifier ownership, check-role, digest, and failure-taxonomy gates. Legacy rows in `records.jsonl`, including grandfathered rows, cannot promote a skill. Cards continue to use the legacy ledger path.
 
-Two consequences worth stating. A candidate edited before it reaches the install threshold does not promote on the old text's signals; it holds until the new text earns its own. And an already-promoted skill that is edited keeps its `promoted` status until new evidence lands, rather than demoting on the edit alone: its score is now the near-zero current cohort, so it earns no `bump`, and the first verified regression on the new text rolls it back. Demoting purely on an edit would punish improvements, so the ratchet waits for a real signal.
+A candidate skill promotes only when both gates pass. Effectiveness requires at least `install_min_helped` independent passes, no trusted hurt, and a Wilson lower bound of at least 0.15. Utility requires two independent passing evidence units and no trusted failure for every verifier-manifest check marked `utility_guardrail`. Retries and `reused_from` copies do not add evidence units. Missing scorecards and incomplete utility evidence produce explicit hold reasons in reconcile and fork output.
+
+Promotion writes `route_policy.policy_version: scorecard.v1` into the status projection and decision receipt. The router grants full authority only when that marker, promoted status, a current scorecard, and zero trusted hurts agree. One trusted current-fingerprint hurt removes broad authority immediately, before cooldown and regardless of physical rollback success. Install and rollback remain side effects; a failed install does not write promoted status.
 
 ### Context eval metric
 
@@ -1211,6 +1213,46 @@ For example, `brief hit rate 0.50 (2/4 files, 2 missed)` means two of four struc
 
 This is a coverage quality signal for skill and runbook ranking, not a claim that the context was useful, sufficient, or correct. Brief parsing is heuristic, and GraphTrail deltas only see structural code changes. Docs-only runs and runs without structural graph changes produce no context eval.
 
+### Receipt scorecard backfill
+
+`brigade outcome backfill scorecard` is a read-only audit of every verify receipt under `.brigade/work/verify-runs/*/receipt.json`. It never mutates `memory/outcome/records.jsonl`, never appends ledger rows, and never joins receipts to ledger `artifact_id` values. Use `--json` for machine-readable output.
+
+```bash
+brigade outcome backfill scorecard --target .
+brigade outcome backfill scorecard --target /path/to/repo --json
+```
+
+Each discovered `receipt.json` path is counted in `total_receipts`, including malformed files. When a file is unreadable or its JSON is not an object, the audit records one ineligible, unattributed row with stable reason `invalid_receipt_json`.
+
+Stable JSON fields:
+
+| Field | Meaning |
+| --- | --- |
+| `total_receipts` | Count of discovered `receipt.json` paths |
+| `eligible` | Receipts that pass scorecard eligibility rules |
+| `ineligible` | `total_receipts - eligible`; numerator for `ineligibility_rate` |
+| `attributed_ineligible` | Ineligible receipts that carry verifier `subject_binding` |
+| `unattributed` | Receipts without attributable `subject_binding` |
+| `attributed` | `total_receipts - unattributed` |
+| `ineligibility_rate` | `ineligible / total_receipts` (0.0 when empty) |
+| `ineligible_by_reason` | Map of stable reason codes to counts; values sum to `ineligible` |
+| `leading_ineligibility_reason` | Highest-count reason in `ineligible_by_reason` (ties break lexicographically) |
+| `exploration_bands` | Attributed subjects by `unseen`, `candidate`, `provisional`, or `promoted` |
+| `latest_receipt_window` | Rolling view of the latest 50 receipt audits |
+| `legacy_records_audit_only` | Always `true` for this command |
+| `legacy_records_note` | Explains ledger rows are not backfilled into scorecards |
+
+`eligible + ineligible` always equals `total_receipts`.
+
+`latest_receipt_window` sorts receipt audits by `started_at`, then `run_id`, then `receipt_path`, all descending (lexicographic). The first 50 audits in that order form the window regardless of eligibility. Nested fields include `limit` (50), `count`, `eligible`, `ineligible`, `ineligibility_rate`, and `leading_ineligibility_reason`.
+
+Operator surfaces reuse the same audit:
+
+- `brigade work brief` copies those fields under `outcome_loop`.
+- `brigade operator checkup --surface outcome` warns when the loop is half-fed (`outcome_loop_half_fed`) or when more than 50% of the latest 50 receipts are ineligible (`outcome_receipt_ineligibility_high`). Its JSON includes `eligible_receipt_count`, `ineligible_receipt_count`, `attributed_ineligible_receipt_count`, `unattributed_receipt_count`, `ineligibility_rate`, `exploration_bands`, and `latest_receipt_window`.
+
+The 1,601 legacy outcome ledger rows reported in the scorecard proposal are audit-only. They cannot be converted into receipt scorecards because scorecards require verifier-attributed verify receipts, not caller-supplied ledger `artifact_id` values.
+
 `brigade operator checkup` runs the six first-run doctors by default and reports optional loop station health alongside them. Missing optional stations warn and do not block the default ready verdict. Use repeatable `--surface` values to run only named checks, `--list-surfaces` to inspect the stable names, or `--preset evidence-loop` to gate only work receipt integrity and outcome capture, GraphTrail health and the latest work receipt delta, and MiseLedger work-receipt import state. Scoped JSON reports `selected_ready`, leaves `overall_ready` unevaluated, and includes selected, skipped, and per-surface elapsed data.
 
 Use `--handoff` to bridge a completed run back into the memory system.