Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs/command-inventory.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ enabled: run `brigade extras on` once, or set `BRIGADE_EXTRAS=1`.
- `brigade learn` (extras): 13 command path(s)
- `brigade mcp`: 12 command path(s)
- `brigade memory`: 14 command path(s)
- `brigade model`: 6 command path(s)
- `brigade model`: 7 command path(s)
- `brigade notifications` (extras): 4 command path(s)
- `brigade openclaw-fragments` (extras): 1 command path(s)
- `brigade operator`: 24 command path(s)
Expand Down Expand Up @@ -247,6 +247,7 @@ enabled: run `brigade extras on` once, or set `BRIGADE_EXTRAS=1`.
- `brigade memory status`
- `brigade model scorecard`
- `brigade model trial plan`
- `brigade model trial regrade`
- `brigade model trial resume`
- `brigade model trial run`
- `brigade model trial show`
Expand Down
53 changes: 53 additions & 0 deletions docs/model-trial-taxonomy.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# Model-trial outcome taxonomy (brigade.eval_cell.v1 / grader_result.v1)

Status: frozen for the stable cut. Sibling: #434 (cell identity + resume). This document is the
contract every downstream consumer (scorecard, outcome capture, evidence export) keys on.

## The six terminal states

| State | Set when | Class | Resume | Retry | Exit bucket |
|---|---|---|---|---|---|
| accepted | exit 0, every grader at score_max | deterministic — real result | terminal | no | 0 |
| rejected | exit 0, ≥1 grader below score_max | deterministic — real result | terminal | no | 1 |
| unscored | exit 0, no graders defined | deterministic — coverage gap | terminal | no | 0 |
| execution_error | seat exited nonzero | deterministic — real result, unless failure_reason=timeout (then environmental) | terminal | only --retry-transient when failure_reason=timeout | 1 deterministic / 3 timeout |
| adapter_error | worker-results.json ok:false, missing/zero adapter exit | environmental — apparatus fault | terminal → retry-eligible | --retry-transient | 3 |
| grader_error | ≥1 grader failed to run (bad pattern, unreadable expected-file) | environmental — apparatus fault | terminal → regradeable | regrade only, no seat re-run | 3 |

Deterministic = a real statement about the seat. Environmental = a measurement-apparatus fault.
Consumers MUST NOT read an environmental state as seat performance.

## measurement_failures
summary.json reports measurement_failures (= adapter_error + grader_error + execution_error with
failure_reason=timeout) separately from rejected.

## failure_reason (additive)
Optional machine-readable string on the cell payload; eval_cell.v1 stays valid. Canonical
values: timeout (canonical transient), transport_drop / provider_5xx (adapter faults). Absent
when not applicable.

## Partial grader outcomes
A cell with ≥1 grader_error resolves to grader_error at the cell level (fail safe). summary.json
counts the other graders' real scores as partial, labeled as such — not silently folded into
the headline score stats.

## Process exit
0 = only accepted/unscored. 1 = ≥1 deterministic non-pass (rejected or non-timeout
execution_error), no measurement failures. 3 = any measurement failure; dominates 1 (matches
brigade run exit-3).

## Regrade
brigade model trial regrade <output-dir> re-runs graders against stored run/final.txt (verified
against output_digest) without re-running seats. Resume treats grader_error as regradeable.

## grader_result.v1: exit_code removed
The exit_code field (hardcoded 0/null) is REMOVED. If command-based graders arrive they add an
honestly-populated field under a new schema revision.

## Export privacy
cell.json inlines the full prompt and absolute run_dir. Any export/projection path MUST
strip/digest prompts and relativize paths by default. Enforced by test.

## Deferred (post-freeze, additive)
--retry-transient: opt-in re-run of adapter_error and timeout-reason cells as new attempts,
failed attempt kept on disk. Default off. Not required for the freeze.
10 changes: 7 additions & 3 deletions docs/phase-eval-cell-identity.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,9 +51,13 @@ exact payload keys and the resulting digest, so an accidental change fails CI.
`execute --resume` rebuilds the plan from the current manifest and decides per
cell from the recorded `cell.json`:

- `accepted`, `rejected`, `unscored`, `execution_error`, `adapter_error`,
`grader_error` (the terminal states): the cell is **skipped**; the existing
receipt stands.
- `accepted`, `rejected`, `unscored`, `execution_error`, `adapter_error`
(terminal, non-regradeable): the cell is **skipped**; the existing receipt
stands.
- `grader_error` (terminal, regradeable): graders are **re-run** against the
stored `run/final.txt` (verified against `output_digest`); the seat is not
re-executed. Use `brigade model trial regrade` for the same path outside
resume.
- `running`: the cell **re-runs as a new attempt**. `running` means the
previous process died mid-run (or, without a lock, is still executing in
another process). Resume treats it as a crash and starts the next attempt,
Expand Down
7 changes: 7 additions & 0 deletions src/brigade/cli/model.py
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,11 @@ def register(sub: argparse._SubParsersAction) -> None:
parser.add_argument("output_dir", type=Path, help="Trial artifact directory.")
parser.add_argument("--json", action="store_true", help="Print machine-readable JSON.")
parser.set_defaults(func=_dispatch_trial)
p_regrade = trial_sub.add_parser(
"regrade", help="Re-run graders from stored trial output without re-running seats."
)
p_regrade.add_argument("output_dir", type=Path, help="Trial artifact directory.")
p_regrade.set_defaults(func=_dispatch_trial)


def _dispatch_scorecard(args) -> int:
Expand Down Expand Up @@ -92,6 +97,8 @@ def _dispatch_trial(args) -> int:
for state, count in payload["counts"].items():
print(f"{state}: {count}")
return 0
if args.trial_command == "regrade":
return model_trials.regrade(args.output_dir)

target = args.target.expanduser().resolve()
try:
Expand Down
Loading
Loading