runs: add an append-only lifecycle journal and replayable run.json projection

[solomonneas]

Problem

Brigade rewrites run.json as a run moves through planning, dispatch, synthesis, recovery, and terminal states. Atomic replacement keeps polling readers from seeing a truncated file, but each write erases the prior state.

After a process dies, Brigade cannot reliably answer:

• Which lifecycle transition committed last?
• Did a dispatch start, finish, or stop between snapshot writes?
• Is a recovery request a retry of the same transition or a new request?
• Can a missing, corrupt, or stale run.json be rebuilt from durable facts?

Receipts remain useful proof artifacts, and the outcome ledger remains the scored-outcome chain. Neither provides ordered lifecycle history for a run.

Brigade already defines brigade.run_event.v1. This issue gives that event type a storage, replay, and migration contract.

Inspiration

The immediate prompt for this issue was Yohei Nakajima's observation that long-running agent systems tend toward immutable event logs:

• “if you are trying to solve long-running agent problems, you will eventually start using immutable event logs”
• A follow-up connecting the pattern to Git history, transaction journals, medical and aviation audit trails, critical-infrastructure event logs, and versioned scientific records

These posts are design prompts, not dependencies or an argument for distributed log infrastructure. Brigade applies the principle at the per-run, local-file boundary defined below.

Concurrency and storage boundary

The current run guard permits one lifecycle writer for a run. watch, show, and similar commands are concurrent readers. Recovery may become the next writer only after it claims a stale run lock.

Use that constraint instead of adding a database:

• Append lifecycle records to <run-dir>/events/lifecycle.jsonl.
• Keep provider and worker event streams separate.
• Require the active run-lock owner for normal lifecycle appends.
• Use one write call per canonical JSON line, flush it, and call fsync before refreshing run.json.
• Treat a partial final line as an interrupted append. Recovery may truncate only that incomplete suffix after preserving diagnostic evidence.
• Store the journal with the same private permissions and gitignore behavior as other local run artifacts.

This is a per-run journal, not a repo-wide broker. If Brigade later needs to atomically append an event and create work for multiple competing consumers, that should be a separate SQLite-backed queue issue with evidence that the single-writer boundary is no longer enough.

Event contract

Each line uses an allowlisted event-type schema and contains:

• schema: brigade.run_event.v1
• contiguous per-run sequence
• event_id
• run_id
• event type
• recorded_at
• stable idempotency key
• SHA-256 digest of the canonical request
• previous event digest
• current event digest
• an allowlisted payload for that event type

Ordering comes from sequence, not timestamps. Canonical digest input must specify sorted keys, UTF-8, separators, timestamp format, numeric rules, and null handling.

The digest chain detects accidental corruption, incomplete writes, and unreviewed mutation. It is not a claim that the journal resists an attacker who can rewrite the entire run directory.

Idempotency behavior

Idempotency keys are scoped to one run.

• A new key appends the event.
• The same key with the same request digest returns the committed event without appending another line.
• The same key with a different request digest returns a conflict and appends nothing.

The append API should accept the expected prior sequence and fail with a concurrency error if the caller is operating on stale state, even though the run guard normally prevents competing writers.

External side effects

The journal cannot make model-provider calls, subprocesses, or notifications transactional.

Record external work as paired facts:

1. Append and sync a requested event.
2. Perform the external action.
3. Append and sync an observed, completed, or failed event.

Recovery is therefore at-least-once. Before repeating a requested action with no terminal observation, Brigade must use the same idempotency identity when the boundary supports it or check existing attempt evidence before reissuing the action. A provider call must not be described as exactly-once.

Intentional pause and approval events

An intentional approval wait is a resumable lifecycle state, not a crash. The journal must be able to represent it without keeping the invoking process or a network connection alive.

Use past-tense events:

• approval.requested
• run.paused
• approval.granted
• approval.rejected
• approval.held
• approval.consumed
• run.resumed

Existing Daily and Tool approval records remain the authorization sources. The run journal records their effect on the run and references the approval by ID, source subsystem, source or contract fingerprint, bounded decision state, decision time, and consuming run ID. It does not duplicate raw arguments, private review reasons, or authorization policy.

Pause and resume behavior:

1. Append and sync approval.requested and run.paused before releasing the run lock.
2. Refresh run.json with a compatibility-safe resumable status, the approval reference, and the last applied event sequence and digest.
3. Exit the active process cleanly. The paused run does not retain the run lock and is not treated as stale-owner recovery.
4. A later resume command acquires a new normal run lock, reloads the approval artifact, and revalidates its source, contract, and evidence fingerprints.
5. A rejected or held approval cannot resume the run. A granted approval is consumed once by the existing approval subsystem before another external action starts.
6. Append the observed approval decision, approval.consumed, and run.resumed. If the process exits after the approval store records consumption but before the journal records it, recovery observes the consumed approval and appends the missing facts without consuming or executing it again.

The event registry may support these facts in the first slice. Creating a new approval store, policy engine, or user interface is outside this issue.

Projection and recovery

Keep run.json as the compatibility snapshot for existing readers.

• Every lifecycle transition appends and syncs its event before refreshing run.json.
• Add projector_version, journal_present, and the last applied event sequence and digest to the snapshot.
• Define a field-by-field mapping from lifecycle event types to run.json.
• Unknown event schemas or event types stop replay with a clear compatibility error. They are never skipped silently.
• brigade runs recover verifies the chain and rebuilds a missing, corrupt, or stale snapshot.
• Recovery never repeats a terminal event whose idempotency key and request digest already match.

During migration, run both paths in shadow mode: build the snapshot through the current writer and through event replay, then compare them after each transition. A mismatch is recorded and blocks automatic projection replacement until the event mapping is corrected.

Privacy, retention, and redaction

Payloads use per-event-type allowlists. Unknown keys are rejected.

Do not store raw prompts, model output, tool arguments, credentials, retrieved private source, provider response bodies, or stack traces in lifecycle events. Store bounded error type and redacted summary fields, then reference a separately classified artifact by relative path, SHA-256 digest, media type, byte size, and privacy class when more evidence is required.

Lifecycle journals follow the run directory's retention and archive policy. The first slice does not compact part of a live journal. Existing retention may archive or remove an entire run after required receipts and evidence have been preserved.

Append-only is the normal-operation rule, not a reason to preserve a leaked credential. Define an explicit operator-only redaction procedure that:

1. quarantines the original artifact,
2. rewrites and re-chains the affected journal,
3. appends or stores a redaction record with the reason and affected sequence range,
4. verifies all projections again,
5. removes quarantined sensitive copies according to the incident procedure.

Smallest first slice

1. Add the canonical event envelope, event-type registry, append API, and chain verifier.
2. Journal the existing run status transitions behind an opt-in feature flag.
3. Build a projector for the current run.json contract.
4. Shadow-compare legacy and replayed snapshots after each transition.
5. Extend brigade runs recover and doctor output with journal verification and projection repair.
6. Make the journal the source for new runs only after parity tests pass.

Acceptance criteria

• A golden lifecycle fixture replays to a byte-identical canonical snapshot on repeated runs.
• Every current run.json field is mapped to an event type or explicitly documented as derived.
• Reusing an idempotency key with the same request digest returns the committed event without a duplicate append.
• Reusing an idempotency key with a different request digest returns a conflict.
• SIGKILL tests cover exit before append, during append, after journal sync, and before snapshot replacement.
• Recovery preserves and reports a partial final line before truncating that incomplete suffix.
• Deleting or corrupting run.json and running recovery rebuilds the same observable run state.
• Sequence gaps, digest mismatches, unknown schemas, unknown event types, and stale projections produce bounded diagnostic errors rather than crash loops.
• Existing brigade runs watch, show, steer, interrupt, recover, and resume behavior remains compatible.
• A legacy run directory without lifecycle.jsonl still loads.
• A new run directory still exposes a compatible run.json to the previous Brigade release.
• A requested external action without a terminal observation is surfaced as recovery work and is not described as exactly-once.
• A paused run releases its lock, survives process exit, and resumes without repeating completed transitions.
• Rejected, held, stale-fingerprint, and previously consumed approvals cannot resume a run.
• A crash after approval consumption but before journal observation is reconciled without a second consumption or external action.
• The compatibility snapshot represents an approval pause using a status understood by the documented migration reader set.
• Event payload allowlists reject unknown fields and the private-data exclusions have regression tests.
• Journal size and append latency are measured across 1,000 representative runs before the feature flag is removed.
• No daemon, network service, or new runtime dependency is introduced.

Non-goals

• Event-source memory cards, documentation, configuration, or every Brigade subsystem.
• Replace readable receipts, the outcome ledger, or Evidence Ledger storage.
• Replace the existing Daily or Tool approval stores, or add a new approval user interface.
• Store raw provider transcripts in the lifecycle journal.
• Add a distributed log, remote broker, work queue, lease system, or server process.
• Claim tamper resistance against an attacker with write access to the run directory.

Migration

1. New runs may opt into lifecycle journaling while keeping the existing snapshot writer.
2. Shadow replay compares the derived and legacy snapshots after every transition.
3. Recovery may rebuild from the journal only after the chain verifies and the projector recognizes every event.
4. Once parity is proven, new runs treat the journal as authoritative and refresh run.json as a compatibility projection.
5. Legacy runs remain snapshot-only and require no migration.

This replaces the narrower plan to retain status history without defining replay, idempotency, external-side-effect boundaries, or privacy enforcement.