Add bounded evidence stream for long runs

Author: chaoz23

CHANGELOG.md

@@ -2,6 +2,9 @@
 
 ## Unreleased
 
+- Add an opt-in bounded NDJSON evidence stream for marathon-scale external consumers. It emits replayable numeric ending/runtime witnesses with global elapsed timestamps as they are retained, then a compact terminal summary without constructing the monolithic full-report JSON string.
+- Treat `--max-time` as a total CLI deadline and retain bounded time/heap headroom for clean report finalization. Explicit heap caps now expose the lower search watermark separately from the full process envelope.
+
 ## 0.7.2 — 2026-07-20
 
 - Add private, source-bound QA evidence pins for runtime errors, assertion violations, and approved goal witnesses. Humans and agents can recheck one exact indexed witness after an edit without another search run; pins intentionally retain no story prose, choice labels, observed values, or report content. A pin is evidence memory, not a replacement for a new broad bounded check after meaningful edits.

README.md

@@ -318,9 +318,9 @@ Maintainers can compare shadow recommendations across independent budget runs wi
 
 `--max-frontier-states` and `--max-frontier-memory` are optional shared-search safety envelopes for retained pending checkpoints. Neither has a default: Inkcheck does not impose a low universal frontier cap. If an explicit envelope binds, the run keeps its findings, reports `truncatedBy.frontier`, and does not mislabel the stop as state-budget exhaustion. The same controls are available as `ci.maxFrontierStates` / `ci.maxFrontierMb` and MCP `explore_story` inputs.
 
-`--max-memory <mb>` caps how much heap the run may use before it stops cleanly. A V8 heap out-of-memory abort cannot be caught after the fact, so inkcheck watches memory during exploration and — before it would crash — stops, keeps everything found so far, and reports `truncatedBy.memory` with a partial report rather than losing the run. The default cap is 85% of the V8 heap limit (which honors any `NODE_OPTIONS=--max-old-space-size` you set), so large runs on modest hardware degrade gracefully instead of dying; pass an explicit value to tighten or loosen it. On a memory stop the `nextRun` verdict is `investigate` (raise `--max-old-space-size`, lower `--max-states`, or split the story) — never `broaden`, since more budget would only hit the wall sooner.
+`--max-memory <mb>` caps how much heap the whole run may use before it stops cleanly. A V8 heap out-of-memory abort cannot be caught after the fact, so inkcheck watches memory during exploration and — before it would crash — stops, keeps everything found so far, and reports `truncatedBy.memory` with a partial report rather than losing the run. Explicit caps retain up to 25% (at most 1 GiB) for final result construction and flushing; the bounded stream reports both the full envelope and the lower search watermark. The default cap is 85% of the V8 heap limit (which honors any `NODE_OPTIONS=--max-old-space-size` you set), so large runs on modest hardware degrade gracefully instead of dying; pass an explicit value to tighten or loosen it. On a memory stop the `nextRun` verdict is `investigate` (raise `--max-old-space-size`, lower `--max-states`, or split the story) — never `broaden`, since more budget would only hit the wall sooner.
 
-`--max-time <s>` is the wall-clock counterpart: after `s` seconds the run stops cleanly and returns the partial report it has (`truncatedBy.time`) instead of running to the state budget. There is no default time limit on the local CLI. It's meant for CI or any context that needs a bounded runtime but still wants the findings so far — the same graceful-partial idea as the memory guard, applied to time. The hosted web checker sets this automatically, just under its hard timeout, so a slow story hands back a partial report rather than being killed.
+`--max-time <s>` is the total wall-clock counterpart: the deadline starts before compilation and source scanning, and the run retains 10% of the grant (at most 60 seconds) to merge retained findings and flush its result. Exploration stops with `truncatedBy.time` instead of running to the state budget. There is no default time limit on the local CLI. It's meant for CI or any context that needs a bounded runtime but still wants the findings so far — the same graceful-partial idea as the memory guard, applied to time. The hosted web checker sets this automatically, just under its hard timeout, so a slow story hands back a partial report rather than being killed.
 
 `--profile` prints a cheap static shape profile of the story — variables and where they are assigned, choice density, the longest divert path — plus the depth limit and pass weights inkcheck would choose for that shape, without running any exploration. `--auto` applies those suggestions: it raises `--max-depth` when static divert paths outrun the default (never lowers it, and your explicit flags always win) and hands the profile's pass weights to the portfolio. On a story whose main path is 40 choices deep, default settings find nothing while `--auto` reaches the ending and proves the story exhaustive in ~111 states.
 
@@ -354,6 +354,7 @@ inkcheck can be driven by a human at a terminal, a CI job, or an optional AI cod
 
 - **Machine-readable interface:** `tool.json` at the repo root describes the CLI flags, MCP tools, exit codes, and `--json` output shape in one file.
 - **`--json`** emits the entire report as a single JSON object (`{ compile, stats, explore }`) on stdout — parse that instead of scraping the pretty output. `explore.passes` keeps a bounded pass-local `discoveryCurve` (“what this explorer found”) plus a portfolio-only `portfolioMarginalCurve` (“what this explorer added first”), separating exact terminals, visible outcomes, runtime errors, assertions, goals/stages, knots, and comparable novelty. Summaries preserve first/latest discovery states and dry-gap facts despite compaction. Portfolio reports also include a run-wide curve in actual interleaved execution order; wall time stays observational in progress events. `explore.schedule` shows how adaptive rounds spent the budget. The versioned `shadowDecision` shows what the future anytime policy would recommend and why, including protected per-pass floors and uncertainty. It is observation-only (`applied: false`): it never changes today's search or claims bounded coverage is proof.
+- **`--json-stream`** emits replayable numeric finding witnesses as NDJSON when they are retained, then a bounded terminal summary instead of materializing the full enriched report or one report-sized JSON string. It currently requires `--concurrency 1`, supports one run (no `--next`), and cannot be combined with `--save-report`. A reserved numeric-tag mode supports InkBench's oracle-neutral scoring without exposing oracle variables to search or streaming every ordinary ending. Use the stream for long external-process evaluations where partial evidence must survive a hard wrapper boundary. See the [bounded evidence stream contract](docs/evidence-ndjson.md).
 - **`--progress=ndjson`** emits versioned lifecycle and work-progress events on stderr for an agent or CI log parser. `statesExplored / stateBudget` is budget use, not story coverage; the final stdout report remains authoritative. See the [NDJSON progress contract](docs/progress-ndjson.md).
 - **`--human`** emits a prioritized fix list grouped by errors, warnings, and notes, with file/line locations where available, choice paths for runtime failures, and a next step for each finding.
 - **`--markdown`** emits a GitHub Step Summary-friendly report for humans reviewing CI.

docs/evidence-ndjson.md

@@ -0,0 +1,23 @@
+# Bounded evidence NDJSON contract
+
+`inkcheck story.ink --json-stream --concurrency 1` writes one JSON object per stdout line. It is an opt-in large-run transport for wrappers that need replayable evidence before a process reaches its final reporting phase. Ordinary `--json`, human, Markdown, progress, and saved-report contracts are unchanged.
+
+Current events use `schemaVersion: 1`:
+
+- `run_start` records the Inkcheck version, effective configuration, and state budget.
+- `ending` records a stable event ID, global `elapsedMs`, pass-local `firstDiscoveredAtState`, numeric `choiceIndices`, and optional `foundBy` pass. It omits choice prose, final text, and variables.
+- `runtime_error` records the same replay coordinates plus the error message and optional source location.
+- `benchmark_signal` is reserved for InkBench's oracle-neutral fixtures. A root `INKBENCH_SIGNAL_MODE` tag suppresses ordinary finding records, and an exact `INKBENCH_SIGNAL:<non-negative integer>` tag emits only that numeric signal, its numeric replay path, and timing. No story prose, variables, or arbitrary tags are copied.
+- `run_end` is the authoritative bounded summary. It records compile status, states explored, finding counts, limits, execution mode, truncation causes, resource envelopes/deadlines, and emitted-evidence counts. It does not contain the full ending, pass, schedule, or discovery-curve arrays.
+
+Events are flushed as newline-delimited records during the run. A consumer may retain and replay complete finding lines even if an outer process guard later interrupts the CLI. Absence of `run_end` means the run was interrupted; it must not be relabeled as clean completion.
+
+The stream deduplicates endings by numeric replay path and runtime errors by message plus numeric path across portfolio passes. A fixed story seed makes a numeric path deterministic. The `elapsedMs` value is measured from the CLI run start and is globally comparable within that process; `firstDiscoveredAtState` remains pass-local and must not be treated as a portfolio-global work position.
+
+The reserved InkBench tags are a scoring transport, not a search input. The portfolio does not score, prioritize, hash, or retain tags in ending identity. InkBench removes planted oracle variables and assignments before invoking Inkcheck, replaces a triggering assignment with one numeric signal tag, and replays only those signaled paths against its separately pinned instrumented story. Ordinary stories should not use the reserved `INKBENCH_SIGNAL_` prefix.
+
+`--json-stream` currently requires `--concurrency 1`, supports one bounded run (no `--next`), and cannot be combined with `--save-report`. These constraints prevent a function callback from crossing worker isolates and prevent a supposedly bounded transport from invoking the monolithic artifact writer.
+
+For a declared `--max-time`, exploration retains 10% of the total grant, capped at 60 seconds, to merge the already-retained internal evidence and flush `run_end`. This reserve is part of the total wall budget, not extra search time.
+
+For operational counters without finding content, keep using [`--progress=ndjson`](progress-ndjson.md) on stderr. Evidence output can contain runtime error text and replay paths, so store it with the same privacy controls as an ordinary report.

package.json

@@ -70,6 +70,7 @@
     "docs/config-schema-v1.json",
     "docs/report-schema-v1.md",
     "docs/machine-output.md",
+    "docs/evidence-ndjson.md",
     "docs/local-artifacts.md",
     "docs/local-checkpoints.md",
     "docs/shared-checkpoint-schema-v1.md",

src/adaptive-concurrent-portfolio.ts

@@ -103,6 +103,7 @@ function sanitizedOptions(options: AdaptiveConcurrentOptions): ExploreOptions {
     activationPilotStatesForTest: _activationPilotStatesForTest,
     onProgress: _onProgress,
     onSnapshot: _onSnapshot,
+    onEvidence: _onEvidence,
     memoryGuard: _memoryGuard,
     timeGuard: _timeGuard,
     loopRiskRegistry: _loopRiskRegistry,