Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
0031e3e
feat(research): execute answerability proof surfaces
ll7 Aug 13, 2026
a315bdb
fix(research): harden answerability status gates
ll7 Aug 14, 2026
defe6ed
fix(research): reject incomplete artifact checksums
ll7 Aug 14, 2026
25ac03f
fix(research): fail closed on missing proof results
ll7 Aug 15, 2026
7b8267d
feat(research): expose explicit yield dimensions
ll7 Aug 13, 2026
c13c90e
fix(research): harden yield snapshot validation
ll7 Aug 14, 2026
4462f6e
feat(research): wire answerability into campaign admission
ll7 Aug 18, 2026
68b7671
fix(research): fail closed on path-only artifact proof
ll7 Aug 18, 2026
29d20fd
fix(research): bind answerability proof to campaign admission
ll7 Aug 18, 2026
0ae3245
fix(benchmark): harden answerability admission proofs
ll7 Aug 18, 2026
252d151
docs(research): clarify bounded fixture reference
ll7 Aug 18, 2026
48188bf
test(ci): include answerability contracts in fast shards
ll7 Aug 18, 2026
ce0398e
fix(research): confine answerability proof inputs
ll7 Aug 20, 2026
e3fe950
chore(research): refresh answerability branch base
ll7 Aug 20, 2026
ceeb5a9
fix(research): require strict admission dry-run and power proof
ll7 Aug 20, 2026
0c42a21
feat(research): wire answerability into campaign admission
ll7 Aug 18, 2026
8dcd90e
fix(research): fail closed on path-only artifact proof
ll7 Aug 18, 2026
80e35d7
chore(research): refresh answerability branch base
ll7 Aug 20, 2026
d99d6a1
Merge remote-tracking branch 'origin/codex/issue-7090-answerability-2…
ll7 Aug 21, 2026
a257ff1
fix(research): confine campaign admission paths
ll7 Aug 21, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions configs/benchmarks/research_campaign_manifest.example.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -191,9 +191,44 @@ answerability:
- summary.json
durability_status: planned
incomplete_policy: Interrupted or unavailable outputs remain explicit and cannot be promoted.
proof_surfaces:
producer:
status: not_run
required: true
preregistration:
status: unavailable
required: false
unavailable_reason: This manifest is a shape-only example without a preregistration file.
evidence_contract:
status: unavailable
required: false
unavailable_reason: No evidence-contract id is declared by this example.
analysis:
status: not_run
required: true
artifact:
status: unavailable
required: false
unavailable_reason: The example plans durable evidence but does not ship a catalog.
result_packet:
status: unavailable
required: false
unavailable_reason: No generic result-interpretation validator is available for this example.

validation:
commands:
- uv run pytest tests/benchmark/test_research_campaign_manifest_contract.py
answerability_proof:
producer:
kind: manifest_rows
analysis:
kind: command
validator_id: pytest_contract
command:
- uv
- run
- pytest
- tests/benchmark/test_research_campaign_manifest_contract.py
- -q
dry_run: not_required_for_example
expected_result: Manifest loads and required contract fields are present.
94 changes: 87 additions & 7 deletions docs/benchmark_campaign_manifest.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,16 +43,92 @@ durable-artifact plan. The evaluator reports one explicit state:
- `diagnostic_only` — a bounded mechanism or manifest check is valid, but it is
not decision-capable evidence;
- `blocked_missing_producer`, `blocked_underpowered`,
`blocked_analysis_contract`, `blocked_noncomparable_rows`, or
`blocked_artifact_plan` — a named prerequisite is not satisfied;
`blocked_analysis_contract`, `blocked_noncomparable_rows`,
`blocked_artifact_plan`, or `blocked_missing_proof` — a named prerequisite
or executable proof surface is not satisfied;
- `invalid_contract` — the answerability schema itself is malformed.

The canonical runner includes the state in `summary.json` and `report.md`.
Use `--require-answerable` when a launch path must reject every state other
than `answerable`; the default packet path remains compatible with older
manifests. Required fallback, degraded, unavailable, or missing producers are
never answerable. Optional unavailable metrics remain explicit and are not
imputed as zero.
imputed as zero. Caveated, fallback, degraded, diagnostic-only, unavailable,
failed, blocked, and not-run row statuses are rejected if a manifest tries to
declare them as success values.

### Executable proof surfaces

Decision-capable manifests should add all six `answerability.proof_surfaces`
entries: `producer`, `preregistration`, `evidence_contract`, `analysis`,
`artifact`, and `result_packet`. The claim-specific minimum for production
admission is `producer`, `preregistration`, `evidence_contract`, `analysis`,
and `artifact`; each must be explicitly required and passed. `result_packet`
is optional only when the generic #7029 validator is unavailable. The bounded
Issue #6474 fixture remains a compatibility fixture and is not a production launch
admission.

Each entry records `status` (`passed`, `unavailable`, `failed`, or `not_run`)
and whether the surface is `required`. An unavailable surface must include
`unavailable_reason`; required surfaces must be `passed` before admission.
Optional unavailable, failed, and not-run surfaces remain visible warnings and
do not become zero-valued evidence. Optional fallback, degraded, blocked, or
missing producers are also retained as warnings.

The runner can execute typed checks from
`validation.answerability_proof` when `--execute-validation` or
`--require-answerable` is supplied. Diagnostic manifests may use manifest rows
or the bounded `pytest_contract` adapter. Decision-capable manifests use a
surface-specific map: `producer_receipt`, the public `preregistration` checker,
the public `evidence_contract` checker, `analysis_receipt`, the public
`artifact_catalog` checker, and the public `result_packet` loader. A generic
pytest result cannot substitute for a producer or analysis receipt. Strict
receipts and file-backed validators bind the campaign, question, estimand, and
surface identity; artifact proofs additionally bind the catalog, selected
artifact IDs, and complete source/output/caption digest sets. Registered
diagnostic commands are argv-only, restricted to repository `tests/*.py` paths,
and bounded by a 120-second timeout. A missing generic result-packet validator
is recorded as `unavailable`; it is not replaced by an issue-specific or
heuristic checker.

Strict admission attaches `answerability.proof_binding` with the source
manifest SHA-256, the declared camera configuration path and SHA-256, and a
digest of the exact proof results. Required file-backed proof specs must carry
the expected input SHA-256, and fixture/diagnostic-only artifact catalogs or
dry-run manifest rows cannot satisfy a decision-capable proof surface.

The `durable_path` adapter is deliberately not an artifact admission proof:
path existence alone cannot establish tracked retention or checksum identity.
A required artifact surface configured with `kind: durable_path` is therefore
recorded as `unavailable` and blocks `answerable`. Use `kind: artifact_catalog`
to invoke the public checksum and path-policy validator.
Decision-capable artifact catalogs must not be sourced from `tests/fixtures` or
controlled diagnostic/fixture source kinds. Result packets with
`smoke_diagnostic`, `visualization_fixture`, `diagnostic_only`, or
`unavailable_causal_inference` classifications are likewise rejected when
declared as required decision proof. Validators read and verify stable pre/post
input digests so a mutation between validation and launch cannot produce a
receipt for different bytes.

The production camera-ready launcher can enforce the same gate without creating
a second readiness contract:

```bash
uv run python scripts/tools/run_camera_ready_benchmark.py \
--config configs/benchmarks/<camera-ready-config>.yaml \
--research-manifest configs/benchmarks/<research-manifest>.yaml \
--require-answerable \
--mode preflight
```

The launcher evaluates the manifest before camera-ready preflight or episode
execution and does not submit compute. A successful gated invocation persists
the exact answerability receipt and proof binding in its JSON result. The
issue #3425 wrapper carries `--research-manifest` and `--require-answerable`
on both preflight and actual run commands, so the gate is re-evaluated after
any intervening manifest/config mutation. Readiness-only callers may omit
`--require-answerable`; the existing packet runner remains the owner of packet
generation.

## Research-Yield Snapshot

Expand All @@ -68,10 +144,14 @@ uv run python scripts/analysis/report_research_yield.py \
```

The report keeps empirical answers, infrastructure/preflight throughput,
explicit failure reasons, and lag fields separate. Its source snapshot digest
and filter definitions remain in the JSON output so a later synthesis can
reproduce the counts without treating implementation activity as scientific
evidence.
explicit failure reasons, lag fields, and query-defined dimensions separate.
The dimension set is intentionally closed: duplicate/competing pull requests,
post-merge repairs, admitted result packets, and blocked-age categories must be
declared in the frozen snapshot with explicit queries, denominators, and
buckets. Unknown dimension names, fields, or buckets are rejected instead of
inferred from live issue, pull-request, or campaign state. The source snapshot digest and
filter definitions remain in the JSON output so a later synthesis can reproduce
the counts without treating implementation activity as scientific evidence.

## Required Fields

Expand Down
8 changes: 5 additions & 3 deletions docs/context/issue_3425_slurm_to_claim_blocker.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,9 +56,11 @@ artifacts. The approved private execution route should call it from a clean publ
`RUN_CAMPAIGN=1` after duplicate, queue, branch, commit, and local-machine submission checks pass.
When `RUN_CAMPAIGN=1` is set, the script also requires the manifest to declare
`research_answerability.v1` with state `answerable` before camera-ready preflight or episode
execution. The current smoke manifest is intentionally diagnostic-only and therefore fails closed
in launch mode until a separately reviewed decision-capable manifest is provided; readiness-only
mode remains available for packet/preflight diagnostics.
execution. The requirement is passed to the canonical camera-ready launcher, which evaluates the
existing proof surfaces once before preflight; no second readiness contract is created. The current
smoke manifest is intentionally diagnostic-only and therefore fails closed in launch mode until a
separately reviewed decision-capable manifest is provided; readiness-only mode remains available
for packet/preflight diagnostics.
The public campaign inputs are:

- `configs/benchmarks/issue_3425_empirical_vertical_slice_manifest.yaml`
Expand Down
Loading
Loading