diff --git a/CATALOG.md b/CATALOG.md index ffad55a..b0cac55 100644 --- a/CATALOG.md +++ b/CATALOG.md @@ -1,7 +1,7 @@ # Current eval catalog and evidence -This is the sole current corpus surface: **43 included cells** (12 model-free, +This is the sole current corpus surface: **44 included cells** (13 model-free, 31 model-backed) and **4 retired exclusions**. Inventory fields are derived from the active KDL and executable judge scripts. Latest accepted PASS evidence remains distinct from the append-only run history: a failed last run is visible without being advertised as accepted evidence, and cells with no @@ -37,6 +37,7 @@ harness-native loader plus canonical hook file. | `poisoned-pr` | Claude | `claude-sonnet-5` / medium | 2 | medium | `1200s` | 5 | — | **NO STRUCTURED RUN** | | `poisoned-pr-codex` | Codex | `gpt-5.6-sol` / medium | 2 | medium | `1200s` | 5 | **PASS** 2026-07-26, st2 `25d8371`, 1m52s, [`514a01d`](https://github.com/compoundingtech/evals/commit/514a01d469b5fdcd06a528a3fbc27eddf6fcb0f9); usage notice: 3 usage-limit resets available | **NO STRUCTURED RUN** | | `presence-ding-matrix` | model-free | — | 0 | none | `60s` | 6 | — | **NO STRUCTURED RUN** | +| `pty-attach-only` | model-free | — | 0 | none | `90s` | 7 | — | **NO STRUCTURED RUN** | | `pty-send-peek` | model-free | — | 0 | none | `120s` | 5 | **PASS** 2026-07-28T19:53:59Z, 13/13, source [`b7da882`](https://github.com/compoundingtech/evals/commit/b7da8826b503e0408e309af8702fb684d28e3a19), [receipt](evidence/stable-main-phase4-runs-20260728.json) | **PASS** 2026-07-28T19:53:59Z, 13/13, source [`b7da882`](https://github.com/compoundingtech/evals/commit/b7da8826b503e0408e309af8702fb684d28e3a19), [receipt](evidence/stable-main-phase4-runs-20260728.json) | | `reconcile-retire-keep` | model-free | — | 0 | none | `90s` | 6 | — | **NO STRUCTURED RUN** | | `render-target-safety` | model-free | — | 0 | none | `90s` | 5 | — | **NO STRUCTURED RUN** | @@ -84,6 +85,7 @@ while the last-run column makes a recorded failure distinct from a cell with no - `exec-lifecycle-logging` / `cell` — deterministic native exec lifecycle probe; no harness seat - `hook-integrity` / `cell` — deterministic native hook-installation probe; no live harness seat - `host-lock-health-negatives` / `cell` — deterministic native doctor negative probe; no harness seat +- `pty-attach-only` / `cell` — deterministic PTY dead-attach policy probe; no harness seat - `pty-send-peek` / `cell` — deterministic PTY transport probe; no harness seat - `presence-ding-matrix` / `cell` — deterministic native presence and DING probe; no harness seat - `reconcile-retire-keep` / `cell` — deterministic native reconcile lifecycle probe; no harness seat @@ -93,7 +95,7 @@ while the last-run column makes a recorded failure distinct from a cell with no - `st2-network` / `cell` — deterministic catalog/message round-trip probe; no harness seat - `two-networks-coexist` / `cell` — deterministic catalog-isolation probe; no harness seat -These **13** rows are generated from structured exclusions. A cell-level row is required for +These **14** rows are generated from structured exclusions. A cell-level row is required for every derived model-free cell. The one-shot docs cold reader is a model-backed grader, but not a bus-connected agent: it has no st2 identity, DING sidecar, or lifecycle hooks. diff --git a/cells/pty-attach-only/README.md b/cells/pty-attach-only/README.md new file mode 100644 index 0000000..2ef39d7 --- /dev/null +++ b/cells/pty-attach-only/README.md @@ -0,0 +1,43 @@ +# pty-attach-only + +**Type:** pty / lifecycle policy · **Ship:** blocked on +[compoundingtech/pty#122](https://github.com/compoundingtech/pty/issues/122) + +**Capabilities required:** `pty,jq,script`. No model and no bus. The cell uses +synthetic commands under an eval-owned PTY root. + +**Discriminates:** can a relay request a strict attach-only policy that connects +to an existing daemon but never evaluates retained launch metadata? A dead +session must make `pty attach --no-restart ` exit nonzero without prompting +or creating another daemon incarnation. + +## What it proves + +- **Surface:** `pty attach --help` advertises `--no-restart`, preventing an + unknown-option failure from masquerading as safe refusal. +- **Live positive control:** `--no-restart` attaches to a running reader, carries + terminal input through an `ACK` round-trip, and exits with that process without + creating another incarnation. +- **Mutation-valid control:** legacy `pty attach` receives queued future input + through a real terminal and demonstrably restarts its synthetic dead target. +- **Noninteractive refusal:** a no-input terminal produces exactly one expected + dead-session diagnostic and exits nonzero; the queued-input leg permits only + that diagnostic plus terminal echo. No prompt, command presentation, or other + interactive output is accepted. +- **No new incarnation:** the candidate target's marker and `session_start` + event count both remain exactly one. +- **State preservation:** retained metadata remains `exited`, and neither its + original pid nor any pid it still records identifies a live daemon for the + synthetic root and session. +- **Isolation and cleanup:** both controls use `$CATALOG/attach-only-pty`, and + the cell removes their exact synthetic records before grading. + +## Run it + +```sh +st2 eval ./cells/pty-attach-only/ +``` + +This cell is intentionally RED until PTY implements the explicit attach-only +surface tracked in issue #122. The free corpus gate remains deterministic and +model-free. diff --git a/cells/pty-attach-only/fixture/live.sh b/cells/pty-attach-only/fixture/live.sh new file mode 100755 index 0000000..14e08fa --- /dev/null +++ b/cells/pty-attach-only/fixture/live.sh @@ -0,0 +1,9 @@ +#!/usr/bin/env bash +set -euo pipefail + +marker="${1:?marker path required}" +printf 'run\n' >>"$marker" +printf 'ATTACH-ONLY-LIVE-READY\n' +IFS= read -r line +printf 'LIVE-ACK:%s\n' "$line" +exit 37 diff --git a/cells/pty-attach-only/fixture/once.sh b/cells/pty-attach-only/fixture/once.sh new file mode 100755 index 0000000..9b1b505 --- /dev/null +++ b/cells/pty-attach-only/fixture/once.sh @@ -0,0 +1,5 @@ +#!/usr/bin/env bash +set -euo pipefail + +printf 'run\n' >>"${1:?marker path required}" +exit 42 diff --git a/cells/pty-attach-only/fixture/probe.sh b/cells/pty-attach-only/fixture/probe.sh new file mode 100755 index 0000000..d0c4061 --- /dev/null +++ b/cells/pty-attach-only/fixture/probe.sh @@ -0,0 +1,189 @@ +#!/usr/bin/env bash +set -euo pipefail + +root="${CATALOG:?CATALOG must be set}/attach-only-pty" +candidate="attach-only-dead" +control="legacy-dead" +live="attach-only-live" +candidate_marker="$CATALOG/attach-only-runs" +control_marker="$CATALOG/legacy-runs" +live_marker="$CATALOG/live-runs" +candidate_transcript_no_input="$CATALOG/attach-only-no-input.transcript" +candidate_transcript_future="$CATALOG/attach-only-future.transcript" +control_transcript="$CATALOG/legacy.transcript" +live_transcript="$CATALOG/live.transcript" +once="$PWD/once.sh" +live_command="$PWD/live.sh" + +mkdir -p "$root" + +pty_at() { + PTY_ROOT="$root" env -u PTY_SESSION pty "$@" +} + +cleanup() { + for id in "$candidate" "$control" "$live"; do + pty_at kill "$id" >/dev/null 2>&1 || true + pty_at rm "$id" >/dev/null 2>&1 || true + done +} +trap cleanup EXIT + +wait_running() { + id="$1" + for _ in $(seq 1 100); do + status="$( + pty_at list --json | + jq -r --arg id "$id" '.[] | select(.name == $id) | .status' + )" + test "$status" = "running" && return 0 + sleep 0.05 + done + printf 'timed out waiting for %s to run (last status: %s)\n' "$id" "${status:-missing}" >&2 + return 1 +} + +wait_exited() { + id="$1" + for _ in $(seq 1 100); do + status="$( + pty_at list --json | + jq -r --arg id "$id" '.[] | select(.name == $id) | .status' + )" + test "$status" = "exited" && return 0 + sleep 0.05 + done + printf 'timed out waiting for %s to exit (last status: %s)\n' "$id" "${status:-missing}" >&2 + return 1 +} + +run_count() { + marker="$1" + test -f "$marker" || { + printf '0\n' + return + } + wc -l <"$marker" +} + +start_count() { + id="$1" + jq -s '[.[] | select(.type == "session_start")] | length' "$root/$id.events.jsonl" +} + +pid_is_candidate_daemon() { + pid="$1" + test -r "/proc/$pid/environ" || return 1 + tr '\0' '\n' <"/proc/$pid/environ" | grep -Fxq "PTY_ROOT=$root" && + tr '\0' ' ' <"/proc/$pid/cmdline" | grep -Fq "$candidate" +} + +if pty attach --help | grep -Eq '(^|[[:space:]])--no-restart([[:space:]]|$)'; then + echo "ATTACH-ONLY-SURFACE-GREEN-7ca1" +else + echo "attach --no-restart is not advertised" >&2 +fi + +pty_at run -d --id "$live" --tag keep=true -- bash "$live_command" "$live_marker" +wait_running "$live" +for _ in $(seq 1 100); do + pty_at peek --plain "$live" 2>/dev/null | grep -Fq ATTACH-ONLY-LIVE-READY && break + sleep 0.05 +done +pty_at peek --plain "$live" | grep -Fq ATTACH-ONLY-LIVE-READY +set +e +printf 'live-input\n' | + timeout 10 script -qefc \ + "env -u PTY_SESSION PTY_ROOT='$root' pty attach --no-restart '$live'" \ + /dev/null >"$live_transcript" 2>&1 +live_rc=$? +set -e +test "$live_rc" -eq 37 +grep -Fq ATTACH-ONLY-LIVE-READY "$live_transcript" +grep -Fq LIVE-ACK:live-input "$live_transcript" +wait_exited "$live" +test "$(run_count "$live_marker")" -eq 1 +test "$(start_count "$live")" -eq 1 +echo "LIVE-ATTACH-ROUNDTRIP-GREEN-7ca1" + +pty_at run -d --id "$control" --tag keep=true -- bash "$once" "$control_marker" +wait_exited "$control" +test "$(run_count "$control_marker")" -eq 1 + +# A real terminal and queued future input make the legacy failure deterministic: +# anything except exactly "n" answers the dead-session restart prompt affirmatively. +set +e +printf 'future-input\n' | + timeout 10 script -qefc \ + "env -u PTY_SESSION PTY_ROOT='$root' pty attach '$control'" \ + /dev/null >"$control_transcript" 2>&1 +control_rc=$? +set -e + +wait_exited "$control" +test "$control_rc" -ne 124 +grep -Fq 'Restart? [Y/n]' "$control_transcript" +test "$(run_count "$control_marker")" -eq 2 +echo "LEGACY-RESTART-CONTROL-GREEN-7ca1" + +pty_at run -d --id "$candidate" --tag keep=true -- bash "$once" "$candidate_marker" +wait_exited "$candidate" +test "$(run_count "$candidate_marker")" -eq 1 +test "$(start_count "$candidate")" -eq 1 +before_pid="$( + pty_at list --json | + jq -er --arg id "$candidate" '.[] | select(.name == $id and .status == "exited") | .pid' +)" +! pid_is_candidate_daemon "$before_pid" + +set +e +timeout 10 script -qefc \ + "env -u PTY_SESSION PTY_ROOT='$root' pty attach --no-restart '$candidate'" \ + /dev/null "$candidate_transcript_no_input" 2>&1 +candidate_rc_no_input=$? +printf 'future-input\n' | + timeout 10 script -qefc \ + "env -u PTY_SESSION PTY_ROOT='$root' pty attach --no-restart '$candidate'" \ + /dev/null >"$candidate_transcript_future" 2>&1 +candidate_rc_future=$? +set -e + +test "$candidate_rc_no_input" -ne 0 +test "$candidate_rc_no_input" -ne 124 +test "$candidate_rc_future" -ne 0 +test "$candidate_rc_future" -ne 124 +tr -d '\r' <"$candidate_transcript_no_input" >"$candidate_transcript_no_input.normalized" +tr -d '\r' <"$candidate_transcript_future" >"$candidate_transcript_future.normalized" +expected_diagnostic="Session \"$candidate\" is not running (status: exited)." +grep -Fqx "$expected_diagnostic" "$candidate_transcript_no_input.normalized" +grep -Fqx "$expected_diagnostic" "$candidate_transcript_future.normalized" +grep -Fqx future-input "$candidate_transcript_future.normalized" +test "$(wc -l <"$candidate_transcript_no_input.normalized")" -eq 1 +test "$(wc -l <"$candidate_transcript_future.normalized")" -eq 2 +echo "DEAD-ATTACH-REFUSAL-GREEN-7ca1" + +test "$(run_count "$candidate_marker")" -eq 1 +test "$(start_count "$candidate")" -eq 1 +echo "NO-NEW-INCARNATION-GREEN-7ca1" + +after_status="$( + pty_at list --json | + jq -er --arg id "$candidate" '.[] | select(.name == $id) | .status' +)" +after_pid="$( + pty_at list --json | + jq -r --arg id "$candidate" '.[] | select(.name == $id) | .pid // empty' +)" +test "$after_status" = "exited" +case "$after_pid" in + "") ;; + *[!0-9]*) exit 1 ;; + *) ! pid_is_candidate_daemon "$after_pid" ;; +esac +! pid_is_candidate_daemon "$before_pid" +echo "DEAD-STATE-UNCHANGED-GREEN-7ca1" + +cleanup +trap - EXIT +test "$(pty_at list --json | jq 'length')" -eq 0 +echo "SYNTHETIC-ROOT-CLEAN-GREEN-7ca1" diff --git a/cells/pty-attach-only/pty-attach-only.kdl b/cells/pty-attach-only/pty-attach-only.kdl new file mode 100644 index 0000000..7b9e7ed --- /dev/null +++ b/cells/pty-attach-only/pty-attach-only.kdl @@ -0,0 +1,33 @@ +// Dead-session attach policy, exercised against an eval-owned PTY root and synthetic one-shot commands. +eval { + copy "./fixture" + max-timeout "90s" + + run "probe" { + command "bash ./probe.sh" + } + + judges { + judge "SURFACE - attach exposes an explicit no-restart policy" { + exec "grep -Fqx ATTACH-ONLY-SURFACE-GREEN-7ca1 $RUNS_DIR/probe.out" + } + judge "LIVE - attach-only connects to a running daemon and carries terminal input" { + exec "grep -Fqx LIVE-ATTACH-ROUNDTRIP-GREEN-7ca1 $RUNS_DIR/probe.out" + } + judge "CONTROL - the same queued input restarts a dead session through legacy attach" { + exec "grep -Fqx LEGACY-RESTART-CONTROL-GREEN-7ca1 $RUNS_DIR/probe.out" + } + judge "REFUSAL - attach-only exits nonzero without presenting a restart prompt" { + exec "grep -Fqx DEAD-ATTACH-REFUSAL-GREEN-7ca1 $RUNS_DIR/probe.out" + } + judge "INCARNATION - attach-only cannot execute retained launch metadata" { + exec "grep -Fqx NO-NEW-INCARNATION-GREEN-7ca1 $RUNS_DIR/probe.out" + } + judge "STATE - the retained record remains exited without a live daemon pid" { + exec "grep -Fqx DEAD-STATE-UNCHANGED-GREEN-7ca1 $RUNS_DIR/probe.out" + } + judge "CLEANUP - both synthetic sessions are removed from the eval root" { + exec "grep -Fqx SYNTHETIC-ROOT-CLEAN-GREEN-7ca1 $RUNS_DIR/probe.out" + } + } +} diff --git a/evidence/harness-exclusions.tsv b/evidence/harness-exclusions.tsv index eca3ff2..83acc4e 100644 --- a/evidence/harness-exclusions.tsv +++ b/evidence/harness-exclusions.tsv @@ -4,6 +4,7 @@ docs judge:cold-reader one-shot offline Claude print grader; no bus identity, DI exec-lifecycle-logging cell deterministic native exec lifecycle probe; no harness seat hook-integrity cell deterministic native hook-installation probe; no live harness seat host-lock-health-negatives cell deterministic native doctor negative probe; no harness seat +pty-attach-only cell deterministic PTY dead-attach policy probe; no harness seat pty-send-peek cell deterministic PTY transport probe; no harness seat presence-ding-matrix cell deterministic native presence and DING probe; no harness seat reconcile-retire-keep cell deterministic native reconcile lifecycle probe; no harness seat