Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions CATALOG.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
<!-- Generated by bin/generate-catalog.sh from cell KDL and evidence/*.tsv. Do not edit. -->
# Current eval catalog and evidence

This is the sole current corpus surface: **57 included cells** (17 model-free,
This is the sole current corpus surface: **58 included cells** (18 model-free,
40 model-backed) and **4 retired exclusions**. Inventory fields are derived from the active
KDL and executable judge scripts. Latest accepted PASS evidence remains distinct from the append-only run
history: a failed last run is visible without being advertised as accepted evidence, and cells with no
Expand Down Expand Up @@ -48,6 +48,7 @@ harness-native loader plus canonical hook file.
| `poisoned-pr` | Claude | `claude-sonnet-5` / medium | 2 | medium | `1200s` | 5 | — | **NO STRUCTURED RUN** |
| `poisoned-pr-codex` | Codex | `gpt-5.6-sol` / medium | 2 | medium | `1200s` | 5 | **PASS** 2026-07-26, st2 `25d8371`, 1m52s, [`514a01d`](https://github.com/compoundingtech/evals/commit/514a01d469b5fdcd06a528a3fbc27eddf6fcb0f9); usage notice: 3 usage-limit resets available | **NO STRUCTURED RUN** |
| `presence-ding-matrix` | model-free | — | 0 | none | `60s` | 6 | — | **NO STRUCTURED RUN** |
| `pty-attach-machine-stream` | model-free | — | 0 | none | `90s` | 5 | — | **NO STRUCTURED RUN** |
| `pty-attach-only` | model-free | — | 0 | none | `90s` | 7 | — | **NO STRUCTURED RUN** |
| `pty-send-peek` | model-free | — | 0 | none | `120s` | 5 | **PASS** 2026-07-28T19:53:59Z, 13/13, source [`b7da882`](https://github.com/compoundingtech/evals/commit/b7da8826b503e0408e309af8702fb684d28e3a19), [receipt](evidence/stable-main-phase4-runs-20260728.json) | **PASS** 2026-07-28T19:53:59Z, 13/13, source [`b7da882`](https://github.com/compoundingtech/evals/commit/b7da8826b503e0408e309af8702fb684d28e3a19), [receipt](evidence/stable-main-phase4-runs-20260728.json) |
| `reconcile-retire-keep` | model-free | — | 0 | none | `90s` | 6 | — | **NO STRUCTURED RUN** |
Expand Down Expand Up @@ -100,6 +101,7 @@ while the last-run column makes a recorded failure distinct from a cell with no
- `exec-lifecycle-logging` / `cell` — deterministic native exec lifecycle probe; no harness seat
- `hook-integrity` / `cell` — deterministic native hook-installation probe; no live harness seat
- `host-lock-health-negatives` / `cell` — deterministic native doctor negative probe; no harness seat
- `pty-attach-machine-stream` / `cell` — deterministic installed PTY attach composition probe; no harness seat
- `pty-attach-only` / `cell` — deterministic PTY dead-attach policy probe; no harness seat
- `pty-send-peek` / `cell` — deterministic PTY transport probe; no harness seat
- `presence-ding-matrix` / `cell` — deterministic native presence and DING probe; no harness seat
Expand All @@ -112,7 +114,7 @@ while the last-run column makes a recorded failure distinct from a cell with no
- `targeted-reconcile-isolation` / `cell` — deterministic selected reconcile isolation probe; no bus-connected agent
- `two-networks-coexist` / `cell` — deterministic catalog-isolation probe; no harness seat

These **18** rows are generated from structured exclusions. A cell-level row is required for
These **19** rows are generated from structured exclusions. A cell-level row is required for
every derived model-free cell. The one-shot docs cold reader is a model-backed grader, but not a bus-connected
agent: it has no st2 identity, DING sidecar, or lifecycle hooks.

Expand Down
38 changes: 38 additions & 0 deletions cells/pty-attach-machine-stream/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# pty-attach-machine-stream

**Type:** pty / installed attach composition · **Upstream contract:**
[compoundingtech/pty#140](https://github.com/compoundingtech/pty/pull/140) and
[compoundingtech/pty#141](https://github.com/compoundingtech/pty/pull/141), merged on PTY main at
[`d5fabc3917407aeb937a012bd97679c303e18033`](https://github.com/compoundingtech/pty/commit/d5fabc3917407aeb937a012bd97679c303e18033).

**Capabilities required:** `pty,jq,node,script`. No model and no bus. The cell uses the `pty` executable on `PATH`,
not a source-tree module or `dist/cli.js` entrypoint.

## What it proves

- **Packaged launcher:** fd 3 crosses the shipped `bin/pty` boundary and carries a parseable v1 stream.
- **Real initial snapshot:** an eval-owned target daemon emits exact `24x80` `GEOMETRY` immediately followed
by `SCREEN`; the screen preserves the red SGR state produced before attach.
- **Real reconnect snapshot:** an installed `pty remote-serve` process exposes the target through a one-shot
local transport proxy. A synthetic `fabric dial` selector drops the first route and withholds the second until
the target has produced another
line and attaches a controlled `13x47` client. The one continuous fd 3 stream must then contain exact
`13x47` `GEOMETRY` followed by `SCREEN`, proving both min-wins geometry and a current snapshot.
- **Framing boundary:** terminal content and `EXIT` are decoded from fd 3, while stdout stays empty and stderr
contains only reconnect status. This catches both descriptor loss and protocol/text contamination.
- **Cleanup:** the target and both route servers are removed from the eval-owned PTY root.
- **Oracle mutations:** wrong geometry, stripped SGR, stale reconnect state, data after exit, truncation, and
stdout/stderr contamination all fail the checker before the real composition runs.

The fixture controls only transport selection and the deliberate connection drop; the installed CLI, PTY daemon, remote routing, attach client,
snapshot serialization, reconnect loop, and packaged launcher are all exercised as shipped. This is broader
than the upstream unit tests and specifically models the composition consumed by a terminal UI such as
Fractal.

## Run it

```sh
st2 eval ./cells/pty-attach-machine-stream/
```

The accepted composition proof uses the packaged PTY flake at the exact merged revision above.
142 changes: 142 additions & 0 deletions cells/pty-attach-machine-stream/fixture/check-stream.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,142 @@
#!/usr/bin/env node
import fs from "node:fs"
import assert from "node:assert/strict"

const geometry = (rows, columns) => {
const payload = Buffer.alloc(4)
payload.writeUInt16BE(rows, 0)
payload.writeUInt16BE(columns, 2)
return payload
}

const frame = (type, payload) => {
const header = Buffer.alloc(5)
header.writeUInt8(type)
header.writeUInt32BE(payload.length, 1)
return Buffer.concat([header, payload])
}

const decode = (data, complete) => {
const packets = []
let offset = 0
while (offset + 5 <= data.length) {
const type = data.readUInt8(offset)
const length = data.readUInt32BE(offset + 1)
if (length > 32 * 1024 * 1024) throw new Error(`oversized frame: ${length}`)
if (offset + 5 + length > data.length) break
packets.push({ type, payload: data.subarray(offset + 5, offset + 5 + length) })
offset += 5 + length
}
if (complete && offset !== data.length) throw new Error("truncated trailing frame")
return packets
}

const snapshotIndexes = (packets) => {
const indexes = []
for (let index = 0; index + 1 < packets.length; index++) {
if (packets[index].type === 10 && packets[index + 1].type === 5) indexes.push(index)
}
return indexes
}

const parseGeometries = (raw) => raw.split(",").map((entry) => {
const match = /^(\d+)x(\d+)$/.exec(entry)
if (!match) throw new Error(`invalid expected geometry: ${entry}`)
return { rows: Number(match[1]), columns: Number(match[2]) }
})

const validateSnapshots = (packets, expectedGeometries, count) => {
const indexes = snapshotIndexes(packets)
if (indexes.length < count) throw new Error(`expected ${count} snapshots, got ${indexes.length}`)
for (let snapshot = 0; snapshot < count; snapshot++) {
const payload = packets[indexes[snapshot]].payload
const expected = expectedGeometries[snapshot]
if (!expected || payload.length !== 4 || payload.readUInt16BE(0) !== expected.rows ||
payload.readUInt16BE(2) !== expected.columns) {
throw new Error(`snapshot ${snapshot + 1} geometry does not match ${expected?.rows}x${expected?.columns}`)
}
}
return indexes
}

const validateFinal = (data, expectedGeometries, stdout = Buffer.alloc(0), stderr = Buffer.alloc(0)) => {
const packets = decode(data, true)
if (packets.length < 5) throw new Error("too few frames")
if (packets[0].type !== 10 || packets[1].type !== 5) {
throw new Error("initial stream does not begin with GEOMETRY, SCREEN")
}
const indexes = validateSnapshots(packets, expectedGeometries, expectedGeometries.length)
if (indexes.length !== expectedGeometries.length) {
throw new Error(`expected ${expectedGeometries.length} snapshots, got ${indexes.length}`)
}
const initial = packets[indexes[0] + 1].payload
const reconnected = packets[indexes[1] + 1].payload
const coloredMarker = Buffer.from("\x1b[31mINITIAL_COLOR_61e8")
if (!initial.includes(coloredMarker) || !reconnected.includes(coloredMarker)) {
throw new Error("snapshot lost the red SGR state around the initial marker")
}
if (!reconnected.includes(Buffer.from("AFTER_DROP_61e8"))) {
throw new Error("reconnect screen is not the current terminal state")
}
const exits = packets.filter((packet) => packet.type === 4)
if (exits.length !== 1 || packets.at(-1).type !== 4) throw new Error("stream does not end in one EXIT")
if (!packets.some((packet) => packet.type === 0 && packet.payload.includes(Buffer.from("FINAL_DATA_61e8")))) {
throw new Error("final terminal DATA was not ordered before EXIT")
}
if (packets.some((packet) => ![0, 4, 5, 10].includes(packet.type))) {
throw new Error("unexpected packet type in machine stream")
}
if (stdout.length !== 0) throw new Error("machine attach wrote to stdout")
const reconnectStatus = Buffer.from("\r\n[reconnecting… — Ctrl-\\ or Ctrl-C to stop]\r\n")
if (!stderr.equals(reconnectStatus)) {
throw new Error("machine attach stderr was not exactly one reconnect status")
}
}

const selfTest = () => {
const expected = parseGeometries("24x80,13x47")
const colored = Buffer.from("\x1b[31mINITIAL_COLOR_61e8\x1b[0m")
const current = Buffer.concat([colored, Buffer.from("\r\nAFTER_DROP_61e8")])
const packets = [
frame(10, geometry(24, 80)), frame(5, colored),
frame(10, geometry(13, 47)), frame(5, current),
frame(0, Buffer.from("FINAL_DATA_61e8")), frame(4, Buffer.alloc(0)),
]
const valid = Buffer.concat(packets)
const reconnectStatus = Buffer.from("\r\n[reconnecting… — Ctrl-\\ or Ctrl-C to stop]\r\n")
validateFinal(valid, expected, Buffer.alloc(0), reconnectStatus)
const uncolored = Buffer.from("\x1b[HINITIAL_COLOR_61e8\x1b[0m")
const mutations = [
() => validateFinal(Buffer.concat([frame(10, geometry(1, 1)), ...packets.slice(1)]), expected),
() => validateFinal(Buffer.concat([frame(10, geometry(24, 80)), frame(5, uncolored), ...packets.slice(2)]), expected),
() => validateFinal(Buffer.concat([packets[0], packets[1], packets[2], frame(5, colored), ...packets.slice(4)]), expected),
() => validateFinal(Buffer.concat([...packets.slice(0, 4), packets[5], packets[4]]), expected),
() => validateFinal(valid.subarray(0, valid.length - 1), expected),
() => validateFinal(valid, expected, Buffer.from("unexpected"), reconnectStatus),
() => validateFinal(valid, expected, Buffer.alloc(0), Buffer.concat([reconnectStatus, Buffer.from("\x1b[32mLEAK")])),
]
for (const mutate of mutations) assert.throws(mutate)
console.log("ORACLE-MUTATIONS-GREEN-61e8")
}

if (process.argv[2] === "--self-test") {
selfTest()
} else {
const [path, mode, expectedRaw, geometryRaw, stdoutPath, stderrPath] = process.argv.slice(2)
if (!path || !mode || !expectedRaw || !geometryRaw) process.exit(2)
const data = fs.existsSync(path) ? fs.readFileSync(path) : Buffer.alloc(0)
const expected = Number(expectedRaw)
const expectedGeometries = parseGeometries(geometryRaw)
if (mode === "snapshots") {
validateSnapshots(decode(data, false), expectedGeometries, expected)
} else if (mode === "final") {
if (!stdoutPath || !stderrPath) process.exit(2)
validateFinal(data, expectedGeometries, fs.readFileSync(stdoutPath), fs.readFileSync(stderrPath))
console.log("PACKAGED-FD-GREEN-61e8")
console.log("INITIAL-SNAPSHOT-GREEN-61e8")
console.log("RECONNECT-SNAPSHOT-GREEN-61e8")
console.log("FRAMED-TERMINAL-STREAM-GREEN-61e8")
} else {
process.exit(2)
}
}
41 changes: 41 additions & 0 deletions cells/pty-attach-machine-stream/fixture/drop-proxy.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
#!/usr/bin/env node
import fs from "node:fs"
import net from "node:net"
import path from "node:path"

const [listenPath, targetPath, dropPath] = process.argv.slice(2)
if (!listenPath || !targetPath || !dropPath) process.exit(2)

const sockets = new Set()
const server = net.createServer((client) => {
const target = net.createConnection(targetPath)
sockets.add(client)
sockets.add(target)
client.pipe(target)
target.pipe(client)
const close = () => {
sockets.delete(client)
sockets.delete(target)
client.destroy()
target.destroy()
}
client.on("close", close)
target.on("close", close)
})

try { fs.unlinkSync(listenPath) } catch {}
server.listen(listenPath)
const watcher = fs.watch(path.dirname(dropPath), (_event, name) => {
if (name !== dropPath.split("/").at(-1) || !fs.existsSync(dropPath)) return
for (const socket of sockets) socket.destroy()
watcher.close()
server.close()
})

const stop = () => {
watcher.close()
for (const socket of sockets) socket.destroy()
server.close(() => process.exit(0))
}
process.on("SIGTERM", stop)
process.on("SIGINT", stop)
24 changes: 24 additions & 0 deletions cells/pty-attach-machine-stream/fixture/fabric
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
#!/usr/bin/env bash
set -euo pipefail

test "${1:-}" = dial
test "${2:-}" = eval-peer
test "${3:-}" = pty-remote
state="${PTY_EVAL_STATE:?PTY_EVAL_STATE must be set}"
mkdir -p "$state"

if mkdir "$state/first-dial" 2>/dev/null; then
printf '%s\n' "${PTY_EVAL_SOCKET_1:?PTY_EVAL_SOCKET_1 must be set}"
exit 0
fi

touch "$state/second-dial"
for _ in $(seq 1 160); do
if test -f "$state/release-second"; then
printf '%s\n' "${PTY_EVAL_SOCKET_2:?PTY_EVAL_SOCKET_2 must be set}"
exit 0
fi
sleep 0.05
done
echo "timed out waiting to release second dial" >&2
exit 1
Loading