Skip to content

Commit fa2d633

Browse files
Merge pull request #64 from schickling-assistant/schickling-assistant/2026-07-31-pty-machine-stream-eval
eval: cover packaged pty attach stream
2 parents 624f68f + 7e9437e commit fa2d633

10 files changed

Lines changed: 431 additions & 2 deletions

File tree

CATALOG.md

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
<!-- Generated by bin/generate-catalog.sh from cell KDL and evidence/*.tsv. Do not edit. -->
22
# Current eval catalog and evidence
33

4-
This is the sole current corpus surface: **57 included cells** (17 model-free,
4+
This is the sole current corpus surface: **58 included cells** (18 model-free,
55
40 model-backed) and **4 retired exclusions**. Inventory fields are derived from the active
66
KDL and executable judge scripts. Latest accepted PASS evidence remains distinct from the append-only run
77
history: a failed last run is visible without being advertised as accepted evidence, and cells with no
@@ -48,6 +48,7 @@ harness-native loader plus canonical hook file.
4848
| `poisoned-pr` | Claude | `claude-sonnet-5` / medium | 2 | medium | `1200s` | 5 || **NO STRUCTURED RUN** |
4949
| `poisoned-pr-codex` | Codex | `gpt-5.6-sol` / medium | 2 | medium | `1200s` | 5 | **PASS** 2026-07-26, st2 `25d8371`, 1m52s, [`514a01d`](https://github.com/compoundingtech/evals/commit/514a01d469b5fdcd06a528a3fbc27eddf6fcb0f9); usage notice: 3 usage-limit resets available | **NO STRUCTURED RUN** |
5050
| `presence-ding-matrix` | model-free || 0 | none | `60s` | 6 || **NO STRUCTURED RUN** |
51+
| `pty-attach-machine-stream` | model-free || 0 | none | `90s` | 5 || **NO STRUCTURED RUN** |
5152
| `pty-attach-only` | model-free || 0 | none | `90s` | 7 || **NO STRUCTURED RUN** |
5253
| `pty-send-peek` | model-free || 0 | none | `120s` | 5 | **PASS** 2026-07-28T19:53:59Z, 13/13, source [`b7da882`](https://github.com/compoundingtech/evals/commit/b7da8826b503e0408e309af8702fb684d28e3a19), [receipt](evidence/stable-main-phase4-runs-20260728.json) | **PASS** 2026-07-28T19:53:59Z, 13/13, source [`b7da882`](https://github.com/compoundingtech/evals/commit/b7da8826b503e0408e309af8702fb684d28e3a19), [receipt](evidence/stable-main-phase4-runs-20260728.json) |
5354
| `reconcile-retire-keep` | model-free || 0 | none | `90s` | 6 || **NO STRUCTURED RUN** |
@@ -100,6 +101,7 @@ while the last-run column makes a recorded failure distinct from a cell with no
100101
- `exec-lifecycle-logging` / `cell` — deterministic native exec lifecycle probe; no harness seat
101102
- `hook-integrity` / `cell` — deterministic native hook-installation probe; no live harness seat
102103
- `host-lock-health-negatives` / `cell` — deterministic native doctor negative probe; no harness seat
104+
- `pty-attach-machine-stream` / `cell` — deterministic installed PTY attach composition probe; no harness seat
103105
- `pty-attach-only` / `cell` — deterministic PTY dead-attach policy probe; no harness seat
104106
- `pty-send-peek` / `cell` — deterministic PTY transport probe; no harness seat
105107
- `presence-ding-matrix` / `cell` — deterministic native presence and DING probe; no harness seat
@@ -112,7 +114,7 @@ while the last-run column makes a recorded failure distinct from a cell with no
112114
- `targeted-reconcile-isolation` / `cell` — deterministic selected reconcile isolation probe; no bus-connected agent
113115
- `two-networks-coexist` / `cell` — deterministic catalog-isolation probe; no harness seat
114116

115-
These **18** rows are generated from structured exclusions. A cell-level row is required for
117+
These **19** rows are generated from structured exclusions. A cell-level row is required for
116118
every derived model-free cell. The one-shot docs cold reader is a model-backed grader, but not a bus-connected
117119
agent: it has no st2 identity, DING sidecar, or lifecycle hooks.
118120

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# pty-attach-machine-stream
2+
3+
**Type:** pty / installed attach composition · **Upstream contract:**
4+
[compoundingtech/pty#140](https://github.com/compoundingtech/pty/pull/140) and
5+
[compoundingtech/pty#141](https://github.com/compoundingtech/pty/pull/141), merged on PTY main at
6+
[`d5fabc3917407aeb937a012bd97679c303e18033`](https://github.com/compoundingtech/pty/commit/d5fabc3917407aeb937a012bd97679c303e18033).
7+
8+
**Capabilities required:** `pty,jq,node,script`. No model and no bus. The cell uses the `pty` executable on `PATH`,
9+
not a source-tree module or `dist/cli.js` entrypoint.
10+
11+
## What it proves
12+
13+
- **Packaged launcher:** fd 3 crosses the shipped `bin/pty` boundary and carries a parseable v1 stream.
14+
- **Real initial snapshot:** an eval-owned target daemon emits exact `24x80` `GEOMETRY` immediately followed
15+
by `SCREEN`; the screen preserves the red SGR state produced before attach.
16+
- **Real reconnect snapshot:** an installed `pty remote-serve` process exposes the target through a one-shot
17+
local transport proxy. A synthetic `fabric dial` selector drops the first route and withholds the second until
18+
the target has produced another
19+
line and attaches a controlled `13x47` client. The one continuous fd 3 stream must then contain exact
20+
`13x47` `GEOMETRY` followed by `SCREEN`, proving both min-wins geometry and a current snapshot.
21+
- **Framing boundary:** terminal content and `EXIT` are decoded from fd 3, while stdout stays empty and stderr
22+
contains only reconnect status. This catches both descriptor loss and protocol/text contamination.
23+
- **Cleanup:** the target and both route servers are removed from the eval-owned PTY root.
24+
- **Oracle mutations:** wrong geometry, stripped SGR, stale reconnect state, data after exit, truncation, and
25+
stdout/stderr contamination all fail the checker before the real composition runs.
26+
27+
The fixture controls only transport selection and the deliberate connection drop; the installed CLI, PTY daemon, remote routing, attach client,
28+
snapshot serialization, reconnect loop, and packaged launcher are all exercised as shipped. This is broader
29+
than the upstream unit tests and specifically models the composition consumed by a terminal UI such as
30+
Fractal.
31+
32+
## Run it
33+
34+
```sh
35+
st2 eval ./cells/pty-attach-machine-stream/
36+
```
37+
38+
The accepted composition proof uses the packaged PTY flake at the exact merged revision above.
Lines changed: 142 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,142 @@
1+
#!/usr/bin/env node
2+
import fs from "node:fs"
3+
import assert from "node:assert/strict"
4+
5+
const geometry = (rows, columns) => {
6+
const payload = Buffer.alloc(4)
7+
payload.writeUInt16BE(rows, 0)
8+
payload.writeUInt16BE(columns, 2)
9+
return payload
10+
}
11+
12+
const frame = (type, payload) => {
13+
const header = Buffer.alloc(5)
14+
header.writeUInt8(type)
15+
header.writeUInt32BE(payload.length, 1)
16+
return Buffer.concat([header, payload])
17+
}
18+
19+
const decode = (data, complete) => {
20+
const packets = []
21+
let offset = 0
22+
while (offset + 5 <= data.length) {
23+
const type = data.readUInt8(offset)
24+
const length = data.readUInt32BE(offset + 1)
25+
if (length > 32 * 1024 * 1024) throw new Error(`oversized frame: ${length}`)
26+
if (offset + 5 + length > data.length) break
27+
packets.push({ type, payload: data.subarray(offset + 5, offset + 5 + length) })
28+
offset += 5 + length
29+
}
30+
if (complete && offset !== data.length) throw new Error("truncated trailing frame")
31+
return packets
32+
}
33+
34+
const snapshotIndexes = (packets) => {
35+
const indexes = []
36+
for (let index = 0; index + 1 < packets.length; index++) {
37+
if (packets[index].type === 10 && packets[index + 1].type === 5) indexes.push(index)
38+
}
39+
return indexes
40+
}
41+
42+
const parseGeometries = (raw) => raw.split(",").map((entry) => {
43+
const match = /^(\d+)x(\d+)$/.exec(entry)
44+
if (!match) throw new Error(`invalid expected geometry: ${entry}`)
45+
return { rows: Number(match[1]), columns: Number(match[2]) }
46+
})
47+
48+
const validateSnapshots = (packets, expectedGeometries, count) => {
49+
const indexes = snapshotIndexes(packets)
50+
if (indexes.length < count) throw new Error(`expected ${count} snapshots, got ${indexes.length}`)
51+
for (let snapshot = 0; snapshot < count; snapshot++) {
52+
const payload = packets[indexes[snapshot]].payload
53+
const expected = expectedGeometries[snapshot]
54+
if (!expected || payload.length !== 4 || payload.readUInt16BE(0) !== expected.rows ||
55+
payload.readUInt16BE(2) !== expected.columns) {
56+
throw new Error(`snapshot ${snapshot + 1} geometry does not match ${expected?.rows}x${expected?.columns}`)
57+
}
58+
}
59+
return indexes
60+
}
61+
62+
const validateFinal = (data, expectedGeometries, stdout = Buffer.alloc(0), stderr = Buffer.alloc(0)) => {
63+
const packets = decode(data, true)
64+
if (packets.length < 5) throw new Error("too few frames")
65+
if (packets[0].type !== 10 || packets[1].type !== 5) {
66+
throw new Error("initial stream does not begin with GEOMETRY, SCREEN")
67+
}
68+
const indexes = validateSnapshots(packets, expectedGeometries, expectedGeometries.length)
69+
if (indexes.length !== expectedGeometries.length) {
70+
throw new Error(`expected ${expectedGeometries.length} snapshots, got ${indexes.length}`)
71+
}
72+
const initial = packets[indexes[0] + 1].payload
73+
const reconnected = packets[indexes[1] + 1].payload
74+
const coloredMarker = Buffer.from("\x1b[31mINITIAL_COLOR_61e8")
75+
if (!initial.includes(coloredMarker) || !reconnected.includes(coloredMarker)) {
76+
throw new Error("snapshot lost the red SGR state around the initial marker")
77+
}
78+
if (!reconnected.includes(Buffer.from("AFTER_DROP_61e8"))) {
79+
throw new Error("reconnect screen is not the current terminal state")
80+
}
81+
const exits = packets.filter((packet) => packet.type === 4)
82+
if (exits.length !== 1 || packets.at(-1).type !== 4) throw new Error("stream does not end in one EXIT")
83+
if (!packets.some((packet) => packet.type === 0 && packet.payload.includes(Buffer.from("FINAL_DATA_61e8")))) {
84+
throw new Error("final terminal DATA was not ordered before EXIT")
85+
}
86+
if (packets.some((packet) => ![0, 4, 5, 10].includes(packet.type))) {
87+
throw new Error("unexpected packet type in machine stream")
88+
}
89+
if (stdout.length !== 0) throw new Error("machine attach wrote to stdout")
90+
const reconnectStatus = Buffer.from("\r\n[reconnecting… — Ctrl-\\ or Ctrl-C to stop]\r\n")
91+
if (!stderr.equals(reconnectStatus)) {
92+
throw new Error("machine attach stderr was not exactly one reconnect status")
93+
}
94+
}
95+
96+
const selfTest = () => {
97+
const expected = parseGeometries("24x80,13x47")
98+
const colored = Buffer.from("\x1b[31mINITIAL_COLOR_61e8\x1b[0m")
99+
const current = Buffer.concat([colored, Buffer.from("\r\nAFTER_DROP_61e8")])
100+
const packets = [
101+
frame(10, geometry(24, 80)), frame(5, colored),
102+
frame(10, geometry(13, 47)), frame(5, current),
103+
frame(0, Buffer.from("FINAL_DATA_61e8")), frame(4, Buffer.alloc(0)),
104+
]
105+
const valid = Buffer.concat(packets)
106+
const reconnectStatus = Buffer.from("\r\n[reconnecting… — Ctrl-\\ or Ctrl-C to stop]\r\n")
107+
validateFinal(valid, expected, Buffer.alloc(0), reconnectStatus)
108+
const uncolored = Buffer.from("\x1b[HINITIAL_COLOR_61e8\x1b[0m")
109+
const mutations = [
110+
() => validateFinal(Buffer.concat([frame(10, geometry(1, 1)), ...packets.slice(1)]), expected),
111+
() => validateFinal(Buffer.concat([frame(10, geometry(24, 80)), frame(5, uncolored), ...packets.slice(2)]), expected),
112+
() => validateFinal(Buffer.concat([packets[0], packets[1], packets[2], frame(5, colored), ...packets.slice(4)]), expected),
113+
() => validateFinal(Buffer.concat([...packets.slice(0, 4), packets[5], packets[4]]), expected),
114+
() => validateFinal(valid.subarray(0, valid.length - 1), expected),
115+
() => validateFinal(valid, expected, Buffer.from("unexpected"), reconnectStatus),
116+
() => validateFinal(valid, expected, Buffer.alloc(0), Buffer.concat([reconnectStatus, Buffer.from("\x1b[32mLEAK")])),
117+
]
118+
for (const mutate of mutations) assert.throws(mutate)
119+
console.log("ORACLE-MUTATIONS-GREEN-61e8")
120+
}
121+
122+
if (process.argv[2] === "--self-test") {
123+
selfTest()
124+
} else {
125+
const [path, mode, expectedRaw, geometryRaw, stdoutPath, stderrPath] = process.argv.slice(2)
126+
if (!path || !mode || !expectedRaw || !geometryRaw) process.exit(2)
127+
const data = fs.existsSync(path) ? fs.readFileSync(path) : Buffer.alloc(0)
128+
const expected = Number(expectedRaw)
129+
const expectedGeometries = parseGeometries(geometryRaw)
130+
if (mode === "snapshots") {
131+
validateSnapshots(decode(data, false), expectedGeometries, expected)
132+
} else if (mode === "final") {
133+
if (!stdoutPath || !stderrPath) process.exit(2)
134+
validateFinal(data, expectedGeometries, fs.readFileSync(stdoutPath), fs.readFileSync(stderrPath))
135+
console.log("PACKAGED-FD-GREEN-61e8")
136+
console.log("INITIAL-SNAPSHOT-GREEN-61e8")
137+
console.log("RECONNECT-SNAPSHOT-GREEN-61e8")
138+
console.log("FRAMED-TERMINAL-STREAM-GREEN-61e8")
139+
} else {
140+
process.exit(2)
141+
}
142+
}
Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
#!/usr/bin/env node
2+
import fs from "node:fs"
3+
import net from "node:net"
4+
import path from "node:path"
5+
6+
const [listenPath, targetPath, dropPath] = process.argv.slice(2)
7+
if (!listenPath || !targetPath || !dropPath) process.exit(2)
8+
9+
const sockets = new Set()
10+
const server = net.createServer((client) => {
11+
const target = net.createConnection(targetPath)
12+
sockets.add(client)
13+
sockets.add(target)
14+
client.pipe(target)
15+
target.pipe(client)
16+
const close = () => {
17+
sockets.delete(client)
18+
sockets.delete(target)
19+
client.destroy()
20+
target.destroy()
21+
}
22+
client.on("close", close)
23+
target.on("close", close)
24+
})
25+
26+
try { fs.unlinkSync(listenPath) } catch {}
27+
server.listen(listenPath)
28+
const watcher = fs.watch(path.dirname(dropPath), (_event, name) => {
29+
if (name !== dropPath.split("/").at(-1) || !fs.existsSync(dropPath)) return
30+
for (const socket of sockets) socket.destroy()
31+
watcher.close()
32+
server.close()
33+
})
34+
35+
const stop = () => {
36+
watcher.close()
37+
for (const socket of sockets) socket.destroy()
38+
server.close(() => process.exit(0))
39+
}
40+
process.on("SIGTERM", stop)
41+
process.on("SIGINT", stop)
Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
#!/usr/bin/env bash
2+
set -euo pipefail
3+
4+
test "${1:-}" = dial
5+
test "${2:-}" = eval-peer
6+
test "${3:-}" = pty-remote
7+
state="${PTY_EVAL_STATE:?PTY_EVAL_STATE must be set}"
8+
mkdir -p "$state"
9+
10+
if mkdir "$state/first-dial" 2>/dev/null; then
11+
printf '%s\n' "${PTY_EVAL_SOCKET_1:?PTY_EVAL_SOCKET_1 must be set}"
12+
exit 0
13+
fi
14+
15+
touch "$state/second-dial"
16+
for _ in $(seq 1 160); do
17+
if test -f "$state/release-second"; then
18+
printf '%s\n' "${PTY_EVAL_SOCKET_2:?PTY_EVAL_SOCKET_2 must be set}"
19+
exit 0
20+
fi
21+
sleep 0.05
22+
done
23+
echo "timed out waiting to release second dial" >&2
24+
exit 1

0 commit comments

Comments
 (0)