Skip to content

eval: cover packaged pty attach stream - #64

Merged
schickling-assistant merged 4 commits into
compoundingtech:mainfrom
schickling-assistant:schickling-assistant/2026-07-31-pty-machine-stream-eval
Jul 31, 2026
Merged

eval: cover packaged pty attach stream#64
schickling-assistant merged 4 commits into
compoundingtech:mainfrom
schickling-assistant:schickling-assistant/2026-07-31-pty-machine-stream-eval

Conversation

@schickling-assistant

@schickling-assistant schickling-assistant commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Why

Terminal UIs need the installed PTY command to expose geometry and terminal bytes as one ordered machine stream. Source-level client tests do not prove that the packaged launcher preserves an inherited descriptor, or that reconnect snapshots remain ordered through the complete remote attach composition.

What

Adds a model-free cell that drives the pty executable on PATH through a real target daemon, installed remote-serve, routed attach, controlled transport drop, reconnect, and exit.

The held-out checks require:

  • fd 3 survives the shipped bin/pty launcher;
  • initial attach begins with adjacent GEOMETRY, SCREEN frames;
  • the same continuous stream begins the replacement route with a fresh GEOMETRY, current SCREEN pair;
  • colored screen state, post-drop state, final DATA, and one trailing EXIT remain framed;
  • terminal bytes never leak to stdout or stderr;
  • every eval-owned process and PTY session is cleaned up.

Rationale

The fixture controls only which local route a synthetic fabric dial returns and when the first transport connection is dropped. It does not import PTY source modules or invoke dist/cli.js. The installed launcher, daemon, remote routing, attach client, snapshot serialization, reconnect loop, and protocol decoder boundary are exercised together.

This specifically models the composition a terminal UI such as Fractal consumes, without mechanically copying the upstream unit suite.

Dependencies and proof

Depends on merged PTY main containing compoundingtech/pty#140 and compoundingtech/pty#141 at d5fabc3917407aeb937a012bd97679c303e18033.

Focused proof against the package built from exact merged PTY commit d5fabc3917407aeb937a012bd97679c303e18033:

  • packaged merged PTY main at d5fabc3: SCORE: 7 PASS / 0 FAIL / 7 gating judges
  • installed pre-feature pty 0.12.0: expected red; the launcher rejects the fd option and emits no machine frames
  • semantic load: 58 maintained folder evals, 11 nested fleet catalogs
  • exact 24x80 to 13x47 min-wins geometry, red SGR preservation, current reconnect state, final DATA/EXIT, and clean side channels: pass
  • deterministic oracle mutations for wrong geometry, stripped SGR, stale state, DATA after EXIT, truncation, and stdout/stderr contamination: pass
  • fixture reset, shell/Node syntax, preflight safety, closed-set mutations, model policy, catalog generation, and diff checks: pass

The composed cell, VRS check, catalog generation, and complete model-free corpus preflight are green against the merged package and the corpus-pinned st2 0fed14b release. The published st2 archive and extracted binary matched their documented SHA256 identities; no gate was weakened or skipped.

Posted on behalf of @schickling
field value
agent_name co1-sage
agent_session_id ed878dac-3735-4276-b3e0-ea1b1cd65291
agent_tool Codex CLI
agent_tool_version 0.145.0
agent_runtime Codex CLI 0.145.0
agent_model unknown
runtime_profile /nix/store/mnx8agbdq3wiyb6vz63lhgscgazkrn98-coding-agent-runtime-profile/share/coding-agents/profile.json
skills_manifest /nix/store/5r69m9k2llmri3na81518zx0a7y0d3cn-agent-skills-corpus/share/agent-skills/manifest.json
worktree evals/schickling-assistant/2026-07-31-pty-machine-stream-eval
machine dev3
tooling_profile dotfiles@0fb7e03

agent-session-id: ed878dac-3735-4276-b3e0-ea1b1cd65291
agent-tool: Codex CLI
agent-tool-version: 0.145.0
agent-model: unknown
agent-runtime-profile: /nix/store/mnx8agbdq3wiyb6vz63lhgscgazkrn98-coding-agent-runtime-profile/share/coding-agents/profile.json
agent-skills-manifest: /nix/store/5r69m9k2llmri3na81518zx0a7y0d3cn-agent-skills-corpus/share/agent-skills/manifest.json
tooling-profile: dotfiles@0fb7e03
agent-session-id: ed878dac-3735-4276-b3e0-ea1b1cd65291
agent-tool: Codex CLI
agent-tool-version: 0.145.0
agent-model: unknown
agent-runtime-profile: /nix/store/mnx8agbdq3wiyb6vz63lhgscgazkrn98-coding-agent-runtime-profile/share/coding-agents/profile.json
agent-skills-manifest: /nix/store/5r69m9k2llmri3na81518zx0a7y0d3cn-agent-skills-corpus/share/agent-skills/manifest.json
tooling-profile: dotfiles@0fb7e03
@schickling-assistant
schickling-assistant force-pushed the schickling-assistant/2026-07-31-pty-machine-stream-eval branch from 1879000 to 62be35b Compare July 31, 2026 12:49
Assert exact initial and reconnect geometry, preserved SGR state, ordered exit, and clean side channels. Add deterministic mutations for each failure boundary.

agent-session-id: ed878dac-3735-4276-b3e0-ea1b1cd65291
agent-tool: Codex CLI
agent-tool-version: 0.145.0
agent-model: unknown
agent-runtime-profile: /nix/store/mnx8agbdq3wiyb6vz63lhgscgazkrn98-coding-agent-runtime-profile/share/coding-agents/profile.json
agent-skills-manifest: /nix/store/5r69m9k2llmri3na81518zx0a7y0d3cn-agent-skills-corpus/share/agent-skills/manifest.json
tooling-profile: dotfiles@0fb7e03
Require exactly one deterministic reconnect status and reject unrelated ANSI output through the mutation gate.

agent-session-id: ed878dac-3735-4276-b3e0-ea1b1cd65291
agent-tool: Codex CLI
agent-tool-version: 0.145.0
agent-model: unknown
agent-runtime-profile: /nix/store/mnx8agbdq3wiyb6vz63lhgscgazkrn98-coding-agent-runtime-profile/share/coding-agents/profile.json
agent-skills-manifest: /nix/store/5r69m9k2llmri3na81518zx0a7y0d3cn-agent-skills-corpus/share/agent-skills/manifest.json
tooling-profile: dotfiles@0fb7e03
@schickling-assistant
schickling-assistant marked this pull request as ready for review July 31, 2026 13:06
@schickling-assistant
schickling-assistant merged commit fa2d633 into compoundingtech:main Jul 31, 2026
@schickling-assistant

Copy link
Copy Markdown
Contributor Author

Thank you, @nathanherald — the ordered packaged attach-stream contract is merged and now has a durable composition eval with exact geometry, color, ordering, and side-channel mutation coverage. Excellent foundation.

Celebration

Posted on behalf of @schickling
field value
agent_name co1-sage
agent_session_id ed878dac-3735-4276-b3e0-ea1b1cd65291
agent_tool Codex CLI
agent_tool_version 0.145.0
agent_runtime Codex CLI 0.145.0
agent_model unknown
runtime_profile /nix/store/mnx8agbdq3wiyb6vz63lhgscgazkrn98-coding-agent-runtime-profile/share/coding-agents/profile.json
skills_manifest /nix/store/5r69m9k2llmri3na81518zx0a7y0d3cn-agent-skills-corpus/share/agent-skills/manifest.json
worktree evals/schickling-assistant/2026-07-31-pty-machine-stream-eval
machine dev3
tooling_profile dotfiles@0fb7e03

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant