Skip to content

test: isolate runtime state and PTY integration - #175

Merged
Ark0N merged 2 commits into
Ark0N:masterfrom
Lint111:agent/split-quick-start-fixture
Jul 31, 2026
Merged

test: isolate runtime state and PTY integration#175
Ark0N merged 2 commits into
Ark0N:masterfrom
Lint111:agent/split-quick-start-fixture

Conversation

@Lint111

@Lint111 Lint111 commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Give the quick-start API suite its own temporary HOME.
  • Install a suite-wide temporary HOME / USERPROFILE and explicit VITEST
    marker before application imports can resolve runtime paths.
  • Preserve the platform Playwright browser cache while isolating all Codeman
    state, then restore the caller environment and remove only the temporary
    fixture directory during teardown.
  • Attach a deterministic live echo PTY during Vitest runs and skip real
    tmux/agent geometry calls, so integration tests exercise input/output without
    creating or controlling host sessions.

Root cause

The test harness did not reliably identify itself as a test environment and
allowed module-level paths to resolve from the developer's real home directory.
The quick-start suite could therefore clean up the real
~/codeman-cases/testcase directory, including the checkout running Vitest.
Session integration tests could also launch real tmux/agent processes.

Scope

This is the test-infrastructure slice extracted from #173. It changes the
quick-start fixture, global test setup, and a VITEST-guarded session adapter.
Normal application execution is unchanged.

Validation

  • npm run test:ci -- test/quick-start.test.ts test/integration-flows.test.ts:
    2 files, 23 tests passed
  • Working tree and git diff --check: clean

@Lint111 Lint111 changed the title test: isolate quick-start case fixtures test: isolate runtime state and PTY integration Jul 29, 2026
@Lint111
Lint111 marked this pull request as ready for review July 29, 2026 17:06
Copilot AI review requested due to automatic review settings July 29, 2026 17:06

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@Ark0N Ark0N left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: Approve. This fixes two real, verified safety holes in the test harness. I reproduced the author's validation and additionally ran the full CI sweep that hadn't been run yet (details below). Recommend merging once GitHub CI has run on the branch.

What it does (3 files, +81/-19)

  • src/session.ts: gates four real-tmux touchpoints behind IS_TEST_MODE (process.env.VITEST): skips setManualWindowSize and queryTmuxWindowSize, skips resizeWindow on resize, and replaces the PTY attach with a deterministic echo transport (node -e 'process.stdin.pipe(process.stdout);') so integration tests get a live input/output loop without touching the host tmux server.
  • test/quick-start.test.ts: gives the suite its own temp HOME and switches to a lazy dynamic import of server.js (the static import is now type-only), so CASES_DIR resolves into the temp home.
  • test/setup.ts: global per-test-file temp HOME/USERPROFILE, an explicit VITEST marker, Playwright browser-cache path preservation, and full restore plus temp-dir removal in afterAll.

The bugs it fixes are real (verified in source)

  • TmuxManager has ~20 IS_TEST_MODE gates, but setManualWindowSize() and resizeWindow() are not gated (src/tmux-manager.ts:3150, :3169), and getAttachCommand()/getAttachArgs() return a real tmux -L <socket> attach-session -t <name>. So before this PR, any test that started an interactive session spawned a real tmux client against the live shared socket. The CLAUDE.md claim that "tests physically cannot interact with real tmux" was false for exactly these paths.
  • The old quick-start suite computed CASES_DIR = join(homedir(), 'codeman-cases') and did rmSync(..., { recursive: true, force: true }) on case names including the default testcase, i.e. it deleted from the real cases tree. The lazy import matters because src/web/route-helpers.ts:29 computes CASES_DIR at module load, so redirecting HOME after import would silently not work. The PR sequences this correctly.

Verification (worktree at PR head bba3d80)

  • tsc --noEmit: pass. prettier --check on all three files: pass.
  • Author's claimed validation reproduces: 23/23 across the two suites.
  • Full npm run test:ci sweep: 187 files passed (1 skipped), 3786 tests passed (12 skipped), zero failures. This was the main open risk, since the setup.ts change affects every suite.
  • Live tmux -L codeman socket snapshot before vs. after the 3786-test sweep, on a machine with 4 live production sessions: byte-identical. The isolation demonstrably works.
  • os.homedir() follows $HOME on Linux (mechanism confirmed); USERPROFILE covers Windows.
  • Cleanly mergeable, 7 commits behind master, zero drift in the touched files since its base.

Findings and suggestions (none blocking)

  1. Echo-PTY output is doubled. Empirically, writing hello\r yields "hello\r\nhello\r\n": once from tty line-discipline echo, once from the pipe. Input without \r comes back only via tty echo (canonical mode buffers the pipe until newline). Today's assertions (buffer.length > 0) are fine, but a future "output contains X exactly once" assertion will mysteriously fail. Suggest process.stdin.setRawMode(true); process.stdin.pipe(process.stdout); as the script, which gives exactly-once, unbuffered echo.
  2. The gates arguably belong in TmuxManager, not Session. The project convention is that the mux layer self-mocks under VITEST, and the root cause here is precisely the two ungated methods plus the real attach command in that layer. Gating there (and returning the echo command from getAttachCommand/Args in test mode) would fix all callers, keep session.ts free of test branches, and make the documented invariant true again. Also, IS_TEST_MODE is now duplicated in a second file, a small drift risk. Reasonable follow-up rather than a rework; the shipped approach works and mirrors the existing pattern.
  3. Residual hole: the direct-PTY fallback paths (pty.spawn('claude', ...) at src/session.ts:1589 and :2025, pty.spawn(shell, ...) at :1919) are still ungated. They are only reachable in tests if the mux path throws and falls back, so this is theoretical, but "tests cannot launch agent processes" still has that asterisk.
  4. quick-start's private temp HOME is redundant in the normal path (setup.ts already provides one per file), but it is justified belt-and-braces: raw npx vitest without --config skips setup.ts entirely, and the in-file fixture keeps this suite safe even then. Keep it. Minor: its ORIGINAL_HOME actually captures setup.ts's temp home, not the real one, so its "restore" is cosmetic; harmless since the worker exits.
  5. One empty temp dir leaks per fully-skipped test file (setup's module-level mkdtempSync runs but afterAll never fires when zero tests execute; observed exactly 1 leak matching the 1 skipped file). Trivial; a process.on('exit') cleanup would close it, or just accept it.
  6. Windows nit: the Playwright fallback when LOCALAPPDATA is unset should be join(originalHome, 'AppData', 'Local', 'ms-playwright'). Cosmetic, since LOCALAPPDATA is essentially always set and CI is Linux.
  7. Docs follow-up: CLAUDE.md's Testing section still describes safety as "TmuxManager no-ops all shell commands" only; the new temp-HOME and echo-PTY behavior deserves a line there (the PR did properly update setup.ts's fileoverview).

Risks

  • No CI has run on this PR yet (workflows pending approval). My local run used the exact CI config, but the real gate should run before merge.
  • Production behavior: unchanged. Every new branch is dead unless VITEST is set, which the production service never sets. Security posture is a strict improvement: tests can no longer mutate the live tmux socket or delete real user directories.

@Ark0N
Ark0N merged commit e8681f5 into Ark0N:master Jul 31, 2026
Ark0N pushed a commit that referenced this pull request Jul 31, 2026
…abilize CI teardown

Follow-ups from the PR #175/#176 reviews:

- Rewake helper self-terminates on its own 6h deadline and when orphaned,
  instead of relying on Claude Code to reap the poller
- Rewake marker versioned (V2) with a version-agnostic ownership prefix, so
  future script updates replace older handlers instead of duplicating them;
  regression test covers the V1 to V2 swap
- HOOK_TIMEOUT_MS renamed to HOOK_TIMEOUT_SECONDS = 10: the hook timeout
  field is seconds (the CLI multiplies by 1000), so the curl hooks have
  effectively had a ~2.8h timeout since COD-54
- Test echo PTY switches to raw mode: each input byte echoes exactly once
  (tty line discipline doubled every line and buffered until Enter)
- test/setup.ts: drain in-flight console-log rpc forwards before environment
  teardown (fixes the EnvironmentTeardownError that failed CI twice on the
  merge commit with all 3820 tests passing), clean the temp home on process
  exit (fully-skipped files leaked it), fix the Windows Playwright cache
  fallback path
- test/webview-proxy.test.ts: stop naming the vitest environment directive in
  prose; vitest matches it inside comments and silently ran the whole file
  under the jsdom environment while the comment claimed node
- CLAUDE.md: document the temp-HOME and echo-PTY test isolation

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants