Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 63 additions & 0 deletions integration-tests/test-helper.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,11 @@ function isProcessAlive(pid: number): boolean {
}
}

const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));

/** Emitted by the stand-in CLI below; nothing else in the run writes it. */
const FORWARD_CANARY = 'PTY_FORWARD_CANARY_11002';

describe('TestRig', () => {
const originalKeepOutput = process.env['KEEP_OUTPUT'];

Expand Down Expand Up @@ -89,6 +94,64 @@ describe('TestRig', () => {
.toBe(false);
});

it("detaches a session's output forwarding during cleanup", async () => {
// KEEP_OUTPUT is what the OpenTUI leg sets, and it is what makes the rig
// forward every PTY byte into this worker's stdout.
process.env['KEEP_OUTPUT'] = 'true';
const rig = new TestRig();
await rig.setup('cleanup detaches interactive output');
// Stands in for the CLI bundle, which traps the SIGHUP that node-pty's
// signal-less kill() sends and keeps rendering while its exit cleanup
// drains. The child therefore outlives cleanup() on purpose and keeps
// producing bytes: what is under test is whether the harness still
// forwards them into a stdout pipe vitest is about to destroy (#11002).
rig.bundlePath = rig.createFile(
'slow-exit-cli.js',
[
"process.on('SIGHUP', () => {});",
`setInterval(() => process.stdout.write('${FORWARD_CANARY}\\n'), 20);`,
'setTimeout(() => process.exit(0), 30000);',
'',
].join('\n'),
);

const { ptyProcess } = rig.runInteractive();
try {
await expect
.poll(() => rig._interactiveOutput.includes(FORWARD_CANARY), {
message: 'the stand-in CLI never produced output',
timeout: 20_000,
})
.toBe(true);

const forwarded: string[] = [];
vi.spyOn(process.stdout, 'write').mockImplementation((chunk) => {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] R1-1: This witness test has no positive control — it never asserts that stdout forwarding was actually live before cleanup, so it can pass vacuously if the forwarding branch is ever silently disabled. The first poll pins rig._interactiveOutput, which is appended unconditionally BEFORE the KEEP_OUTPUT gate (test-helper.ts:960-963), so it proves only that the onData handler fired — never that the gated process.stdout.write(data) ran. The spy is installed only after that poll, and forwarded.length = 0 discards whatever it captured before cleanup, so a future one-line regression — deleting process.stdout.write(data) or inverting the gate — produces exactly the empty output the post-cleanup assertion expects, and the suite's only guard over the #11002 fix passes while the path it guards is dead.

Witness:

intact tree:   "✓ detaches a session's output forwarding during cleanup 556ms"
mutant (process.stdout.write(data) deleted): "✓ ... 558ms" — identical pass, forwarding dead
mutant + positive control: "× AssertionError: stdout forwarding never went live before cleanup: expected false to be true"

Suggested fix — insert a positive control after the spy block, before cleanup:

await expect
  .poll(() => forwarded.some((chunk) => chunk.includes(FORWARD_CANARY)), {
    message: 'stdout forwarding never went live before cleanup',
    timeout: 20_000,
  })
  .toBe(true);

The fix rests on the gate if (env.KEEP_OUTPUT === 'true' || env.VERBOSE === 'true') (integration-tests/test-helper.ts:961-963) where env is the live process.env (test-helper.ts:12) — the positive control must exercise that same gate, not a locally re-implemented condition. The positive-control assertion is its own pin: deleting process.stdout.write(data) (test-helper.ts:962) or inverting the KEEP_OUTPUT gate turns it red instead of the test passing silently.

中文说明

该见证测试缺少阳性对照——它从未断言清理之前 stdout 转发确实处于启用状态,因此一旦转发分支被静默禁用,测试就会空洞地通过。第一个 expect.poll 钉住的是 rig._interactiveOutput,而它是在 KEEP_OUTPUT 门(test-helper.ts:960-963)之前无条件追加的,所以它只能证明 onData 回调触发过,无法证明受门控的 process.stdout.write(data) 真的执行过。spy 在该 poll 之后才安装,且 forwarded.length = 0 会丢弃清理前捕获到的所有内容——于是未来任何一行回归(删除 process.stdout.write(data) 或反转门条件)都会恰好产生清理后断言所期望的空数组,这个套件中针对 #11002 修复的唯一守卫会在其所守护的路径已经失效时依然通过。

建议修复:在清理之前加入阳性对照——安装 spy 后先轮询直到观察到转发(上方代码块),然后再清零 forwarded、执行 rig.cleanup(),并保留现有的清理后空数组断言。

修复约束:所要断言的门是 if (env.KEEP_OUTPUT === 'true' || env.VERBOSE === 'true')(integration-tests/test-helper.ts:961-963),其中 env 是实时的 process.env(test-helper.ts:12)——阳性对照必须真实经过同一个门,而不是本地重新实现的条件。修复验收标准:阳性对照断言本身即是钉桩——删除 process.stdout.write(data)(test-helper.ts:962)或反转 KEEP_OUTPUT 门会使其变红,而不再是测试静默通过。

— qwen3.8-max via Qwen Code /review (v0.23.0)

forwarded.push(String(chunk));
return true;
});

await rig.cleanup();
// Still alive: it swallowed SIGHUP. That is the window the real CLI's
// graceful shutdown opens between cleanup() and the child's own exit.
expect(isProcessAlive(ptyProcess.pid)).toBe(true);

forwarded.length = 0;
await sleep(500);
Comment on lines +138 to +139

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] R1-4: The witness certifies the producer child is alive only at the instant cleanup() returns, never that it stays alive and keeps writing during the 500ms observation window — so the empty-forwarded assertion cannot distinguish "forwarding was detached" from "the producer died". On any lane where the child dies shortly after cleanup — a future node-pty whose signal-less kill() delivers a fatal signal instead of the trapped SIGHUP, a runtime where the SIGHUP handler is not installed before the signal lands, or an OOM kill on a loaded runner — forwarded stays empty even if session.forwardOutput.dispose() were deleted, because a dead child emits no bytes, and the #11002 regression witness ships green while certifying nothing. This is distinct from R1-1: the pre-cleanup canary control does not close this hole.

Witness:

intact tree: PASS (0 canary chunks in window)
mutant (dispose removed), producer alive: 25 canary chunks → FAIL (discriminates on today's lane)
mutant (dispose removed), producer SIGKILLed after T0: 0 chunks → PASS as written (vacuous green), 5/5 runs
same mutant + post-window isProcessAlive re-check: FAIL (flips red)
Suggested change
forwarded.length = 0;
await sleep(500);
forwarded.length = 0;
await sleep(500);
expect(isProcessAlive(ptyProcess.pid)).toBe(true);

The re-check must use the same isProcessAlive helper as the existing liveness check at test-helper.test.ts:136, which asserts life at cleanup return only. The new assertion is its own pin: SIGKILL the child immediately after await rig.cleanup() returns — with the re-check added the test goes red; today that mutant passes.

中文说明

该见证测试只在 cleanup() 返回的那一刻证明了子进程存活,从未证明它在 500ms 观察窗口内持续存活并持续输出——因此"转发已被摘除"与"子进程已死亡"在这条空数组断言下无法区分。在任何子进程会在清理后不久死亡的通道上(未来某个 node-pty 的无信号 kill() 投递致命信号而非被捕获的 SIGHUP、SIGHUP 处理器未在信号到达前完成安装的运行时、负载较高的共享 runner 上的 OOM kill),即使从 cleanup() 中删除 session.forwardOutput.dispose()forwarded 依然为空——死掉的子进程不会发出任何字节,#11002 的回归见证测试将在未证明任何东西的情况下绿灯通过。这与 R1-1 不同:清理前的 canary 对照无法堵上这个洞。

建议修复:在窗口之后重新断言子进程存活(上方 suggestion 块;若偶发不稳,可加宽窗口并在窗口末尾复查,而不是去掉该检查)。

修复约束:复查必须使用与现有存活检查(test-helper.test.ts:136)相同的 isProcessAlive 辅助函数——该现有检查只断言清理返回时刻的存活。修复验收标准:新增断言本身就是钉桩——在 await rig.cleanup() 返回后立即 SIGKILL 子进程:加上复查后测试变红;当前该变异可以通过。

— qwen3.8-max via Qwen Code /review (v0.23.0)


expect(
forwarded.filter((chunk) => chunk.includes(FORWARD_CANARY)),
'cleanup() left the session forwarding PTY bytes into stdout',
).toEqual([]);
} finally {
vi.restoreAllMocks();
try {
process.kill(ptyProcess.pid, 'SIGKILL');
} catch {
// Already gone
}
}
});

it.each([
[
'telemetry events',
Expand Down
23 changes: 17 additions & 6 deletions integration-tests/test-helper.ts
Original file line number Diff line number Diff line change
Expand Up @@ -200,7 +200,12 @@ export class TestRig {
testName?: string;
_lastRunStdout?: string;
_interactiveOutput = '';
private readonly interactiveProcesses: pty.IPty[] = [];
// Each interactive PTY paired with the listener forwarding its bytes to
// this worker's stdout, so cleanup() can detach that forwarding.
private readonly interactiveSessions: Array<{
ptyProcess: pty.IPty;
forwardOutput: pty.IDisposable;
}> = [];

constructor() {
this.bundlePath = join(__dirname, '..', 'dist/cli.js');
Expand Down Expand Up @@ -497,9 +502,16 @@ export class TestRig {
// A session a test never closed keeps its CLI child forwarding PTY bytes
// into this worker's stdout; after vitest tears the worker down those
// writes EPIPE and fail an otherwise all-green run (#10969).
for (const ptyProcess of this.interactiveProcesses.splice(0)) {
//
// Killing is not enough: node-pty's signal-less kill() sends SIGHUP, which
// the CLI traps into an asynchronous graceful shutdown, so the child still
// renders after cleanup() returns and one more forwarded byte can reach a
// stdout pipe vitest has since destroyed. Detaching the listener closes
// that window however long the child takes to die (#11002).
for (const session of this.interactiveSessions.splice(0)) {
session.forwardOutput.dispose();
try {
ptyProcess.kill();
session.ptyProcess.kill();
} catch {
// Process may have already exited
}
Expand Down Expand Up @@ -944,14 +956,13 @@ export class TestRig {
...e2eRendererEnv(renderer),
} as { [key: string]: string },
});
this.interactiveProcesses.push(ptyProcess);

ptyProcess.onData((data) => {
const forwardOutput = ptyProcess.onData((data) => {
this._interactiveOutput += data;
if (env.KEEP_OUTPUT === 'true' || env.VERBOSE === 'true') {
process.stdout.write(data);
}
});
this.interactiveSessions.push({ ptyProcess, forwardOutput });

const promise = new Promise<{
exitCode: number;
Expand Down
Loading