Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 25 additions & 4 deletions integration-tests/test-helper.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,12 @@ function isProcessAlive(pid: number): boolean {
}
}

// How long the stand-in below stays alive after it is signalled. The real CLI
// traps SIGHUP and exits only once its own exit-cleanup chain has drained, so
// a stand-in that dies on the default action would let cleanup() return early
// and still look correct.
const STAND_IN_EXIT_DELAY_MS = 750;

describe('TestRig', () => {
const originalKeepOutput = process.env['KEEP_OUTPUT'];

Expand Down Expand Up @@ -63,7 +69,7 @@ describe('TestRig', () => {
expect(existsSync(testDir)).toBe(true);
});

it('kills an interactive session a test never closed during cleanup', async () => {
it('waits for an interactive session a test never closed to end', async () => {
// KEEP_OUTPUT is what CI sets, and it makes cleanup() keep the test
// directory — the spawned child must not survive that path either.
process.env['KEEP_OUTPUT'] = 'true';
Expand All @@ -72,15 +78,30 @@ describe('TestRig', () => {
// Stands in for the CLI bundle: what is under test is that cleanup ends
// whatever runInteractive spawned, not what the CLI itself does.
rig.bundlePath = rig.createFile(
'idle-cli.js',
'setInterval(() => {}, 1000);\n',
'slow-exit-cli.js',
'process.on("SIGHUP", () => setTimeout(() => process.exit(129), ' +
`${STAND_IN_EXIT_DELAY_MS}));\n` +
'setInterval(() => {}, 1000);\n' +
'process.stdout.write("STAND_IN_READY\\n");\n',
);

const { ptyProcess } = rig.runInteractive();
expect(isProcessAlive(ptyProcess.pid)).toBe(true);
// Signal before the handler is installed and the default action ends the
// child at once, measuring nothing. A real session is booted by the time
// its test ends, so wait for the stand-in to report itself up.
expect(await rig.waitForText('STAND_IN_READY', 30_000)).toBe(true);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] R1-6: This regression test assumes the bundle lane. On the installed-release lane (INTEGRATION_TEST_USE_INSTALLED_GEMINI=true), _getCommandAndArgs (integration-tests/test-helper.ts:292-303) returns command 'qwen' with no bundlePath, so runInteractive spawns the installed CLI and never executes the stand-in — STAND_IN_READY never prints, waitForText polls the full 30s, and the test fails red for a reason unrelated to what that lane validates. No CI lane sets this variable, so the breakage is confined to manual installed-release verification runs. Skip the test on that lane: it.skipIf(process.env.INTEGRATION_TEST_USE_INSTALLED_GEMINI === 'true')(...) (combinable with a win32 platform skip).

Witness:

probe, env var set, unmodified PR:
× waits for an interactive session a test never closed to end 30049ms
  → AssertionError: expected false to be true ❯ test-helper.test.ts:93
with the it.skipIf fix: 6 passed | 1 skipped (green)
bundle lane with the fix: 7 passed (the test is not skipped)

Note: the lane sentinel is process.env.INTEGRATION_TEST_USE_INSTALLED_GEMINI === 'true' at integration-tests/test-helper.ts:297 — the skip must key off that exact variable and value.

中文说明

该回归测试假定了 bundle 通道。在安装版通道(INTEGRATION_TEST_USE_INSTALLED_GEMINI=true)下,_getCommandAndArgsintegration-tests/test-helper.ts:292-303)返回命令 'qwen' 且不带 bundlePath,因此 runInteractive 启动的是已安装的 CLI,替身脚本根本不会执行 —— STAND_IN_READY 永不出现,waitForText 会轮询满 30 秒,测试以与该通道验证目标无关的原因失败。没有任何 CI 通道设置该变量,因此影响仅限手工的安装版验证。建议在该通道上跳过本测试:it.skipIf(process.env.INTEGRATION_TEST_USE_INSTALLED_GEMINI === 'true')(...)(可与 win32 平台跳过合并)。

注意:通道哨兵是 integration-tests/test-helper.ts:297 处的 process.env.INTEGRATION_TEST_USE_INSTALLED_GEMINI === 'true' —— 跳过条件必须精确使用该变量与取值。

— qwen3.8-max via Qwen Code /review (v0.23.0)


const cleanupStartedAt = Date.now();
await rig.cleanup();

const cleanupTookMs = Date.now() - cleanupStartedAt;

// Signalling alone returns straight through the delay above, leaving the
// child forwarding PTY bytes into a worker vitest is tearing down.
expect(
cleanupTookMs,
'cleanup() returned before the interactive CLI child exited',
).toBeGreaterThanOrEqual(STAND_IN_EXIT_DELAY_MS);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] R1-7: The duration assertion is lower-bounded only, so the test cannot tell "cleanup resolved because exited settled" from "cleanup fell through the full 10s grace" — the onExit wiring added in runInteractive is effectively unwitnessed. A mutation making exited never settle (new Promise<void>(() => {})) still passes green after 10209ms (961ms intact): broken exited wiring ships green, and every leaked-session cleanup would from then on silently cost the full grace per session. Add an upper bound — the stand-in exits 750ms after the signal, so a few seconds of slack stays far below the 10s grace.

Witness:

intact:  'waits for an interactive session a test never closed to end' 961ms ✓
mutant (exited never settles): the same test passes green at 10209ms ✓
— no assertion discriminates the two arms
Suggested change
).toBeGreaterThanOrEqual(STAND_IN_EXIT_DELAY_MS);
).toBeGreaterThanOrEqual(STAND_IN_EXIT_DELAY_MS);
expect(cleanupTookMs).toBeLessThan(5_000);

The same test is the fix witness: once the upper bound exists, the never-settling-exited mutation makes cleanup take ≥10s and the new assertion must go red.

中文说明

时长断言只有下限,因此测试无法区分“因 exited 落定而返回”与“等满 10 秒宽限后返回”—— runInteractive 新增的 onExit 接线实际上没有 witness。变异实验:让 exited 永不 settle(new Promise<void>(() => {})),测试仍会在 10209ms 后通过(完好时为 961ms):损坏的 exited 接线会静默上线,此后每次泄漏会话的清理都会默默消耗整个宽限期。建议增加上限 —— 替身在收到信号后 750ms 退出,留出数秒余量仍远低于 10 秒宽限。

同一测试即修复验收标准:上限加上之后,令 exited 永不 settle 的变异会使清理耗时 ≥10 秒,新断言必须变红。

— qwen3.8-max via Qwen Code /review (v0.23.0)

await expect
.poll(() => isProcessAlive(ptyProcess.pid), {
message: 'the interactive CLI child outlived cleanup()',
Expand Down
37 changes: 33 additions & 4 deletions integration-tests/test-helper.ts
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,26 @@ export function validateModelOutput(
return true;
}

// The CLI traps SIGHUP and exits only once `runExitCleanup()` has drained, a
// chain it bounds at 5s (packages/cli/src/utils/cleanup.ts). Waiting longer
// than that bound is what makes cleanup() return with the child actually gone.
const INTERACTIVE_EXIT_GRACE_MS = 10_000;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] R1-1: The 10-second per-session grace sits exactly on vitest's default hookTimeout, and cleanup() runs inside afterEach hooks. integration-tests/vitest.config.ts sets testTimeout but not hookTimeout, so the 10s default applies — whenever the grace fully expires, cleanup() consumes the entire hook budget plus the remaining test-dir removal, and vitest aborts the hook with a generic "Hook timed out" that blames the hook rather than the un-exited child. The still-alive child can then still EPIPE-crash the worker at teardown, so the failure this PR removes can recur masked behind a misleading timeout. Keep the grace strictly inside the hook budget (e.g. 8_000ms — still above the CLI's 5s bound), or set hookTimeout explicitly in integration-tests/vitest.config.ts with a comment tying it to this constant.

Witness:

probe, leaked SIGHUP-ignoring child, cleanup() called from afterEach:
PROBE_HOOK_OBSERVATION cleanup completed in 10012ms
Error: Hook timed out in 10000ms.   (3/3 attempts)
vitest default: resolved.hookTimeout ??= ... 1e4 (node_modules/vitest/dist/chunks/coverage.DfSpMS-b.js:3922)
Suggested change
const INTERACTIVE_EXIT_GRACE_MS = 10_000;
const INTERACTIVE_EXIT_GRACE_MS = 8_000;

Note: integration-tests/vitest.config.ts sets testTimeout but no hookTimeout, and vitest ^3.2.4's default hookTimeout is 10_000ms — any retained grace must stay under the hook budget or the hook must be raised explicitly.

中文说明

每会话 10 秒的宽限恰好等于 vitest 默认的 hookTimeout,而 cleanup() 运行在 afterEach 钩子里。integration-tests/vitest.config.ts 设置了 testTimeout 却没有设置 hookTimeout,因此适用 10 秒默认值 —— 宽限一旦耗尽,cleanup() 会吃满整个钩子预算外加测试目录清理,vitest 将以泛化的 "Hook timed out" 中止钩子,指向钩子而不是未退出的子进程。存活的子进程随后仍可能在 worker 拆除时引发 EPIPE 崩溃,即本 PR 要消除的故障会以误导性的超时形式复现。建议将宽限严格保持在钩子预算之内(如 8_000ms,仍高于 CLI 的 5 秒上限),或在 integration-tests/vitest.config.ts 中显式设置 hookTimeout 并用注释与本常量关联。

注意:integration-tests/vitest.config.ts 未设置 hookTimeout,vitest ^3.2.4 默认值为 10_000ms —— 保留的宽限必须低于钩子预算,否则需显式提高钩子预算。

— qwen3.8-max via Qwen Code /review (v0.23.0)


// Resolves when `promise` settles, or after `ms` if it never does. The timer
// is cleared and unrefed so a won race leaves no handle holding the worker's
// event loop open.
function settleWithin(promise: Promise<unknown>, ms: number): Promise<void> {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] R1-2: settleWithin is the repo's third promise-vs-setTimeout racer and a semantic twin of withTimeout in packages/cli/src/utils/cleanup.ts:44 — settle-on-timeout semantics that now live in independent copies across the production/test boundary, with a reject-on-timeout sibling at integration-tests/qwen-live-harness.ts:96. If the CLI's exit bound or timer hygiene ever changes, the twin here does not move with it, and the next teardown-timing fix (this class already recurred as #10969 then #10990) must rediscover and re-patch up to three sites. Neither existing helper is cleanly reusable — the exact twin is module-private production source, and the importable one rejects on timeout and drags a heavy import graph — so keeping the local 12-line helper is defensible under simplicity-first; this is awareness for whoever next touches either bound.

Witness:

witness: not run — quality/duplication claim settled by reading the three
definitions at the cited lines (cleanup.ts:44, qwen-live-harness.ts:96,
test-helper.ts:135); no run capability produces "future drift"

Note: OVERALL_CLEANUP_TIMEOUT_MS = 5_000 at packages/cli/src/utils/cleanup.ts:36 — any restructuring must keep INTERACTIVE_EXIT_GRACE_MS above the CLI's overall exit-cleanup bound, as the added comment above this function relies on.

中文说明

settleWithin 是仓库中第三个 promise 与 setTimeout 的竞速器,与 packages/cli/src/utils/cleanup.ts:44withTimeout 语义相同 —— “超时即安顿”的语义如今跨生产/测试边界存在独立拷贝,另有 integration-tests/qwen-live-harness.ts:96 的超时即 reject 版本。若 CLI 的退出上限或 timer 清理方式发生变化,这里的孪生实现不会同步,下一次 teardown 时序修复(此类问题已先后以 #10969#10990 复现)将不得不重新发现并修补多达三处。两个现有助手均不可干净复用 —— 完全同构的那份是生产代码的模块私有函数,可导入的那份超时即 reject 且拖入沉重的依赖图 —— 因此按简洁优先原则保留本地 12 行助手是可以辩护的;此条用于提醒后续触碰任一上限的人。

注意:packages/cli/src/utils/cleanup.ts:36OVERALL_CLEANUP_TIMEOUT_MS = 5_000 —— 任何重构都必须保持 INTERACTIVE_EXIT_GRACE_MS 高于 CLI 的整体退出清理上限(本函数上方的注释即依赖于此)。

— qwen3.8-max via Qwen Code /review (v0.23.0)

return new Promise((resolve) => {
const timer = setTimeout(resolve, ms);
timer.unref();
const settle = () => {
clearTimeout(timer);
resolve();
};
void promise.then(settle, settle);
});
}

// Simulates typing a string one character at a time to avoid paste detection.
export async function type(ptyProcess: pty.IPty, text: string) {
const delay = 5;
Expand Down Expand Up @@ -200,7 +220,10 @@ export class TestRig {
testName?: string;
_lastRunStdout?: string;
_interactiveOutput = '';
private readonly interactiveProcesses: pty.IPty[] = [];
private readonly interactiveProcesses: Array<{
ptyProcess: pty.IPty;
exited: Promise<unknown>;
}> = [];

constructor() {
this.bundlePath = join(__dirname, '..', 'dist/cli.js');
Expand Down Expand Up @@ -496,13 +519,16 @@ export class TestRig {
async cleanup() {
// A session a test never closed keeps its CLI child forwarding PTY bytes
// into this worker's stdout; after vitest tears the worker down those
// writes EPIPE and fail an otherwise all-green run (#10969).
for (const ptyProcess of this.interactiveProcesses.splice(0)) {
// writes EPIPE and fail an otherwise all-green run (#10969). Signalling
// alone still returns with the child alive and writing, so wait for it to
// actually go away (#10990).
for (const { ptyProcess, exited } of this.interactiveProcesses.splice(0)) {
try {
ptyProcess.kill();
} catch {
// Process may have already exited
}
await settleWithin(exited, INTERACTIVE_EXIT_GRACE_MS);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] R1-3: The timeout arm of settleWithin — the "bounded" half of "block until exited, within a bounded timeout" — is exercised by no test: every existing stand-in exits, so the setTimeout branch never runs. A mutation removing the timeout arm keeps the existing suite green (7/7 passed), while a leaked child that never exits hangs cleanup() forever under the mutant — a future edit that breaks the timer arm re-introduces an unbounded teardown hang in place of the EPIPE crash, and nothing goes red. Add a test whose stand-in installs a no-op SIGHUP handler and never exits, and assert rig.cleanup() still resolves within a bounded window (make the grace injectable to pass a small value, or accept the ~10s run).

Witness:

intact probe (SIGHUP-ignoring stand-in):
PROBE_OBSERVATION outcome=returned elapsedMs=10003 childAlive=true
mutant (timeout arm removed): test-helper.test.ts 7 passed (7), 958ms
probe flips: AssertionError: cleanup() must return even when the child never
exits: expected 'hung' to be 'returned'

Note: the bound must stay above the real CLI's exit-cleanup chain — INTERACTIVE_EXIT_GRACE_MS = 10_000 was chosen against the CLI's 5s runExitCleanup() bound (packages/cli/src/utils/cleanup.ts:36). The new test itself is the fix witness: it must assert cleanup() resolves despite a never-exiting child, and removing the timer arm must turn it red.

中文说明

settleWithin 的超时分支 —— “有界等待”中的“界” —— 没有任何测试覆盖:现有替身都会退出,setTimeout 分支从未执行。变异实验:移除超时分支后现有套件仍全绿(7/7 通过),而一个永不退出的泄漏子进程在该变异下会让 cleanup() 永远挂起 —— 未来若有人改坏 timer 分支,将以无界 teardown 挂起取代 EPIPE 崩溃且无任何测试变红。建议增加一个替身捕获 SIGHUP 但永不退出的测试,断言 rig.cleanup() 仍在有界时间内返回(可将宽限改为可注入以缩短测试时长,或接受约 10 秒的运行时间)。

注意:上限必须保持高于真实 CLI 的退出清理链 —— INTERACTIVE_EXIT_GRACE_MS = 10_000 是相对 CLI 的 5 秒 runExitCleanup() 上限(packages/cli/src/utils/cleanup.ts:36)选取的。新测试本身即是修复验收标准:它必须断言子进程永不退出时 cleanup() 仍能返回,且移除 timer 分支后该测试必须变红。

— qwen3.8-max via Qwen Code /review (v0.23.0)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] R1-4: When the 10s grace expires, settleWithin resolves and cleanup() returns with the child still alive — and nothing logs which child was given up on. A CLI child that outlives its own 5s exit-cleanup bound survives the grace, vitest tears the worker down, and the child keeps forwarding PTY bytes into it: the exact EPIPE failure class this PR fixes (#10990) recurring with zero output pointing at the expired wait. Oncall sees the identical pre-fix crash and must re-derive that cleanup() has a 10s give-up. Have settleWithin report which side won (e.g. resolve false/undefined on timeout) and emit console.warn with ptyProcess.pid in cleanup() when the grace expires:

const exitedInTime = await settleWithin(exited, INTERACTIVE_EXIT_GRACE_MS);
if (!exitedInTime) {
  console.warn(
    `interactive CLI child (pid ${ptyProcess.pid}) did not exit within `
    + `${INTERACTIVE_EXIT_GRACE_MS}ms; continuing cleanup`,
  );
}

Witness:

probe (SIGHUP-ignoring child):
PROBE_OBSERVATION outcome=returned elapsedMs=10003 childAlive=true
— no warning emitted by cleanup(); the timeout branch is a bare
  'const timer = setTimeout(resolve, ms);' with no log

Note: exited must stay resolve-only (integration-tests/test-helper.ts:973-975) — adding a reject path would become an unhandled rejection in every test that never hits it. Fix witness: a new case in test-helper.test.ts whose stand-in traps SIGHUP but never exits should assert cleanup() still resolves and the warning was emitted (spy on console.warn) — removing the warning branch must turn it red.

中文说明

10 秒宽限到期时,settleWithin 会 resolve、cleanup() 会在子进程仍存活的情况下返回 —— 且没有任何日志记录被放弃的是哪个子进程。一个超过自身 5 秒退出清理上限的 CLI 子进程会活过宽限期,随后 vitest 拆除 worker,子进程继续向其转发 PTY 字节:本 PR 修复的 EPIPE 故障类别(#10990)在毫无诊断输出的情况下复现。值班人员看到的是与修复前完全相同的崩溃,必须重新推导才知道 cleanup() 有 10 秒的放弃点。建议让 settleWithin 报告哪一侧获胜(如超时返回 false/undefined),并在宽限到期时由 cleanup() 输出带 ptyProcess.pidconsole.warn(见上方代码)。

注意:exited 必须保持只 resolve(integration-tests/test-helper.ts:973-975)—— 引入 reject 路径会在所有未触发它的测试中变成未处理 rejection。修复验收标准:test-helper.test.ts 中新增一个替身捕获 SIGHUP 但永不退出的用例,断言 cleanup() 仍会返回且警告已输出(spy console.warn)—— 移除警告分支后该测试必须变红。

— qwen3.8-max via Qwen Code /review (v0.23.0)

}

// Clean up test directory
Expand Down Expand Up @@ -944,7 +970,10 @@ export class TestRig {
...e2eRendererEnv(renderer),
} as { [key: string]: string },
});
this.interactiveProcesses.push(ptyProcess);
const exited = new Promise<void>((resolve) => {
ptyProcess.onExit(() => resolve());
});
this.interactiveProcesses.push({ ptyProcess, exited });

ptyProcess.onData((data) => {
this._interactiveOutput += data;
Expand Down