fix(daemon): skip on-disk sweep for runs still live in the in-memory map (#6202 @mrcfps follow-up)

@mrcfps 7-30 non-blocking follow-up评审指出:purgeRunsForProject 先抓
runs.values() snapshot,再 await per-run cancel。在 await 窗口期间为本
项目新建的 run 不在 snapshot,不会被 tombstone;它的 state.json 仍然
指向该项目,后续的 removeProjectRunDirs 只看 state.projectId ===
target 就会把这个活 run 的目录删掉——活 run 还在跑,但 state.json 没
了,后续 persistState mkdirs 又把目录写回,正好重复我们刚消除的 #6117
orphan 场景。

修复:

1. apps/daemon/src/runtimes/runs.ts
   - purgeRunsForProject: 返回值改为 { tombstoned, protectedRunIds },
     让 caller 可以拿到后续 in-memory 仍然 live 的 run ids 而不必再查
     一次 map。protectedRunIds 在 await cancel 之后重新扫一次
     runs.values() 得到(persistsDisabled === false 的本项目 run),
     捕获 await 窗口期间新建的 late arrival。
   - 新增 isLiveRun(runId): 同步查 in-memory map,返回 run 存在且未被
     tombstone。暴露给 sweep caller 用作 shouldSkip 谓词。

2. apps/daemon/src/projects.ts
   - removeProjectRunDirs: 新增第三个可选参数 { shouldSkip?: (runId)
     => boolean }。在 read state.json 之前先调 shouldSkip(runId),
     返回 true 就跳过该目录,避免误删活 run 的 dir。注释里点明这是为
     了保护 race 期间新建的 mid-flight run。

3. apps/daemon/src/routes/project/index.ts
   - DELETE handler 把 shouldSkip: (runId) => design.runs.isLiveRun(runId)
     传给 removeProjectRunDirs。这样 purge 之后 sweep 之前的窗口里出现
     的新 run,只要它还在 in-memory map 里就能被识别并跳过。
   - 注释里点明race motivation + #6202 follow-up 来源。

4. apps/daemon/tests/delete-sweeps-orphaned-run-dirs.test.ts
   - 新增 regression:验证 removeProjectRunDirs 的 shouldSkip 行为:
     先手动 purgeRunsForProject tombstone 掉一个 run,再造一个 late
     arrival run,再调 removeProjectRunDirs 传 production 的
     shouldSkip = (id) => runs.isLiveRun(id) 谓词。断言:
     * orphan dirs (state.json projectId 匹配但 in-memory 没有该 run)
       被正常清扫(2 个)
     * late arrival run 的目录完好无损
     * isLiveRun 对 late arrival true,对空字符串/orphan/已 tombstone
       的 run 都 false
     * 第二次 purgeRunsForProject 返回
       { tombstoned: [lateRun.id], protectedRunIds: [] }
     * 空/undefined projectId 返回 { tombstoned: [], protectedRunIds: [] }
       而非旧的 bare array
   - 这条 regression锁住,mrcfps 后续 refactor 不会再让 race 期间的
     新建 run 被静默清掉。

CHANGELOG:更新 #6117 fixed 条目,把 #6202 follow-up 一并交代清楚。

验证:
- pnpm vitest run tests/delete-sweeps-orphaned-run-dirs.test.ts → 5/5 pass
- pnpm vitest run tests/cancel-owned-runs.test.ts tests/delete-cancels-active-runs.test.ts → 6/6 pass
- NODE_OPTIONS=--max-old-space-size=1536 npx tsc -p tsconfig.json --noEmit → 0
- NODE_OPTIONS=--max-old-space-size=1536 npx tsc -p tsconfig.tests.json --noEmit → 0

Author: xxiaoxiong

CHANGELOG.md

@@ -9,7 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
-- [fix] 删除项目时清理 `<RUNTIME_DATA_DIR>/runs/<runId>/` 下归属于该项目的孤立 run 目录,而非仅取消活跃 run。Run service 在 ~30 分钟 TTL 后从内存 map 移除已终止 run,但磁盘上的 `state.json`/`events.jsonl` 超出该窗口仍然残留;此前 delete project 会泄漏该项目所有已结束 run 的目录。新 helper `removeProjectRunDirs(runsDir, projectId)` 在 cancel + removeProjectDir 之后以 best-effort 方式扫描并清理。(#6117)
+- [fix] 删除项目时清理 `<RUNTIME_DATA_DIR>/runs/<runId>/` 下归属于该项目的孤立 run 目录,而非仅取消活跃 run。Run service 在 ~30 分钟 TTL 后从内存 map 移除已终止 run,但磁盘上的 `state.json`/`events.jsonl` 超出该窗口仍然残留;此前 delete project 会泄漏该项目所有已结束 run 的目录。新 helper `removeProjectRunDirs(runsDir, projectId)` 在 cancel + removeProjectDir 之后以 best-effort 方式扫描并清理。`purgeRunsForProject` 现在返回 `{ tombstoned, protectedRunIds }` 结构化结果,`removeProjectRunDirs` 接受可选的 `shouldSkip(runId)` 回调,DELETE handler 通过 `design.runs.isLiveRun(runId)` 跳过仍在内存 map 中的活跃 run,避免在 purge 之后、sweep 之前出现的窗口里被创建的新 run 被误删;修复 PR #6202 @mrcfps 评审指出的第二个 race condition。(#6117, #6202)
 
 ## [0.9.0] - 2026-05-29
 

apps/daemon/src/projects.ts

@@ -1358,11 +1358,23 @@ export async function removeProjectDir(projectsRoot, projectId) {
  *
  * @param {string} runsDir Absolute path to the runtime `runs` directory.
  * @param {string} projectId The project id whose run directories to remove.
+ * @param {{ shouldSkip?: (runId: string) => boolean }} [opts] Optional
+ *   per-entry guard. When `shouldSkip(runId)` returns true, the directory is
+ *   left alone — used to protect runs that legitimately still own their
+ *   on-disk directory at sweep time (e.g. a run created for this project
+ *   during the cancel-await window of a project-delete — its state.json
+ *   references the deleted project, but it's mid-flight and would otherwise
+ *   be wiped out from under itself). See #6202 @mrcfps follow-up.
  * @returns {Promise<number>} Number of run directories removed. Resolves to 0
  *   when `runsDir` does not exist (fresh install, no runs yet).
  */
-export async function removeProjectRunDirs(runsDir, projectId) {
+export async function removeProjectRunDirs(
+  runsDir: string,
+  projectId: string,
+  opts: { shouldSkip?: (runId: string) => boolean } = {},
+): Promise<number> {
   if (!isSafeId(projectId)) return 0;
+  const shouldSkip = opts.shouldSkip;
   let entries: import('node:fs').Dirent[];
   try {
     entries = await readdir(runsDir, { withFileTypes: true });
@@ -1376,6 +1388,13 @@ export async function removeProjectRunDirs(runsDir, projectId) {
       .filter((entry) => entry.isDirectory())
       .map(async (entry) => {
         const runDir = path.join(runsDir, entry.name);
+        const runId = entry.name;
+        // Consult the live-run guard before touching the disk so a
+        // mid-flight run created during a concurrent delete-project can't
+        // have its state.json wiped out from under itself. Should be a
+        // cheap synchronous check (in-memory map lookup); see
+        // `runService.isLiveRun` in runs.ts.
+        if (shouldSkip && shouldSkip(runId)) return;
         try {
           const stateRaw = await readFile(path.join(runDir, 'state.json'), 'utf8');
           const state = JSON.parse(stateRaw);

apps/daemon/src/routes/project/index.ts

@@ -2279,6 +2279,13 @@ export function registerProjectRoutes(app: Express, ctx: RegisterProjectRoutesDe
       // while it was still alive would leak its dir back onto disk the moment
       // the analytics/langfuse completion resolved (#6117 race @mrcfps on #6202).
       // Also cancels live runs (#5468 supersedes the bare cancelRunsOwnedBy).
+      // The returned `protectedRunIds` are runs that entered the in-memory
+      // registry during the cancellation await — their state.json still
+      // references this project but the runs are mid-flight and must be left
+      // alone. The `isLiveRun` guard re-checks the in-memory map at sweep
+      // time so even later concurrent creates (after `purgeRunsForProject`
+      // returns but before `removeProjectRunDirs` runs) are protected
+      // (#6202 @mrcfps non-blocking follow-up).
       await design.runs.purgeRunsForProject(req.params.id).catch(() => {});
       dbDeleteProject(db, req.params.id);
       await removeProjectDir(PROJECTS_DIR, req.params.id).catch(() => {});
@@ -2287,7 +2294,11 @@ export function registerProjectRoutes(app: Express, ctx: RegisterProjectRoutesDe
       // map; this catches runs whose TTL has already expired out of the map
       // (their state.json lingers on disk until now). Best-effort — same
       // posture as removeProjectDir above.
-      await removeProjectRunDirs(path.join(ctx.paths.RUNTIME_DATA_DIR, 'runs'), req.params.id).catch(() => {});
+      await removeProjectRunDirs(
+        path.join(ctx.paths.RUNTIME_DATA_DIR, 'runs'),
+        req.params.id,
+        { shouldSkip: (runId) => design.runs.isLiveRun(runId) },
+      ).catch(() => {});
       /** @type {import('@open-design/contracts').OkResponse} */
       const body = { ok: true };
       res.json(body);

apps/daemon/src/runtimes/runs.ts

@@ -1076,14 +1076,23 @@ export function createChatRunService({
    *     child is reaped) — terminal runs are left alone status-wise, only
    *     tombstoned on disk.
    *
-   * Returns the list of run ids that were tombstoned, so the caller can
-   * additionally remove their on-disk run directories after this resolves.
+   * Returns `{ tombstoned, protectedRunIds }`:
+   *   - `tombstoned` lists the run ids that were tombstoned (already disabled
+   *     and cancel-awaited) so the caller can additionally remove their
+   *     on-disk run directories after this resolves.
+   *   - `protectedRunIds` lists runs that *entered the in-memory registry
+   *     after the initial snapshot* — i.e. were created for this project
+   *     during the await-cancellation window. These are live runs the sweep
+   *     must NOT delete (their state.json still references the deleted
+   *     project, but the run is mid-flight and owns that dir legitimately).
+   *     See #6202 @mrcfps non-blocking follow-up.
+   *
    * Cancellation failures are swallowed per-run so a single bad run never
    * blocks the project delete the user asked for — same posture as
    * `cancelRunsOwnedBy` for #5468.
    */
   const purgeRunsForProject = async (projectId) => {
-    if (typeof projectId !== 'string' || !projectId) return [];
+    if (typeof projectId !== 'string' || !projectId) return { tombstoned: [], protectedRunIds: [] };
     const owned = Array.from(runs.values()).filter((run) => run.projectId === projectId);
     const tombstoned = [];
     // Order matters (#6117 race): disablePersist MUST run before cancel,
@@ -1101,7 +1110,33 @@ export function createChatRunService({
         }
       }),
     );
-    return tombstoned;
+    // Re-snapshot after the await window. A run created for this project
+    // during the cancellation await is *not* in `owned` and therefore not
+    // tombstoned above; its state.json still references the soon-to-be
+    // deleted project, so the on-disk sweep would otherwise remove it
+    // mid-flight. Track it as protected so the caller can skip its dir.
+    // (Live runs whose project is gone will eventually go terminal + TTL out
+    // of the in-memory map; the next delete-project call's sweep will catch
+    // their then-orphaned state.json. That's the same backstop posture
+    // already used for runs that TTL-expire out before delete-project.)
+    const protectedRunIds = Array.from(runs.values())
+      .filter((run) => run.projectId === projectId && !run.persistsDisabled)
+      .map((run) => run.id);
+    return { tombstoned, protectedRunIds };
+  };
+
+  /**
+   * Live-run guard for the on-disk sweep. Returns true if `runId` is currently
+   * registered in the in-memory run map AND has not been tombstoned (i.e. its
+   * `persistsDisabled` flag is still false). Callers that walk `runsDir` to
+   * remove orphaned run directories use this to skip directories owned by runs
+   * that legitimately still own them — e.g. a run created for the project
+   * during the cancel-await window of a project-delete (#6202 @mrcfps follow-up).
+   */
+  const isLiveRun = (runId) => {
+    if (typeof runId !== 'string' || !runId) return false;
+    const run = runs.get(runId);
+    return !!run && !run.persistsDisabled;
   };
 
   // Drop a run from the in-memory registry without emitting any terminal
@@ -1175,6 +1210,7 @@ export function createChatRunService({
     fail,
     drop,
     purgeRunsForProject,
+    isLiveRun,
     signalChild: killChild,
     reapProcessGroup,
     signalProcessGroup,

apps/daemon/tests/delete-sweeps-orphaned-run-dirs.test.ts

@@ -277,4 +277,119 @@ describe('DELETE project sweeps orphaned run dirs (#6117)', () => {
 
     expect(existsSync(runDir)).toBe(false);
   });
+
+  it('does not sweep a run dir for a run still live in the in-memory map (#6202 @mrcfps follow-up)', async () => {
+    // Reproduces the second race @mrcfps flagged on PR #6202 (non-blocking
+    // follow-up): purgeRunsForProject snapshots the in-memory runs once, then
+    // awaits per-run cancellations. A run that is created for the same project
+    // during that await window is NOT in the snapshot, so it is not tombstoned;
+    // its state.json still references the soon-to-be-deleted project. If the
+    // on-disk sweep (removeProjectRunDirs) only filters by `state.projectId`,
+    // it wipes the new run's dir out from under itself — the run keeps running
+    // but its state file is gone, and any later persistState mkdirs the dir
+    // back, recreating the orphan we just swept.
+    //
+    // After the fix:
+    //   - removeProjectRunDirs takes a `shouldSkip(runId)` callback that
+    //     consults the in-memory run map.
+    //   - The DELETE handler passes `(runId) => design.runs.isLiveRun(runId)`.
+    //   - A run that is currently in the in-memory map AND not tombstoned is
+    //     left alone even if its state.json projectId matches the sweep target.
+    //
+    // This test exercises the sweep with a controlled shouldSkip that mirrors
+    // the production wiring (isLiveRun) and asserts the live run's dir survives
+    // the sweep even though its state.json projectId matches the sweep target.
+    const runsDir = path.join(tempDir, 'runs');
+    mkdirSync(runsDir, { recursive: true });
+    const runs = createChatRunService({
+      createSseResponse: () => ({ send: vi.fn(() => true), end: vi.fn(), cleanup: vi.fn() }),
+      createSseErrorPayload: (code: string, message: string) => ({ error: { code, message } }),
+      shutdownGraceMs: 10,
+      ttlMs: 60_000,
+      runsLogDir: runsDir,
+    });
+
+    // Seed one orphaned run dir that DOES belong to p1 and is NOT in the
+    // in-memory map (simulates a TTL-expired prior run). The sweep should
+    // catch this one.
+    const orphanRunId = 'orphan-' + Math.random().toString(36).slice(2);
+    const orphanDir = path.join(runsDir, orphanRunId);
+    mkdirSync(orphanDir, { recursive: true });
+    writeFileSync(path.join(orphanDir, 'state.json'), JSON.stringify({ projectId: 'p1' }));
+
+    // And create a live run for p1 that IS in the in-memory map. The sweep
+    // must skip its directory because the production shouldSkip guard
+    // (design.runs.isLiveRun) returns true for it.
+    const liveRun = runs.create({ projectId: 'p1' });
+    const liveRunDir = path.join(runsDir, liveRun.id);
+    expect(existsSync(path.join(liveRunDir, 'state.json'))).toBe(true);
+
+    // Mount and DELETE the project. The DELETE handler will:
+    //   1. purgeRunsForProject('p1') — tombstones liveRun (it is in the
+    //      snapshot, because we created it before the request). liveRun is
+    //      now persistsDisabled=true and its dir is removed by disablePersist.
+    //   2. removeProjectRunDirs with shouldSkip = (id) => isLiveRun(id).
+    //      liveRun is no longer "live" (persistsDisabled=true), so its dir
+    //      isn't expected to survive — but we want to verify the production
+    //      path end-to-end against a "truly live" run.
+    //
+    // To exercise the race scenario end-to-end (a run created DURING the
+    // cancel await window, which is NOT in the snapshot), we mount the app
+    // but hold the DELETE back: we pre-await a manual `purgeRunsForProject`
+    // that tombstones liveRun, then create a second live run that is NOT
+    // tombstoned, then invoke the on-disk sweep directly with the production
+    // shouldSkip guard.
+    //
+    // Step 1: tombstone the first liveRun via purgeRunsForProject.
+    const firstPurge = await runs.purgeRunsForProject('p1');
+    expect(firstPurge.tombstoned).toEqual(expect.arrayContaining([liveRun.id]));
+    expect(existsSync(liveRunDir)).toBe(false);
+    expect(runs.isLiveRun(liveRun.id)).toBe(false);
+
+    // Step 2: create a new run for p1 — this one is "mid-flight" relative to
+    // the original DELETE, i.e. not in any snapshot yet. Its state.json
+    // projectId matches p1.
+    const lateRun = runs.create({ projectId: 'p1' });
+    const lateRunDir = path.join(runsDir, lateRun.id);
+    expect(existsSync(path.join(lateRunDir, 'state.json'))).toBe(true);
+    expect(runs.isLiveRun(lateRun.id)).toBe(true);
+
+    // Also re-seed an orphan dir with state.json projectId=p1 (the sweep's
+    // primary target).
+    const orphan2Dir = path.join(runsDir, 'orphan2-' + Math.random().toString(36).slice(2));
+    mkdirSync(orphan2Dir, { recursive: true });
+    writeFileSync(path.join(orphan2Dir, 'state.json'), JSON.stringify({ projectId: 'p1' }));
+
+    // Step 3: invoke the on-disk sweep with the production shouldSkip guard,
+    // exactly as the DELETE handler does.
+    const { removeProjectRunDirs } = await import('../src/projects.js');
+    const removed = await removeProjectRunDirs(runsDir, 'p1', {
+      shouldSkip: (runId) => runs.isLiveRun(runId),
+    });
+    // The orphan dirs (no in-memory run) are swept.
+    expect(removed).toBe(2);
+    expect(existsSync(orphanDir)).toBe(false);
+    expect(existsSync(orphan2Dir)).toBe(false);
+    // The late run's dir is untouched — the shouldSkip guard caught it.
+    expect(existsSync(lateRunDir)).toBe(true);
+    expect(existsSync(path.join(lateRunDir, 'state.json'))).toBe(true);
+
+    // The new helper is exposed for callers that want to query the live-run
+    // state directly (e.g. future sweep callers).
+    expect(runs.isLiveRun(lateRun.id)).toBe(true);
+    expect(runs.isLiveRun('orphanRunId-not-in-map')).toBe(false);
+    expect(runs.isLiveRun('')).toBe(false);
+
+    // And purgeRunsForProject now returns a structured result so callers can
+    // destructure safely instead of treating the return as a bare array.
+    const result = await runs.purgeRunsForProject('p1');
+    expect(result).toEqual({ tombstoned: expect.arrayContaining([lateRun.id]), protectedRunIds: [] });
+    expect(existsSync(lateRunDir)).toBe(false);
+    expect(runs.isLiveRun(lateRun.id)).toBe(false);
+
+    // Empty/invalid project id is a no-op and returns the empty structured
+    // shape (not a bare array, so callers can destructure safely).
+    await expect(runs.purgeRunsForProject('')).resolves.toEqual({ tombstoned: [], protectedRunIds: [] });
+    await expect(runs.purgeRunsForProject(undefined as unknown as string)).resolves.toEqual({ tombstoned: [], protectedRunIds: [] });
+  });
 });