Skip to content

Commit 2fb5c4a

Browse files
committed
Isolate judge subprocess from project settings (closes #3)
Spawn the judge with cwd=os.tmpdir() so it cannot walk up from the user's project applying their .claude/settings.json, hooks, and project-scoped tool overrides. Switch --setting-sources from 'project' to 'user' for predictable, project-independent behavior. Without this, judge scores were silently contaminated by per-project hooks (which could log/mutate the prompt or response), permission rules, and a project-level defaultModel that could override --model haiku — making scores quietly differ between projects for reasons unrelated to the variants under test.
1 parent bd7519a commit 2fb5c4a

2 files changed

Lines changed: 14 additions & 3 deletions

File tree

docs/PLAN.md

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -217,8 +217,12 @@ The judge is **per-variant**, powered by **Haiku**. It runs independently for ea
217217
--output-format json \
218218
--tools "" \
219219
--allowedTools "" \
220-
--strict-mcp-config
220+
--strict-mcp-config \
221+
--setting-sources user
221222
```
223+
- `cwd` = `os.tmpdir()` (the judge must NOT inherit the user's project cwd, or it would walk up applying their `.claude/settings.json`, hooks, and project-scoped tool overrides — silent score contamination).
224+
- `--setting-sources user` keeps the source explicit and predictable across projects.
225+
222226
Judge uses no tools — it reads its inputs from the prompt and emits structured JSON. Output shape enforced via prompt + ajv validation (with `--json-schema` fallback if supported by the installed CLI).
223227
5. On schema failure, retry once with an error-explaining follow-up. Still invalid → `judge.json.status = "errored"` with the error surfaced in the UI; the column's run results are untouched.
224228
6. Write `<run>/judge.json`. Push `judge.updated` (or `judge.errored`) for that column over SSE.

src/server/judge.ts

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
import { spawn } from 'node:child_process';
2+
import { tmpdir } from 'node:os';
23
import { join } from 'node:path';
34
import JSON5 from 'json5';
45
import {
@@ -260,13 +261,19 @@ function spawnJudge(claudeBin: string, prompt: string, opts: SpawnJudgeOptions):
260261
'',
261262
'--strict-mcp-config',
262263
'--setting-sources',
263-
'project',
264+
'user',
264265
'--disable-slash-commands',
265266
];
266267
if (opts.jsonSchema) {
267268
args.push('--json-schema', JSON.stringify(JUDGE_MODEL_JSON_SCHEMA));
268269
}
269-
const proc = spawn(claudeBin, args, { stdio: ['ignore', 'pipe', 'pipe'] });
270+
// Spawn from os.tmpdir() so the judge never inherits the user's project cwd.
271+
// Combined with `--setting-sources user`, this prevents project `.claude/settings.json`,
272+
// hooks, and tool overrides from silently contaminating judge scores (issue #3).
273+
const proc = spawn(claudeBin, args, {
274+
stdio: ['ignore', 'pipe', 'pipe'],
275+
cwd: tmpdir(),
276+
});
270277
let stdout = '';
271278
let stderr = '';
272279
proc.stdout.on('data', (d) => (stdout += d.toString()));

0 commit comments

Comments
 (0)