Commit 2fb5c4a
committed
Isolate judge subprocess from project settings (closes #3)
Spawn the judge with cwd=os.tmpdir() so it cannot walk up from the
user's project applying their .claude/settings.json, hooks, and
project-scoped tool overrides. Switch --setting-sources from
'project' to 'user' for predictable, project-independent behavior.
Without this, judge scores were silently contaminated by per-project
hooks (which could log/mutate the prompt or response), permission
rules, and a project-level defaultModel that could override --model
haiku — making scores quietly differ between projects for reasons
unrelated to the variants under test.1 parent bd7519a commit 2fb5c4a
2 files changed
Lines changed: 14 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
217 | 217 | | |
218 | 218 | | |
219 | 219 | | |
220 | | - | |
| 220 | + | |
| 221 | + | |
221 | 222 | | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
222 | 226 | | |
223 | 227 | | |
224 | 228 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
| 2 | + | |
2 | 3 | | |
3 | 4 | | |
4 | 5 | | |
| |||
260 | 261 | | |
261 | 262 | | |
262 | 263 | | |
263 | | - | |
| 264 | + | |
264 | 265 | | |
265 | 266 | | |
266 | 267 | | |
267 | 268 | | |
268 | 269 | | |
269 | | - | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
270 | 277 | | |
271 | 278 | | |
272 | 279 | | |
| |||
0 commit comments