Skip to content

Commit 002666d

Browse files
committed
Pin JUDGE_MODEL to a concrete model ID, fix #13
The `haiku` CLI alias can be repointed to a future generation or removed; either case silently shifts judge score baselines or breaks the judge entirely, breaking historical comparisons. Pin to `claude-haiku-4-5` and route the preflight auth smoke test through the same constant so it exercises the model the judge actually uses.
1 parent cdc29cd commit 002666d

2 files changed

Lines changed: 7 additions & 3 deletions

File tree

src/server/preflight.ts

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ import { promisify } from 'node:util';
33
import { homedir, tmpdir } from 'node:os';
44
import { join } from 'node:path';
55
import { pathExists, atomicWriteFile, ensureDir, readJsonIfExists } from './fsUtil.js';
6-
import { PROJECT_MARKERS, STORAGE_DIR_NAME } from '@shared/constants.js';
6+
import { JUDGE_MODEL, PROJECT_MARKERS, STORAGE_DIR_NAME } from '@shared/constants.js';
77
import { readdir, readFile, writeFile, unlink } from 'node:fs/promises';
88
import { log } from './log.js';
99

@@ -104,7 +104,7 @@ export async function authSmokeTest(bin: string): Promise<void> {
104104
'-p',
105105
'ping',
106106
'--model',
107-
'haiku',
107+
JUDGE_MODEL,
108108
'--output-format',
109109
'json',
110110
'--tools',

src/shared/constants.ts

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,11 @@ export const PROJECT_MARKERS = [
2020
'pyproject.toml',
2121
];
2222

23-
export const JUDGE_MODEL = 'haiku';
23+
// Pin to a concrete model ID, not the `haiku` CLI alias: aliases can be
24+
// repointed to a future generation or removed, which would silently shift
25+
// score baselines or break the judge entirely (issue #13). Bump manually
26+
// when a new Haiku ships and rebaselining is acceptable.
27+
export const JUDGE_MODEL = 'claude-haiku-4-5';
2428

2529
// Judge input budgets (plan § Judge flow step 2).
2630
export const JUDGE_PROMPT_CAP_BYTES = 4 * 1024;

0 commit comments

Comments
 (0)