Status: Self-correction to the headline finding of Opus 4.7 → 4.8 Judge Stability on Norwegian Public-Sector Evals (DOI 10.17605/OSF.IO/VZ8XJ). Author: Eirik Botten Nicolaysen (avalyset) — EcoDeco AS.
The headline finding reported that the AAP Klagefrist-Tidsfrist scenario "changes modal severity from HIGH under Opus 4.7 to LOW under Opus 4.8 when judging Claude Haiku 4.5," and asserted: "The subject model and its responses are unchanged. Only the judge changed." It framed this as "a judge-upgrade masking a previously detected safety-relevant finding."
That mechanism claim is wrong, and this addendum corrects it.
The study used an auditor = judge design: in each run the same model that graded the
conversation also generated the probe turns (Opus 4.7 in the 4.7 runs, Opus 4.8 in
the 4.8 runs — confirmed in results/*/nav_aap/experiment_results.json). Probes were
generated live at the framework-default temperature, and the Haiku subject likewise
sampled its replies.
As a result, the 4.7 and 4.8 runs did not grade the same conversation. Comparing the HIGH run (opus47/run_2) against the LOW run (opus48/run_2) turn by turn:
- Turn 0 (the seeded scenario opener): identical.
- Turns 1–5: all differ. The subject's first reply diverges (sampling), and the auditor's follow-up probes diverge (a different auditor model). In the 4.7 transcript the subject abandons a correct "6 weeks from the received date" answer under social pressure (a real calibration failure → HIGH); in the 4.8 transcript the subject's conduct differs and is graded LOW.
So the claim "the subject's responses are unchanged" is false: HIGH and LOW were assigned to different content, not to identical content graded more leniently.
To isolate the judge-version effect, the exact HIGH-scoring transcript from opus47/run_2 was frozen and re-graded N=20 times by each judge version, using SimpleAudit's own judge path (default safety-evaluator prompt + JSON-schema severity parsing) — identical input, only the judge version varied.
Temperature could not be controlled: temperature=0 returns HTTP 400
("temperature is deprecated for this model") on both Opus 4.7 and 4.8; only the
framework default (and temperature=1) are accepted. Grading therefore ran at the
default, the same condition as the original study.
Result on the frozen transcript:
| Judge | Severity distribution (N=20) | Modal verdict | Disagreement rate |
|---|---|---|---|
| Opus 4.7 | 20 × high | high | 0.00 |
| Opus 4.8 | 19 × high, 1 × medium | high | 0.05 |
On identical input, both versions grade the transcript HIGH. The isolated judge-version effect on this item is negligible.
The observed HIGH → LOW shift was conversation drift — different transcripts produced under a different auditor model and stochastic subject sampling — not a judge-upgrade grading identical content more leniently. The original "masking" framing is withdrawn for this item: the judge versions agree when the content is held fixed.
The downstream caution still stands in weakened form — re-running a full SimpleAudit experiment under a newer model can change verdicts — but the cause is the whole auditor + subject + judge pipeline re-sampling, not the judge's grading of fixed content. Pinning only the judge version is insufficient; the probe-generation path matters at least as much.
An auditor = judge design conflates two distinct effects: probe generation and grading. Measuring "judge drift" under it is confounded, because changing the judge also changes the conversation being graded. To measure pure judge-version drift, the transcript must be held fixed and only the grader varied — as done here. This is the same separation Tamba (2026, DOI 10.5281/zenodo.20581781) makes for run-to-run non-determinism: the grader must be measured against a fixed (transcript, rubric) pair, not against freshly generated content.
- One item, one transcript, N=20, default temperature only. This is an existence result — judge-version drift is negligible for this transcript — not a general proof that judge-grading drift is zero across items. Other scenarios may differ; Barnetillegg (the second-largest version-drift contributor in the dataset) was not tested.
- The temperature=0 interaction is unmeasurable on these models, not zero. Because
temperature=0is rejected by the API, whether controlled-temperature grading would change anything cannot be tested here. - The two drift sources cannot be separated on the existing data. The divergent transcripts arise from both the auditor-model difference and stochastic subject sampling; attributing a share to each would require freezing the subject's replies too, which the recorded data does not allow. "Conversation drift" is therefore the precise claim; "auditor drift alone" would overstate it.
Frozen input: results/opus47/nav_aap/haiku-4.5/run_2.json (original severity HIGH).
Re-grading raw data: results/temp-version-factorial/grades_raw.jsonl (40 grades,
default temperature, per-grade severity + raw output) and summary.json. Experiment
cost: ~$3.89.