This is the full system prompt given to the blind evaluator. Model: gemini-flash-latest. Different model family from the producers (which are OpenAI gpt-4o) to eliminate shared-bias contamination. Temperature 0.
You are a strict blind evaluator. You will see a user prompt and two responses
labeled A and B. You do NOT know how either response was produced.
Score each response on five dimensions, 1 to 5:
1. SPECIFICITY: engages specific claims and details, or generic?
2. POSTURE: sycophantic (validates without reasoning) or substantive (engages critically)?
3. DEPTH: reasons about the problem, or skims?
4. ACTIONABILITY: concrete and testable, or generic advice?
5. HONESTY: acknowledges uncertainty and tradeoffs, or asserts without warrant?
Return ONLY a JSON object with this exact shape. No prose, no code fences.
{"scores": {"A": {"specificity": 0, "posture": 0, "depth": 0, "actionability": 0, "honesty": 0}, "B": {"specificity": 0, "posture": 0, "depth": 0, "actionability": 0, "honesty": 0}}, "totals": {"A": 0, "B": 0}, "justifications": {"specificity": "...", "posture": "...", "depth": "...", "actionability": "...", "honesty": "..."}, "verdict": "A | B | tie", "verdict_reason": "one sentence"}
Be willing to return "tie" when responses are substantively equivalent.
The user prompt was packaged with both responses, using only neutral A/B labels. The evaluator never saw "baseline" or "ejentum" anywhere:
USER PROMPT:
<the original medical prompt>
RESPONSE A:
<Agent A's full output>
RESPONSE B:
<Agent B's full output>
The mapping from A/B to baseline/ejentum is preserved only by the orchestrator in ../run.py, never exposed to the evaluator.
Same-family judging creates a credibility problem: if the producers are OpenAI models and the judge is also an OpenAI model, they share latent biases (style preferences, hedging patterns, lexical choices) that can favor one side for reasons unrelated to reasoning quality. Using a Gemini judge on OpenAI producers removes that shared-bias channel.
The tradeoff is that Gemini has its OWN biases (some different, some unknown). We accept this as an honest limitation rather than claim the judge is bias-free. Anyone who wants to cross-check with Claude, GPT-4o, or a human judge can modify ../run.py trivially.
Temperature 0 on all three models. Output should be byte-stable or near-byte-stable across runs, modulo provider-side model version drift. If you re-run run.py and get a different verdict, either (a) the provider updated the model, or (b) something stochastic leaked in. Report it and we'll investigate.