Skip to content

fix(f1_radio_rag): give the assertion judge the retrieved messages - #30

Merged
LeoRoccoBreedt merged 2 commits into
mainfrom
fschlz/fix-test-suite-assertion-context
Jul 1, 2026
Merged

fix(f1_radio_rag): give the assertion judge the retrieved messages#30
LeoRoccoBreedt merged 2 commits into
mainfrom
fschlz/fix-test-suite-assertion-context

Conversation

@fschlz

@fschlz fschlz commented Jun 30, 2026

Copy link
Copy Markdown
Collaborator

Problem

The f1_radio_rag test suite checks groundedness assertions — e.g. "grounded in the provided radio messages", "does not invent events absent from the messages", "refers only to drivers named in the messages".

But Opik's assertion judge (run_tests) builds its prompt from only the trace input and output. The task returned the retrieved messages as a separate top-level context key:

return {"input": query, "output": output, "context": context}  # context never reaches the assertion judge

So the judge saw query + summary but never the messages it was supposed to check against. Every groundedness assertion was effectively unverifiable → trivial passes and unstable verdicts.

(The evaluate() path was already fine: ContextRecall / Hallucination declare context as a first-class argument and read the top-level key directly.)

Fix

Add _suite_task, which folds the retrieved messages into input so the judge can see them, and point run_tests at it:

def _suite_task(item: dict) -> dict:
    result = answer(item["query"])
    return {
        "input": {"query": result["input"], "messages": result["context"]},
        "output": result["output"],
    }
  • evaluate() keeps _rag_task — the metrics read top-level context, so they are unaffected.
  • expected_output is deliberately kept out of input (per the Building Test Suites docs), so the judge can't use it to pass assertions that should fail.
  • README documents that the assertion judge reads only input / output, hence the fold.

Verification

  • python -m py_compile clean on evaluation.py.
  • uv run f1rag eval dry-run path unchanged (no creds needed).
  • With creds, the groundedness assertions now receive the radio messages in the judge prompt.

🤖 Generated with Claude Code

fschlz and others added 2 commits June 30, 2026 16:34
Test-suite assertions are judged on the trace input/output only, but the task returned the retrieved messages as a separate top-level context key that the run_tests judge never reads. The groundedness assertions (grounded in the messages; does not invent events absent from them) therefore had no source to verify against, yielding trivial passes and unstable verdicts.

Add _suite_task, which folds the messages into input (keeping expected_output out so the judge cannot cheat), and point run_tests at it. evaluate() keeps _rag_task: the ContextRecall and Hallucination metrics read a top-level context arg and are unaffected. README documents the input/output-only judge.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@LeoRoccoBreedt
LeoRoccoBreedt merged commit 818e339 into main Jul 1, 2026
7 checks passed
@LeoRoccoBreedt
LeoRoccoBreedt deleted the fschlz/fix-test-suite-assertion-context branch July 1, 2026 14:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants