diff --git a/use-cases/f1_radio_rag/README.md b/use-cases/f1_radio_rag/README.md index 9e4d77b..7cb2a77 100644 --- a/use-cases/f1_radio_rag/README.md +++ b/use-cases/f1_radio_rag/README.md @@ -13,8 +13,8 @@ On top of that RAG app it demonstrates the four Opik lifecycle steps, each as a - **`ingest`** — load the radio messages into ChromaDB (fully offline). - **`ask`** — retrieve relevant messages and summarise them with Claude (traced in Opik). - **`eval`** — create an Opik **dataset** and a **test suite** (plain-English assertions), then run - both: `run_tests` for the assertions and `evaluate` with the `ContextRecall` and `Hallucination` - metrics. + both: `run_tests` for the assertions (pass/fail → **pass rate**) and `evaluate` with the + `ContextRecall` and `Hallucination` metrics (numeric **feedback scores**). - **`optimize`** — run **Optimization Studio** (`opik-optimizer`) to improve the summariser prompt against the dataset, scored by an `AnswerRelevance` judge. - **`promote`** — save the optimised prompt to the Opik **Prompt Library** (re-running versions it). @@ -84,9 +84,16 @@ bash run.sh (`GEN_MODEL` defaults to `anthropic/claude-sonnet-4-6`, overridable via `OPIK_EXAMPLES_MODEL`). It's decorated with `@opik.track`, so each call appears as a trace in Opik. 3. **Eval** (`evaluation.py`) — builds an Opik dataset and a test suite, then scores the live RAG - task. The test suite checks plain-English **assertions**; `evaluate` runs the `ContextRecall` - (retrieval quality) and `Hallucination` (faithfulness) metrics. The eval cases live in - `data/eval_cases.json`. + task two ways. The eval cases live in `data/eval_cases.json`. + - The **test suite** (`run_tests`) checks plain-English **assertions** (LLM-judged) and yields a + **pass rate** — shown as "Pass rate" on the experiment. Test suites do *not* attach per-row + numeric scores, so test-suite experiments show `-` in the **Feedback Scores** column. That is + expected, not a bug — their result is the pass rate. The judge reads only the task's + `input` and `output`, so the suite task (`_suite_task`) folds the retrieved messages into + `input`; that is what lets the groundedness assertions check against the source. + - `evaluate` runs the `ContextRecall` (retrieval quality) and `Hallucination` (faithfulness) + metrics, which produce numeric **feedback scores** that *do* populate the Feedback Scores column + (e.g. `context_recall_metric`, `hallucination_metric`). 4. **Optimize** (`optimization.py`) — `MetaPromptOptimizer.optimize_prompt(...)` improves the `ChatPrompt` against the dataset, scored by a callable that wraps Opik's `AnswerRelevance` judge. 5. **Promote** (`prompts.py`) — `client.create_chat_prompt(...)` saves the optimised messages to the diff --git a/use-cases/f1_radio_rag/src/f1_radio_rag/evaluation.py b/use-cases/f1_radio_rag/src/f1_radio_rag/evaluation.py index e5e4876..398b1f9 100644 --- a/use-cases/f1_radio_rag/src/f1_radio_rag/evaluation.py +++ b/use-cases/f1_radio_rag/src/f1_radio_rag/evaluation.py @@ -17,6 +17,19 @@ def _rag_task(item: dict) -> dict: return answer(item["query"]) +def _suite_task(item: dict) -> dict: + # The assertion judge reads ONLY `input` and `output`, so fold the retrieved messages + # into `input` — otherwise the groundedness assertions ("grounded in the messages", + # "does not invent events absent from the messages") have no source to check against. + # Keep expected_output OUT of `input`, or the judge can use it to pass assertions that + # should fail. + result = answer(item["query"]) + return { + "input": {"query": result["input"], "messages": result["context"]}, + "output": result["output"], + } + + def build_dataset(client, eval_cases: list[dict]): dataset = client.get_or_create_dataset(name=config.DATASET_NAME) dataset.insert( @@ -51,12 +64,15 @@ def run_eval(eval_cases: list[dict]): dataset = build_dataset(client, eval_cases) suite = build_suite(client, eval_cases) + # Assertions, LLM-judged -> a pass rate. Test suites do NOT attach per-row feedback scores, + # so these experiments show "-" in the UI's Feedback Scores column (expected; result is pass rate). suite_result = run_tests( test_suite=suite, - task=_rag_task, + task=_suite_task, model=config.JUDGE_MODEL, experiment_name=experiment_name(), ) + # scoring_metrics produce the numeric feedback scores that populate the UI's Feedback Scores column. eval_result = evaluate( dataset=dataset, task=_rag_task,