Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 12 additions & 5 deletions use-cases/f1_radio_rag/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ On top of that RAG app it demonstrates the four Opik lifecycle steps, each as a
- **`ingest`** — load the radio messages into ChromaDB (fully offline).
- **`ask`** — retrieve relevant messages and summarise them with Claude (traced in Opik).
- **`eval`** — create an Opik **dataset** and a **test suite** (plain-English assertions), then run
both: `run_tests` for the assertions and `evaluate` with the `ContextRecall` and `Hallucination`
metrics.
both: `run_tests` for the assertions (pass/fail → **pass rate**) and `evaluate` with the
`ContextRecall` and `Hallucination` metrics (numeric **feedback scores**).
- **`optimize`** — run **Optimization Studio** (`opik-optimizer`) to improve the summariser prompt
against the dataset, scored by an `AnswerRelevance` judge.
- **`promote`** — save the optimised prompt to the Opik **Prompt Library** (re-running versions it).
Expand Down Expand Up @@ -84,9 +84,16 @@ bash run.sh
(`GEN_MODEL` defaults to `anthropic/claude-sonnet-4-6`, overridable via `OPIK_EXAMPLES_MODEL`).
It's decorated with `@opik.track`, so each call appears as a trace in Opik.
3. **Eval** (`evaluation.py`) — builds an Opik dataset and a test suite, then scores the live RAG
task. The test suite checks plain-English **assertions**; `evaluate` runs the `ContextRecall`
(retrieval quality) and `Hallucination` (faithfulness) metrics. The eval cases live in
`data/eval_cases.json`.
task two ways. The eval cases live in `data/eval_cases.json`.
- The **test suite** (`run_tests`) checks plain-English **assertions** (LLM-judged) and yields a
**pass rate** — shown as "Pass rate" on the experiment. Test suites do *not* attach per-row
numeric scores, so test-suite experiments show `-` in the **Feedback Scores** column. That is
expected, not a bug — their result is the pass rate. The judge reads only the task's
`input` and `output`, so the suite task (`_suite_task`) folds the retrieved messages into
`input`; that is what lets the groundedness assertions check against the source.
- `evaluate` runs the `ContextRecall` (retrieval quality) and `Hallucination` (faithfulness)
metrics, which produce numeric **feedback scores** that *do* populate the Feedback Scores column
(e.g. `context_recall_metric`, `hallucination_metric`).
4. **Optimize** (`optimization.py`) — `MetaPromptOptimizer.optimize_prompt(...)` improves the
`ChatPrompt` against the dataset, scored by a callable that wraps Opik's `AnswerRelevance` judge.
5. **Promote** (`prompts.py`) — `client.create_chat_prompt(...)` saves the optimised messages to the
Expand Down
18 changes: 17 additions & 1 deletion use-cases/f1_radio_rag/src/f1_radio_rag/evaluation.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,19 @@ def _rag_task(item: dict) -> dict:
return answer(item["query"])


def _suite_task(item: dict) -> dict:
# The assertion judge reads ONLY `input` and `output`, so fold the retrieved messages
# into `input` — otherwise the groundedness assertions ("grounded in the messages",
# "does not invent events absent from the messages") have no source to check against.
# Keep expected_output OUT of `input`, or the judge can use it to pass assertions that
# should fail.
result = answer(item["query"])
return {
"input": {"query": result["input"], "messages": result["context"]},
"output": result["output"],
}


def build_dataset(client, eval_cases: list[dict]):
dataset = client.get_or_create_dataset(name=config.DATASET_NAME)
dataset.insert(
Expand Down Expand Up @@ -51,12 +64,15 @@ def run_eval(eval_cases: list[dict]):
dataset = build_dataset(client, eval_cases)
suite = build_suite(client, eval_cases)

# Assertions, LLM-judged -> a pass rate. Test suites do NOT attach per-row feedback scores,
# so these experiments show "-" in the UI's Feedback Scores column (expected; result is pass rate).
suite_result = run_tests(
test_suite=suite,
task=_rag_task,
task=_suite_task,
model=config.JUDGE_MODEL,
experiment_name=experiment_name(),
)
# scoring_metrics produce the numeric feedback scores that populate the UI's Feedback Scores column.
eval_result = evaluate(
dataset=dataset,
task=_rag_task,
Expand Down
Loading