Merge remote-tracking branch 'origin/main' into LeoRoccoBreedt/gh-workflow-test-automation

Author: LeoRoccoBreedt

.claude/settings.json

@@ -0,0 +1,29 @@
+{
+  "permissions": {
+    "allow": [
+      "Bash(uv:*)",
+      "Bash(uv run:*)",
+      "Bash(git:*)",
+      "Bash(python:*)",
+      "Bash(python3:*)",
+      "Bash(pip:*)",
+      "Bash(ls:*)",
+      "Bash(find:*)",
+      "Bash(grep:*)",
+      "Bash(jq:*)",
+      "Bash(echo:*)",
+      "Read(*)"
+    ],
+    "deny": [
+      "Read(.env)",
+      "Bash(rm -rf /*)",
+      "Bash(git reset --hard:*)",
+      "Bash(git rebase:*)",
+      "Bash(git merge:*)",
+      "Bash(git branch -D:*)",
+      "Bash(gh pr merge:*)",
+      "Bash(gh pr close:*)",
+      "Bash(gh pr review:*)"
+    ]
+  }
+}

.claude/skills/add-dataset-items/SKILL.md

@@ -0,0 +1,82 @@
+---
+name: add-dataset-items
+description: Add new items to an Opik evaluation dataset in this repo. Use when the user says "add a case to the dataset", "add items to the Opik dataset", "extend the eval dataset", "seed the dataset", "add test data for the eval", or names new query/expected-output pairs to evaluate. Appends to the example's canonical source data file first, then syncs to Opik via get_or_create_dataset + dataset.insert. Require a target example (or dataset name) and the item content; refuse to invent dataset rows.
+---
+
+# Add items to an Opik dataset
+
+An Opik dataset is the set of inputs an example is evaluated against. In this repo each
+example keeps a **canonical source file** (e.g. `data/eval_cases.json`) that a loader reads
+and inserts into Opik — so dataset items live in two places and must stay in sync.
+
+Reference implementation: [`use-cases/f1_radio_rag`](../../../use-cases/f1_radio_rag) —
+source `data/eval_cases.json`, loaded by `data.py`, inserted by `build_dataset` in
+[`evaluation.py`](../../../use-cases/f1_radio_rag/src/f1_radio_rag/evaluation.py).
+
+## When to refuse
+
+You need both:
+
+- A **target example** (a folder under `use-cases/`, `guides/`, etc.) or an explicit dataset name.
+- The **item content** — at minimum the query/input; ideally the expected output and any
+  context fields the metrics need.
+
+If either is missing, ask one short question. Do not fabricate dataset rows — bad data
+silently corrupts every downstream eval and optimization run.
+
+## Step 1 — Match the existing item schema
+
+Read the example's source data file and copy the **exact key set** of existing items. Opik
+dataset items are free-form dicts, but the evaluation task and metrics read specific keys.
+For f1_radio_rag each item is:
+
+```json
+{
+  "query": "What tyre problems did drivers report?",
+  "expected_output": "Verstappen reported front-left graining ...",
+  "messages": ["VER: Front-left is graining ...", "..."],
+  "assertions": ["Reports the front-left graining ...", "..."]
+}
+```
+
+New items must provide every key the task/metrics consume. Mismatched or missing keys throw
+at eval time (`KeyError` in the task) — not at insert time.
+
+## Step 2 — Append to the canonical source file
+
+Add the new item(s) to the source JSON (the SSOT), preserving formatting. This is the change
+that gets committed and reviewed. Keep entries grounded and self-consistent (the expected
+output must actually follow from the provided context).
+
+## Step 3 — Sync to Opik
+
+Insert is **idempotent**: Opik content-hashes each item, so re-inserting the whole file only
+adds genuinely new rows. The simplest sync is re-running the example's dataset builder.
+
+```python
+import opik
+
+client = opik.Opik()  # env-driven; never hardcode workspace/keys
+dataset = client.get_or_create_dataset(name=DATASET_NAME)
+dataset.insert(items)  # items = the full list from the source file; identical rows are deduped
+```
+
+For f1_radio_rag, `build_dataset(client, load_eval_cases())` does exactly this, and
+`uv run f1rag eval` calls it as part of the eval — so adding to `data/eval_cases.json` and
+running eval is enough; no separate insert step needed.
+
+## Step 4 — Dry-run / credentials
+
+`dataset.insert` needs `OPIK_API_KEY` + `OPIK_WORKSPACE`. Without them, the example runs in
+`DRY_RUN` and prints what it would create instead of calling Opik (see the repo convention in
+[AGENTS.md](../../../AGENTS.md)). Verify the new items appear in the dry-run output:
+
+```bash
+cd use-cases/f1_radio_rag && uv run f1rag eval   # [DRY RUN] lists the dataset items it would create
+```
+
+## Verify
+
+- The new item(s) show up in the dry-run listing (or in the Opik UI dataset when creds are set).
+- The item's keys match what the eval task and metrics read — otherwise eval throws later.
+- The source file is still valid JSON (`jq . data/eval_cases.json`).

.claude/skills/add-eval-suite-items/SKILL.md

@@ -0,0 +1,76 @@
+---
+name: add-eval-suite-items
+description: Add new items (plain-English assertions) to an Opik test suite in this repo. Use when the user says "add an assertion", "add a test case to the suite", "extend the eval suite", "add a check to the test suite", "the model should also be tested for X", or describes a pass/fail criterion for an example's output. Appends to the canonical source file, then syncs via get_or_create_test_suite + suite.insert. Require a target example/suite and the assertion text; refuse to invent checks.
+---
+
+# Add items to an Opik test suite
+
+A test suite checks an example's output against **plain-English assertions** scored by an
+LLM judge — complementary to numeric metrics. Each suite item pairs the input `data` with a
+list of `assertions`. Suite-wide assertions and the run policy are set once on the suite.
+
+Reference implementation: `build_suite` in
+[`evaluation.py`](../../../use-cases/f1_radio_rag/src/f1_radio_rag/evaluation.py); source
+assertions live per-case in
+[`data/eval_cases.json`](../../../use-cases/f1_radio_rag/data/eval_cases.json).
+
+## When to refuse
+
+You need both a **target example/suite name** and the **assertion text** (or a clear
+criterion you can phrase as one). Don't invent checks — a vague or wrong assertion makes the
+judge fail good outputs (or pass bad ones).
+
+## Step 1 — Write good assertions
+
+- **Plain English, judged true/false** about the output. e.g. *"Reports the front-left
+  graining described in the radio messages."*
+- **Grounded in the item's context** — only assert what the provided input/context supports.
+- **One claim per assertion.** Split compound checks so a failure points at one thing.
+- **Per-item vs global:** item-specific facts → the item's `assertions`. Checks that must
+  hold for *every* item (e.g. "grounded in the provided messages", "invents nothing") →
+  `global_assertions` on the suite.
+
+## Step 2 — Append to the canonical source file
+
+Add the assertion(s) to the item's `assertions` array in the source JSON (the SSOT that gets
+committed). Keep the existing shape:
+
+```json
+{ "data": { "query": "..." }, "assertions": ["Check one ...", "Check two ..."] }
+```
+
+## Step 3 — Sync to Opik
+
+```python
+suite = client.get_or_create_test_suite(
+    name=SUITE_NAME,
+    global_assertions=[
+        "The answer is grounded in the provided context",
+        "The answer does not invent events absent from the context",
+    ],
+    global_execution_policy={"runs_per_item": 2, "pass_threshold": 2},
+)
+suite.insert([{"data": {"query": c["query"]}, "assertions": c["assertions"]} for c in cases])
+```
+
+- `global_execution_policy`: `runs_per_item` re-runs each item to catch nondeterminism;
+  `pass_threshold` is how many of those runs must pass for the item to pass.
+- `suite.insert` is idempotent the same way datasets are — re-running the builder is safe.
+
+For f1_radio_rag, `uv run f1rag eval` rebuilds and runs the suite, so editing the source JSON
+and running eval is the whole flow.
+
+## Step 4 — Dry-run / credentials
+
+Needs `OPIK_API_KEY` + `OPIK_WORKSPACE`; otherwise `DRY_RUN` prints the assertions it would
+create:
+
+```bash
+cd use-cases/f1_radio_rag && uv run f1rag eval   # [DRY RUN] lists each query + its assertions
+```
+
+## Verify
+
+- New assertions appear in the dry-run listing (or the suite in the Opik UI).
+- Each assertion is grounded in that item's context and phrased as a single true/false claim.
+- Source file is still valid JSON (`jq . data/eval_cases.json`).

.claude/skills/run-evals/SKILL.md

@@ -0,0 +1,80 @@
+---
+name: run-evals
+description: Run an Opik evaluation for an example in this repo — metrics (evaluate) and/or assertions (run_tests) against a dataset/test suite. Use when the user says "run the eval", "evaluate the example", "run the test suite", "score the model", "check the metrics", "what's the pass rate", or wants to measure an example's quality in Opik. Covers the credentials/DRY_RUN gate, metric selection, experiment naming, and reading results. Require a target example; refuse to guess which one.
+---
+
+# Run an Opik evaluation
+
+Two complementary scorers, usually run together:
+
+- **`run_tests(test_suite, ...)`** — judges the plain-English assertions in a test suite.
+  Returns `.pass_rate` and `.experiment_url`.
+- **`evaluate(dataset, ...)`** — runs numeric `scoring_metrics` over a dataset.
+
+Reference: `run_eval` in
+[`evaluation.py`](../../../use-cases/f1_radio_rag/src/f1_radio_rag/evaluation.py); CLI
+[`cli.py`](../../../use-cases/f1_radio_rag/src/f1_radio_rag/cli.py) `eval` command.
+
+## When to refuse
+
+Need a **target example** (folder + its dataset/suite). If the repo has several, ask which.
+Don't run evals blindly — they call LLM-judge models and cost tokens/time.
+
+## Step 1 — Credentials gate
+
+`evaluate` / `run_tests` need `OPIK_API_KEY` + `OPIK_WORKSPACE`, and the judge/generation
+models need their provider key (e.g. `ANTHROPIC_API_KEY`). Without Opik creds the example
+runs in `DRY_RUN` and prints what it would create. Run the dry-run first to confirm the
+dataset/suite content, then set creds for the real run.
+
+```bash
+cd use-cases/f1_radio_rag
+uv run f1rag eval                       # DRY_RUN: lists dataset items + assertions
+export ANTHROPIC_API_KEY=... OPIK_API_KEY=... OPIK_WORKSPACE=...
+uv run f1rag eval                       # real run: prints pass rate + experiment URL
+```
+
+## Step 2 — The task and the shape it returns
+
+The evaluation task is `(item: dict) -> dict`. It runs the live app on the item and returns
+the fields the metrics consume — for RAG: `{"input", "output", "context"}`. Reuse the
+example's existing task (don't reimplement the app inside the eval).
+
+```python
+from opik.evaluation import evaluate, run_tests
+from opik.evaluation.metrics import ContextRecall, Hallucination
+
+def task(item: dict) -> dict:
+    return app_under_test(item["query"])   # -> {"input","output","context"}
+```
+
+## Step 3 — Pick metrics that match the failure modes
+
+- `ContextRecall` — did retrieval surface the needed context? (RAG retrieval quality)
+- `Hallucination` — is the answer faithful to the context? (faithfulness)
+- `AnswerRelevance` — does the answer address the question?
+- Add others from `opik.evaluation.metrics` as the use case needs. Each LLM-judge metric
+  takes `model=...`.
+
+```python
+run_tests(test_suite=suite, task=task, model=JUDGE_MODEL, experiment_name=experiment_name())
+evaluate(dataset=dataset, task=task,
+         scoring_metrics=[ContextRecall(model=JUDGE_MODEL), Hallucination(model=JUDGE_MODEL)],
+         experiment_name=experiment_name())
+```
+
+Use a unique `experiment_name` per run (the example does `f"{project}-{secrets.token_hex(3)}"`)
+so runs don't overwrite each other in the UI.
+
+## Step 4 — Read results
+
+- `suite_result.pass_rate` — fraction of suite items passing their assertions.
+- `suite_result.experiment_url` / `result.get(...)` — link to the run in Opik for per-item
+  drill-down. Surface the URL to the user.
+
+## Verify
+
+- Dry-run lists the expected dataset/suite content before the real run.
+- Real run prints a pass rate and an experiment URL; open it to confirm per-item scores.
+- If the task throws `KeyError`, a dataset item is missing a key the task/metrics read — fix
+  the data (see the `add-dataset-items` skill), not the task.

.claude/skills/run-optimizations/SKILL.md

@@ -0,0 +1,95 @@
+---
+name: run-optimizations
+description: Run an Opik prompt optimization (opik-optimizer) for an example in this repo, and optionally promote the result to the Prompt Library. Use when the user says "optimize the prompt", "run the optimizer", "improve the prompt", "run Optimization Studio", "tune the prompt", or wants a measured better prompt for an example. Covers the metric-callable shape, reading the score delta + run link, and versioning the winner. Require a target example with an eval dataset; refuse to optimize without a dataset to score against.
+---
+
+# Run a prompt optimization
+
+`opik-optimizer` searches for a better prompt by scoring candidates against a dataset with a
+metric. It tunes the **prompt and model params** — not the retriever. Retrieval quality is a
+separate concern, watched via `ContextRecall` in `run-evals`.
+
+Reference: `run_optimization` in
+[`optimization.py`](../../../use-cases/f1_radio_rag/src/f1_radio_rag/optimization.py);
+`promote` in [`prompts.py`](../../../use-cases/f1_radio_rag/src/f1_radio_rag/prompts.py);
+CLI [`cli.py`](../../../use-cases/f1_radio_rag/src/f1_radio_rag/cli.py) `optimize`/`promote`.
+
+## When to refuse
+
+Need a **target example with an eval dataset** to score against — optimization is meaningless
+without a metric over real data. If there's no dataset, route to `add-dataset-items` first.
+
+## Step 1 — Credentials gate
+
+Needs `OPIK_API_KEY` + `OPIK_WORKSPACE` and the model provider key (e.g. `ANTHROPIC_API_KEY`).
+Dry-run first:
+
+```bash
+cd use-cases/f1_radio_rag
+uv run f1rag optimize                   # DRY_RUN: prints what it would optimize
+export ANTHROPIC_API_KEY=... OPIK_API_KEY=... OPIK_WORKSPACE=...
+uv run f1rag optimize                   # real run: prints initial -> optimised score + run link
+```
+
+## Step 2 — The metric is a plain callable
+
+Optimizer metrics are **not** the metric classes directly — they're callables
+`(dataset_item: dict, llm_output: str) -> float` with `__name__` set (the name shows in the
+Opik UI). Wrap an LLM-judge metric to score against something real:
+
+```python
+from opik.evaluation.metrics import AnswerRelevance
+
+def _relevance_metric(dataset_item: dict, llm_output: str) -> float:
+    result = AnswerRelevance(model=JUDGE_MODEL).score(
+        input=dataset_item["query"], output=llm_output, context=dataset_item["messages"],
+    )
+    return result.value
+
+_relevance_metric.__name__ = "answer_relevance"
+```
+
+## Step 3 — Build the prompt and run the optimizer
+
+```python
+from opik_optimizer import ChatPrompt, MetaPromptOptimizer
+
+prompt = ChatPrompt(name=PROMPT_NAME, system=SYSTEM_PROMPT, user=USER_TEMPLATE, model=GEN_MODEL)
+optimizer = MetaPromptOptimizer(model=OPTIMIZER_MODEL, n_threads=4, skip_perfect_score=False)
+result = optimizer.optimize_prompt(
+    prompt=prompt, dataset=dataset, metric=_relevance_metric, max_trials=12, n_samples=8,
+)
+```
+
+- `ChatPrompt.user` uses single-brace `{field}` placeholders that bind to dataset item fields.
+- `skip_perfect_score=False` forces every round even when the baseline already scores high.
+- `max_trials` / `n_samples` trade cost for thoroughness — keep them modest for demos.
+
+## Step 4 — Read the result
+
+- `result.initial_score` → `result.score` — the improvement (or lack of it). Report both.
+- `result.get_run_link()` — link to the run in Optimization Studio. Surface it.
+- `result.prompt.get_messages()` — the optimised messages, for promotion.
+
+## Step 5 — Promote (optional, only when asked or score improved)
+
+Saving versions the prompt in the Prompt Library; re-using the same `name` appends a version
+(upsert — no separate update call, no in-place edit).
+
+```python
+client.create_chat_prompt(
+    name=PROMPT_NAME,
+    messages=result.prompt.get_messages(),
+    change_description=f"optimised score {result.score:.3f}",
+    tags=["optimised"],
+)
+```
+
+Don't promote a regression. If `result.score <= result.initial_score`, report it and stop —
+ask before saving a worse prompt over a working one.
+
+## Verify
+
+- Dry-run prints the prompt/dataset it would optimize before the real run.
+- Real run prints `initial -> optimised` score and a run link; open it to inspect trials.
+- After promote, the new version appears under the prompt name in the Opik Prompt Library UI.