feat(score_traces): offline scoring of production traces with the Opik SDK (#61)

## What

A new opik-examples entry — `scripts/score_traces/` — that scores
**existing production
traces** in an Opik project offline and logs the feedback scores (with
reasons) back onto
those traces, using the same judge/metric classes and variable mappings
an online
evaluation rule would use. Designed so teams whose GenAI gateway isn't
yet reachable from
the Opik platform can run evals on a cron now and migrate to online
rules later with **no
re-authoring** (same score names, same variable mapping).

## How it works

Composes three public SDK primitives in a loop:
`search_traces` → `metric.score()` → `log_traces_feedback_scores` (one
batched write).

- `model.py` — judge-model / gateway wiring (LiteLLM; importable without
a gateway)
- `metrics.py` — the `EVALS` list: `hallucination` (preset), `relevance`
(G-Eval), `exact_match` (custom `BaseMetric`)
- `paths.py` — resolves a dotted `variables` mapping into each metric's
`score()` kwargs
- `score_traces.py` — the runner (windowed search → score → batched
log); per-eval error isolation; DRY_RUN-safe
- `utils/seed_traces.py` — seeds ~10 hardcoded Q&A traces (no LLM) for
an end-to-end demo
- `tests/test_score_traces.py` — 18 unit tests (LLM mocked)

## Notes

- Runs as plain scripts (`uv run python score_traces.py` /
`utils/seed_traces.py`) — no wheel/entry-point packaging.
- **DRY_RUN-safe:** with no `OPIK_API_KEY`/`OPIK_WORKSPACE`, both entry
points exit 0 without touching the network (the CI contract).
- G-Eval is fed a labeled `INPUT:/OUTPUT:` payload so `relevance` is
judged against the question (G-Eval's `score()` takes a single string).
- Docs cover judge auth (the LiteLLM `openai/` route also needs
`OPENAI_API_KEY`), never committing `OPIK_API_KEY` (prefer a service
account), and pipeline/CI env config.

## Testing

`uv run pytest` → **18 passed**; `uv run ruff check .` clean; both entry
points verified DRY_RUN-safe (exit 0, no network).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Author: LeoRoccoBreedt

README.md

@@ -57,6 +57,7 @@ Standalone scripts for automating and managing Opik resources.
 | [scripts/automate_annotation_queue](scripts/automate_annotation_queue/) | Route traces into annotation queues via batch or real-time assignment |
 | [scripts/usage_stats](scripts/usage_stats/) | Fetch trace and span counts per project and visualise trends |
 | [scripts/leaderboard_dashboard](scripts/leaderboard_dashboard/) | Create an Experiment Leaderboard dashboard via the REST API |
+| [scripts/score_traces](scripts/score_traces/) | Score existing production traces offline with stock SDK judges/metrics, logging feedback back under the same names an online rule would use |
 
 ## Community
 

scripts/score_traces/.env.example

@@ -0,0 +1,28 @@
+# Copy to .env (git-ignored) and fill in. Credentials come from Opik → User menu → API key.
+# SECURITY: never commit real keys. In CI/cron, inject them from your secret store rather
+# than a file, and prefer a dedicated Opik *service account* over a personal API key.
+OPIK_API_KEY=
+OPIK_WORKSPACE=
+# Self-hosted only — the Opik API base URL (default: Opik Cloud).
+OPIK_URL_OVERRIDE=https://www.comet.com/opik/api
+# Project whose production traces will be scored (must already contain traces;
+# run `uv run python utils/seed_traces.py` to populate a test project).
+OPIK_PROJECT_NAME=score-traces-example
+
+# How far back each run looks (stateless time window). An hourly cron uses 1.
+EVAL_WINDOW_HOURS=1
+# Max traces fetched per run. Size against your window + traffic; if a run hits
+# this cap the summary prints a truncation warning (raise it or shorten the window).
+EVAL_MAX_RESULTS=1000
+
+# --- LLM judge model, via LiteLLM (see model.py) ---
+# Path A (OpenAI-compatible gateway): base URL + key for LiteLLM.
+GATEWAY_BASE_URL=
+GATEWAY_API_KEY=
+GATEWAY_MODEL=gpt-4o
+# CI routes judges to a cheap model via OPIK_EXAMPLES_MODEL (used when GATEWAY_MODEL is unset).
+# OPIK_EXAMPLES_MODEL=
+# The judge is routed as `openai/$GATEWAY_MODEL`, so LiteLLM also reads the OpenAI
+# provider key from OPENAI_API_KEY. Set this (to your gateway/OpenAI key) if the judge
+# errors on auth — it is required for the openai/ route even when GATEWAY_* are set.
+OPENAI_API_KEY=

scripts/score_traces/README.md

@@ -0,0 +1,178 @@
+# Score Production Traces
+
+Score **existing production traces** in an Opik project from your own pipeline — using
+the same judge/metric classes an [online evaluation rule](https://www.comet.com/docs/opik/production/rules)
+would use — and log the scores back onto those traces. For teams whose GenAI gateway
+is reachable **from their code but not yet from the Opik platform**: run evals on a cron
+now, then switch to platform online rules later with **no re-authoring** (same score
+names, same variable mapping).
+
+> **This is not `opik.evaluate()`.** `evaluate()` runs a *task* over a *dataset* to
+> generate new outputs and create an experiment (pre-production). This scores traces that
+> **already exist** and attaches feedback in place (post-production) — the job online
+> rules do. There is no single turnkey SDK call for that, so this example composes three
+> primitives: `search_traces` → `metric.score()` → `log_traces_feedback_scores`.
+
+## Prerequisites
+
+- Python 3.12+
+- [uv](https://docs.astral.sh/uv/) — this folder is a `uv` project; run `uv sync`
+- A GenAI gateway reachable from where you run this (for the LLM judges)
+
+## Environment Setup
+
+| Variable | Required | Default | Description |
+|---|---|---|---|
+| `OPIK_API_KEY` | for a live run | — | Opik API key (unset → DRY_RUN, no network) |
+| `OPIK_WORKSPACE` | for a live run | — | Opik workspace name |
+| `OPIK_URL_OVERRIDE` | self-hosted | `https://www.comet.com/opik/api` | Opik API base URL |
+| `OPIK_PROJECT_NAME` | no | `score-traces-example` | Project whose traces are scored |
+| `EVAL_WINDOW_HOURS` | no | `1` | How far back each run looks |
+| `EVAL_MAX_RESULTS` | no | `1000` | Max traces per run; warns on truncation |
+| `GATEWAY_BASE_URL` | for LLM judges | — | OpenAI-compatible gateway URL (Path A) |
+| `GATEWAY_API_KEY` | for LLM judges | — | Gateway API key |
+| `GATEWAY_MODEL` | no | `gpt-4o` | Judge model name (local knob) |
+| `OPIK_EXAMPLES_MODEL` | no | — | CI routes judges to a cheap model via this (used when `GATEWAY_MODEL` is unset) |
+| `OPENAI_API_KEY` | often | — | OpenAI provider key LiteLLM uses for the `openai/` judge route — set alongside `GATEWAY_*` if the judge errors on auth (see note below) |
+
+> **Judge auth (LiteLLM).** The judges run through LiteLLM, which routes the model as
+> `openai/$GATEWAY_MODEL`. Set your gateway via `GATEWAY_BASE_URL` + `GATEWAY_API_KEY`, and
+> **also set `OPENAI_API_KEY`** (to your gateway/OpenAI key) — the `openai/` route reads the
+> provider key from it and will error on auth without it, even when the `GATEWAY_*` vars are set.
+
+### Credentials & security
+
+- **Never commit or hard-code `OPIK_API_KEY`** or gateway keys — keep them out of `run.sh`,
+  source, and the committed `.env.example`. Inject them at runtime from your CI secret store
+  or a secret manager; your real `.env` should stay git-ignored.
+- Prefer a **dedicated Opik service account** for scheduled/CI runs over a personal API key,
+  so the job's access is scoped and can be revoked independently of any individual.
+
+### Running in a pipeline / on a schedule
+
+`OPIK_WORKSPACE` and `OPIK_PROJECT_NAME` are plain env vars — set them wherever the job runs:
+export them in your CI/cron pipeline, or edit `run.sh` (the CI entry point). With
+`OPIK_API_KEY`/`OPIK_WORKSPACE` unset the run falls back to DRY_RUN and exits 0 without
+touching the network, so CI stays green before credentials are wired in.
+
+## Workflow
+
+```mermaid
+flowchart LR
+    A["search_traces<br/>(last N hours)"] --> B["for each trace × eval:<br/>resolve variables → metric.score()"]
+    B --> C["log_traces_feedback_scores<br/>(one batched call, same names + reason)"]
+    C --> D["scores visible on the<br/>same traces in the Opik UI"]
+    subgraph later["when the gateway reaches Opik (Stage 1)"]
+        E["same metrics.py"] --> F["create online rules<br/>(same names + variable mapping)"]
+    end
+    C -. same names & mapping .-> F
+```
+
+### Step 0 — seed a test project (optional, no LLM)
+
+```bash
+export OPIK_API_KEY=... OPIK_WORKSPACE=...
+export OPIK_PROJECT_NAME=score-traces-example
+uv run python utils/seed_traces.py   # ~10 hardcoded Q&A traces, good/bad mix
+```
+
+### Step 1 — configure the judge model
+
+Edit `model.py`. Path A (default) points LiteLLM at an
+OpenAI-compatible gateway via `GATEWAY_BASE_URL` / `GATEWAY_API_KEY`. For a non-standard
+gateway, use Path B (a custom `OpikBaseModel` subclass). Both paths, with full worked
+examples, are documented here:
+<https://www.comet.com/docs/opik/evaluation/metrics/custom_model>
+([`LiteLLMChatModel`](https://www.comet.com/docs/opik/python-sdk-reference/Objects/LiteLLMChatModel.html) ·
+[`OpikBaseModel`](https://www.comet.com/docs/opik/python-sdk-reference/Objects/OpikBaseModel.html)).
+Set `GATEWAY_BASE_URL` + `GATEWAY_API_KEY` + `OPENAI_API_KEY` (see the **Judge auth** note above).
+
+### Step 2 — run the evals
+
+```bash
+export EVAL_WINDOW_HOURS=1
+uv run python score_traces.py     # scores the last hour, logs feedback back
+```
+
+Open the project in Opik — each trace now carries `hallucination`, `relevance`, and
+`exact_match` feedback scores, each with the judge's **reason**.
+
+## How it works (the primitives)
+
+The runner is nothing more than three public SDK primitives composed in a loop:
+
+**1. Pull existing traces**
+
+```python
+import opik
+
+client = opik.Opik(project_name="score-traces-example")
+traces = client.search_traces(
+    project_name="score-traces-example",
+    filter_string='start_time >= "2026-07-29T00:00:00Z"',
+    max_results=1000,
+)
+trace = traces[0]
+print(trace.id, trace.input, trace.output)  # input/output are dicts
+```
+
+**2. Score one trace with a metric + variable mapping**
+
+```python
+from opik.evaluation.metrics import Hallucination
+
+judge = Hallucination(model="gpt-4o", name="hallucination")
+
+# variable mapping: metric score() param -> trace field path
+variables = {"input": "input.question", "output": "output.answer", "context": "output.context"}
+kwargs = {
+    "input": trace.input["question"],
+    "output": trace.output["answer"],
+    "context": trace.output["context"],
+}
+result = judge.score(**kwargs)
+print(result.value, result.reason)
+```
+
+**3. Write the score back onto the trace**
+
+```python
+client.log_traces_feedback_scores(
+    [
+        {"id": trace.id, "name": "hallucination", "value": result.value, "reason": result.reason},
+    ]
+)
+```
+
+**Put it together:** `score_traces.py` is exactly these three steps — for every eval in
+`EVALS`, over every trace in the window, with one batched write at the end. Schedule
+it on a cron (e.g. hourly with `EVAL_WINDOW_HOURS=1`).
+
+## Adding or changing evals
+
+Edit the `EVALS` list in `metrics.py`. Each `Eval` has a `name` (the feedback-score name),
+a live `metric` object, and a `variables` mapping (metric `score()` param → trace field
+path). Copy a block to add one. Three styles ship:
+
+| Eval | Type | Migrates to an online rule as |
+|---|---|---|
+| `hallucination` | built-in preset judge | the platform preset — same engine, **same score name** (strong, not a literal string copy) |
+| `relevance` | G-Eval custom judge | an `llm_as_judge` rule — criteria text + name map directly (**clean**) |
+| `exact_match` | Python metric (`metrics.py`) | a `user_defined_metric_python` rule — the exact source round-trips (**perfect**) |
+
+The `variables` mapping is the same **"variable mapping"** you set on a rule in the Opik UI —
+so when you migrate, the mapping and score names carry over verbatim and your dashboards
+don't change. G-Eval's `score()` evaluates a single labeled `output` string, not separate
+kwargs, so for `relevance` the runner composes that string from `variables` (e.g.
+`INPUT: ...\nOUTPUT: ...`) — the mapping still names the same trace fields an online G-Eval
+rule would map.
+
+## Roadmap (not in this example)
+
+- **Stage 1 — promote to online rules.** A small script reading the same `metrics.py`
+  and creating the online rules (reuses
+  [`scripts/online_eval_rules`](../online_eval_rules)). For multi-output custom judges it
+  adds an output-schema hint.
+- **Stage 2 — productionize.** Config-driven CLI; **watermarking** (persist a last-processed
+  timestamp instead of a fixed window); **span- and thread-scope** evaluation. A large
+  custom Path B model can also move into its own `model_provider.py`.

scripts/score_traces/metrics.py

@@ -0,0 +1,85 @@
+"""Metrics + eval definitions — single source of truth for offline evaluation.
+
+Contains a minimal user-defined Opik metric (no LLM) for the Python-metric example
+(the SDK's own metric interface: subclass BaseMetric and implement score()), plus
+the EVALS list: the judges & metrics to run (copy a block to add one).
+
+The score `name` and `variables` mapping here are identical to what an online
+evaluation rule needs, so migrating later is a drop-in (see README, Stage 1).
+"""
+
+from dataclasses import dataclass
+from typing import Any
+
+from opik.evaluation.metrics import GEval, Hallucination, base_metric, score_result
+from opik.evaluation.metrics.base_metric import BaseMetric
+
+from model import build_judge_model
+
+
+class ExactMatch(base_metric.BaseMetric):
+    """Scores 1.0 when output exactly equals reference, else 0.0."""
+
+    def __init__(self, name: str = "exact_match"):
+        super().__init__(name=name, track=False)
+
+    def score(self, output: str, reference: str, **ignored: Any) -> score_result.ScoreResult:
+        hit = output == reference
+        return score_result.ScoreResult(
+            value=1.0 if hit else 0.0,
+            name=self.name,
+            reason="exact match" if hit else f"output {output!r} != reference {reference!r}",
+        )
+
+
+judge_model = build_judge_model()
+
+
+@dataclass
+class Eval:
+    """One evaluation: its score name, the live SDK metric, and the variable mapping.
+
+    `variables` maps each metric score() parameter to a dotted trace field-path
+    (the Opik UI "variable mapping"). `name` is the feedback-score name.
+    """
+
+    name: str
+    metric: BaseMetric
+    variables: dict[str, str]
+
+
+# ============================================================
+# 2. EVALS — your judges & metrics (copy a block to add one)
+# ============================================================
+EVALS: list[Eval] = [
+    # Built-in preset judge (lead example).
+    Eval(
+        name="hallucination",
+        metric=Hallucination(model=judge_model, name="hallucination"),
+        variables={
+            "input": "input.question",
+            "output": "output.answer",
+            "context": "output.context",
+        },
+    ),
+    # G-Eval custom judge (author criteria as text).
+    Eval(
+        name="relevance",
+        metric=GEval(
+            model=judge_model,
+            name="relevance",
+            task_introduction=("You judge whether an answer (OUTPUT) is relevant to the question (INPUT)."),
+            evaluation_criteria=(
+                "Return 1 if the OUTPUT directly addresses the INPUT question, 0 if it is "
+                "off-topic or evasive."
+            ),
+        ),
+        variables={"input": "input.question", "output": "output.answer"},
+    ),
+    # User-defined Python metric (no LLM).
+    Eval(
+        name="exact_match",
+        metric=ExactMatch(name="exact_match"),
+        variables={"output": "output.answer", "reference": "input.expected"},
+    ),
+]

scripts/score_traces/model.py

@@ -0,0 +1,55 @@
+"""Judge model / GenAI gateway wiring.
+
+Point the LLM judges at your GenAI gateway (edit once). This is the ONLY
+file about the model.
+"""
+
+import os
+
+from opik.evaluation import models
+from opik.evaluation.models import base_model
+
+JudgeModel = str | base_model.OpikBaseModel
+
+
+# ============================================================
+# 1. JUDGE MODEL — set up your GenAI gateway here (edit once)
+#    Path A: LiteLLMChatModel(...)           ← OpenAI-compatible gateway (default)
+#    Path B: a custom OpikBaseModel subclass ← non-standard gateway
+#    Full worked examples for BOTH paths:
+#      https://www.comet.com/docs/opik/evaluation/metrics/custom_model
+# ============================================================
+def build_judge_model() -> JudgeModel:
+    """Return the model every LLM judge uses.
+
+    Path A (default): an OpenAI-compatible gateway via LiteLLM. Set GATEWAY_BASE_URL
+    + GATEWAY_API_KEY + GATEWAY_MODEL. LiteLLMChatModel forwards base_url/api_key to
+    litellm.completion, so the judge's calls route through your gateway. Because the
+    model is routed as ``openai/<GATEWAY_MODEL>``, LiteLLM also reads OPENAI_API_KEY as
+    the provider key — set it (to your gateway/OpenAI key) if the judge errors on auth.
+
+    If GATEWAY_BASE_URL is unset, falls back to a bare model name (GATEWAY_MODEL, or
+    OPIK_EXAMPLES_MODEL), which lets the module import and unit-test without a gateway.
+    """
+    # GATEWAY_MODEL is the local knob; OPIK_EXAMPLES_MODEL lets CI route judges to a
+    # cheap model (used when GATEWAY_MODEL is unset).
+    model_name = os.environ.get("GATEWAY_MODEL") or os.environ.get("OPIK_EXAMPLES_MODEL", "gpt-4o")
+    base_url = os.environ.get("GATEWAY_BASE_URL")
+    if not base_url:
+        return model_name  # importable / testable without a gateway
+    return models.LiteLLMChatModel(
+        model_name=f"openai/{model_name}",
+        base_url=base_url,
+        api_key=os.environ.get("GATEWAY_API_KEY"),
+        temperature=0.0,
+    )
+
+    # ---- Path B: non-standard gateway (uncomment & implement) --------------
+    # from opik.evaluation.models import OpikBaseModel
+    # class MyGatewayModel(OpikBaseModel):
+    #     def __init__(self, model_name: str):
+    #         super().__init__(model_name=model_name)
+    #     def generate_string(self, input: str, **kwargs) -> str: ...
+    #     def generate_provider_response(self, **kwargs): ...
+    # return MyGatewayModel(model_name)
+    # See: https://www.comet.com/docs/opik/evaluation/metrics/custom_model

scripts/score_traces/paths.py

@@ -0,0 +1,36 @@
+"""Resolve dotted trace field-paths into the kwargs a metric's score() expects.
+
+A `variables` mapping looks like {"input": "input.question", "output": "output.answer"}:
+each key is a metric score() parameter, each value a dotted path into the trace's
+top-level dicts (`input`, `output`, `metadata`). This is the Opik "variable mapping".
+"""
+
+from typing import Any
+
+
+class MissingField(Exception):
+    """A variable's dotted path could not be resolved against a trace."""
+
+
+def resolve_path(trace_data: dict[str, Any], path: str) -> Any:
+    """Read `path` (e.g. "output.context") out of trace_data.
+
+    First segment selects a top-level key (input/output/metadata); remaining
+    segments index into nested dicts. List/scalar leaf values are returned as-is.
+    Raises MissingField if any segment is absent or a non-dict is indexed.
+    """
+    segments = path.split(".")
+    current: Any = trace_data
+    traversed: list[str] = []
+    for segment in segments:
+        if not isinstance(current, dict) or segment not in current:
+            where = ".".join(traversed) or "<root>"
+            raise MissingField(f"cannot resolve '{path}': '{segment}' missing under '{where}'")
+        current = current[segment]
+        traversed.append(segment)
+    return current
+
+
+def resolve_variables(trace_data: dict[str, Any], variables: dict[str, str]) -> dict[str, Any]:
+    """Map {param: dotted_path} to {param: resolved_value}. Propagates MissingField."""
+    return {param: resolve_path(trace_data, path) for param, path in variables.items()}