Skip to content

Commit 17a4d0b

Browse files
LeoRoccoBreedtLeoRoccoBreedt
authored
feat: online evaluation rules example (SDK & REST) (#33)
## What New `scripts/online_eval_rules/` example: create and manage Opik **online evaluation rules** (automation rule evaluators) from code — Python SDK and raw REST side by side, across every common rule type. ## Why Online eval rules auto-score traces/threads/spans as they arrive (LLM-as-judge or user-defined Python metric). This example provisions and manages them programmatically (project onboarding, CI, bulk setup) instead of clicking through the UI. ## Highlights - **One payload, two surfaces** — `build_payload()` produces the JSON the REST API accepts; `via_sdk()` maps it to the typed `AutomationRuleEvaluatorWrite_*` union, `via_rest()` sends it with `requests`. `--surface` picks the live path. - **Dry-run shows both** — prints the SDK call *and* an equivalent `curl` from the same payload, so the two never drift. Runs with no credentials. - **Rule types**: `llm_as_judge`, `user_defined_metric_python`, `trace_thread_llm_as_judge`, `span_llm_as_judge`, `span_user_defined_metric_python`. - **Full lifecycle**: create, list, get, update (sampling rate / enabled), delete. - Project resolved-or-created by name; actionable error when an LLM provider key is missing for judge rules. - Tests in `test_create_online_eval_rules.py` (271 lines). ## Notes - `update` always uses REST (SDK update-union types differ) — carries `project_ids` forward so the evaluators PATCH endpoint doesn't 400 on `"At least one project must be specified"`. - `.gitignore`: ignore `*.log`. ## Test plan - `uv run create-online-eval-rules create-llm-judge --name relevance --dry-run` — prints SDK snippet + curl, no creds. - Live (creds set): create each rule type, then `list` / `get` / `update --sampling-rate 0.2 --disabled` / `update --enabled` / `delete --yes`. - `uv run pytest` in `scripts/online_eval_rules/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
2 parents a2a48fe + c525029 commit 17a4d0b

9 files changed

Lines changed: 918 additions & 0 deletions

File tree

‎scripts/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,3 +8,4 @@ Standalone utility scripts for automating and managing Opik resources programmat
88
| [automate_annotation_queue/](./automate_annotation_queue/) | Automatically route traces into annotation queues via batch or real-time assignment |
99
| [usage_stats/](./usage_stats/) | Fetch trace, thread, and span counts per project and visualise daily and cumulative trends |
1010
| [leaderboard_dashboard/](./leaderboard_dashboard/) | Create an Experiment Leaderboard dashboard entirely via the REST API |
11+
| [online_eval_rules/](./online_eval_rules/) | Create & manage online evaluation rules (automation rule evaluators) via the SDK and REST, all rule types, full CRUD |

‎scripts/online_eval_rules/.env.example‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
# Copy to .env and fill in. With no key/workspace the script runs in DRY_RUN.
2+
OPIK_API_KEY=
3+
OPIK_WORKSPACE=
4+
# Self-hosted only — the Opik API base URL (default: Opik Cloud).
5+
OPIK_URL_OVERRIDE=https://www.comet.com/opik/api
6+
# Project the rules attach to (created if it doesn't exist).
7+
OPIK_PROJECT_NAME=online-eval-rules-example

‎scripts/online_eval_rules/.gitignore‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
.env
2+
__pycache__/
3+
*.pyc
4+
.venv/
5+
uv.lock
6+
*.log

‎scripts/online_eval_rules/README.md‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
# Online Evaluation Rules — SDK & REST
2+
3+
Create and manage Opik **online evaluation rules** (automation rule evaluators) from code —
4+
shown with the Python SDK and the raw REST API side by side, across every common rule type.
5+
6+
## What this does
7+
8+
Online evaluation rules automatically score your traces, threads, and spans as they arrive —
9+
using an LLM-as-judge prompt or a user-defined Python metric. This example provisions and manages
10+
those rules programmatically (project onboarding, CI, bulk setup) instead of clicking through the
11+
UI. One payload is built per rule and sent through either surface; in dry-run it prints **both**
12+
the SDK call and an equivalent `curl`, so you can copy whichever fits your stack.
13+
14+
Rule types covered: `llm_as_judge`, `user_defined_metric_python`, `trace_thread_llm_as_judge`,
15+
`span_llm_as_judge`, `span_user_defined_metric_python`. Full lifecycle: create, list, get, update
16+
(sampling rate / enabled), delete.
17+
18+
## Prerequisites
19+
20+
```bash
21+
uv sync # recommended (this folder is a uv project)
22+
# or: pip install "opik>=2.0" requests
23+
```
24+
25+
| Environment variable | Required | Description |
26+
|---|---|---|
27+
| `OPIK_API_KEY` | for a live run | Opik API key. Unset → the script runs in DRY_RUN |
28+
| `OPIK_WORKSPACE` | for a live run | Opik workspace name |
29+
| `OPIK_URL_OVERRIDE` | No | Opik API base URL for self-hosted (default: `https://www.comet.com/opik/api`) |
30+
| `OPIK_PROJECT_NAME` | No | Project the rules attach to; created if absent (default: `online-eval-rules-example`) |
31+
32+
> **LLM-as-judge rules also need an LLM provider key configured in your workspace**
33+
> (Opik → Configuration → AI providers). Python-metric rules do not. If it's missing, a live
34+
> create prints an actionable error. The project itself is created automatically by name.
35+
36+
## Running it
37+
38+
```bash
39+
# Dry-run — no credentials needed. Prints the SDK call AND the curl for the rule.
40+
uv run create-online-eval-rules create-llm-judge --name relevance --dry-run
41+
42+
# Live — set credentials, then create each rule type.
43+
export OPIK_API_KEY="<your-key>"
44+
export OPIK_WORKSPACE="<your-workspace>"
45+
46+
uv run create-online-eval-rules create-llm-judge --name relevance
47+
uv run create-online-eval-rules create-python --name exact-match
48+
uv run create-online-eval-rules create-thread --name convo-quality
49+
uv run create-online-eval-rules create-span --name span-relevance
50+
uv run create-online-eval-rules create-span --name span-check --python
51+
52+
# Choose the live surface (default sdk):
53+
uv run create-online-eval-rules create-llm-judge --name relevance --surface rest
54+
55+
# Manage:
56+
uv run create-online-eval-rules list
57+
uv run create-online-eval-rules get --id <rule-id>
58+
uv run create-online-eval-rules update --id <rule-id> --sampling-rate 0.2 --disabled
59+
uv run create-online-eval-rules update --id <rule-id> --enabled # re-enable
60+
uv run create-online-eval-rules delete --id <rule-id> --yes
61+
```
62+
63+
> Note: `update` always uses the REST surface (the SDK update-union types differ).
64+
65+
## How it works
66+
67+
1. **Config** — credentials + project come from env vars; `DRY_RUN` is on whenever the
68+
key/workspace pair is absent (or `--dry-run` is passed).
69+
2. **One payload** — `build_payload()` returns the plain JSON dict the REST API accepts, per rule
70+
type (thread judges omit `variables`; Python-metric rules embed `metric_example.py` as source).
71+
3. **Two adapters** — `via_sdk()` maps the dict to the typed `AutomationRuleEvaluatorWrite_*` union
72+
and calls `opik.Opik().rest_client.automation_rule_evaluators.*`; `via_rest()` sends the same
73+
dict with `requests`. `--surface` picks the live path.
74+
4. **Dry-run shows both** — `render_sdk_snippet()` and `render_curl()` print the SDK call and the
75+
curl from the one payload, so the two surfaces never drift.

0 commit comments

Comments
 (0)