Skip to content

Commit 58db059

Browse files
LeoRoccoBreedtLeoRoccoBreedt
authored
Merge branch 'main' into googleadk-integration
2 parents 77291c4 + d31accb commit 58db059

27 files changed

Lines changed: 948 additions & 380 deletions

File tree

‎.github/workflows/compliance.yml‎

Lines changed: 30 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -3,12 +3,13 @@ name: Example Compliance Check
33
# Verifies that new example folders added in a PR follow repo conventions.
44
# Only checks folders that are new (not previously on the base branch).
55
# Rules enforced:
6-
# 1. run.sh must exist and contain set -e
7-
# 2. OPIK_PROJECT_NAME must be set — either exported in run.sh (scripts)
8-
# or referenced in a .py file (use-cases/guides with a config module)
6+
# 1. run.sh must exist and contain set -e — EXCEPT notebook examples (folders with a
7+
# .ipynb), which are executed by test-notebooks.yml and need no run.sh
8+
# 2. OPIK_PROJECT_NAME must be set — exported in run.sh (scripts), or referenced in a
9+
# .py file (use-cases/guides) or a notebook cell (notebook examples)
910
# 3. README.md must exist
1011
# 4. pyproject.toml must exist; requirements.txt must not
11-
# 5. If litellm is a dependency, OPIK_EXAMPLES_MODEL must be referenced in a .py file
12+
# 5. If litellm is a dependency, OPIK_EXAMPLES_MODEL must be referenced in a .py/.ipynb file
1213
#
1314
# This check is secrets-free so it runs safely on PRs from forks.
1415

@@ -86,11 +87,20 @@ jobs:
8687
echo ""
8788
echo "Checking: $folder"
8889
89-
if [[ ! -f "$folder/run.sh" ]]; then
90+
# Notebook examples (folders with a .ipynb) are executed by test-notebooks.yml
91+
# and don't need run.sh.
92+
is_notebook=false
93+
if ls "$folder"/*.ipynb >/dev/null 2>&1; then
94+
is_notebook=true
95+
fi
96+
97+
if [[ -f "$folder/run.sh" ]]; then
98+
echo " OK run.sh"
99+
elif [[ "$is_notebook" == true ]]; then
100+
echo " OK notebook example (run.sh not required — executed by test-notebooks.yml)"
101+
else
90102
echo "::error file=$folder/run.sh::$folder is missing run.sh"
91103
FAILED=1
92-
else
93-
echo " OK run.sh"
94104
fi
95105
96106
if [[ ! -f "$folder/README.md" ]]; then
@@ -119,24 +129,24 @@ jobs:
119129
else
120130
echo " OK set -e in run.sh"
121131
fi
132+
fi
122133
123-
# OPIK_PROJECT_NAME may be exported in run.sh (scripts)
124-
# or defined as a constant in a .py file (use-cases/guides with config.py).
125-
if grep -q "export OPIK_PROJECT_NAME" "$folder/run.sh"; then
126-
echo " OK OPIK_PROJECT_NAME in run.sh"
127-
elif grep -rq "OPIK_PROJECT_NAME" "$folder" --include="*.py" 2>/dev/null; then
128-
echo " OK OPIK_PROJECT_NAME in .py config"
129-
else
130-
echo "::error::$folder must set OPIK_PROJECT_NAME — either export it in run.sh or define it in a .py config file"
131-
FAILED=1
132-
fi
134+
# OPIK_PROJECT_NAME may be exported in run.sh (scripts), defined as a constant
135+
# in a .py file (use-cases/guides), or set in a notebook cell (notebook examples).
136+
if [[ -f "$folder/run.sh" ]] && grep -q "export OPIK_PROJECT_NAME" "$folder/run.sh"; then
137+
echo " OK OPIK_PROJECT_NAME in run.sh"
138+
elif grep -rq "OPIK_PROJECT_NAME" "$folder" --include="*.py" --include="*.ipynb" 2>/dev/null; then
139+
echo " OK OPIK_PROJECT_NAME in .py/.ipynb"
140+
else
141+
echo "::error::$folder must set OPIK_PROJECT_NAME — export it in run.sh, or define it in a .py config or notebook cell"
142+
FAILED=1
133143
fi
134144
135145
# If litellm is declared as a dependency, OPIK_EXAMPLES_MODEL must be referenced.
136146
# Match a quoted dependency entry ("litellm", "litellm>=...", "litellm[extra]")
137147
# so a bare mention in a comment doesn't trip the check.
138148
if grep -qE '"litellm' "$folder/pyproject.toml" 2>/dev/null; then
139-
if grep -rq "OPIK_EXAMPLES_MODEL" "$folder" --include="*.py" 2>/dev/null; then
149+
if grep -rq "OPIK_EXAMPLES_MODEL" "$folder" --include="*.py" --include="*.ipynb" 2>/dev/null; then
140150
echo " OK OPIK_EXAMPLES_MODEL referenced (litellm dep detected)"
141151
else
142152
echo "::error::$folder uses litellm but no .py file references OPIK_EXAMPLES_MODEL"
@@ -163,8 +173,8 @@ jobs:
163173
echo "Folders checked: \`${{ steps.changed-folders.outputs.folders }}\`"
164174
echo ""
165175
echo "**Rules:**"
166-
echo "- \`run.sh\` must exist and contain \`set -e\`"
167-
echo "- \`OPIK_PROJECT_NAME\` must be set — via \`export\` in \`run.sh\` (scripts) or as a constant in a \`.py\` config file (use-cases/guides)"
176+
echo "- \`run.sh\` must exist and contain \`set -e\` (except notebook examples, executed by test-notebooks.yml)"
177+
echo "- \`OPIK_PROJECT_NAME\` must be set — via \`export\` in \`run.sh\` (scripts), a \`.py\` config (use-cases/guides), or a notebook cell"
168178
echo "- \`README.md\` must exist"
169179
echo "- \`pyproject.toml\` must exist; \`requirements.txt\` is not allowed"
170180
echo "- If \`litellm\` is a dependency, a \`.py\` file must reference \`OPIK_EXAMPLES_MODEL\`"

‎.github/workflows/test-notebooks.yml‎

Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,144 @@
1+
name: PR Notebook Tests
2+
3+
# Executes notebook examples changed in a PR end-to-end against Opik.
4+
# - Notebook examples are credential-gated: they log live traces, so there is no
5+
# secrets-free dry-run. Fork PRs (no secrets) skip execution; the compliance
6+
# check still runs. Same-repo PRs run them live with real Opik credentials.
7+
# - Each notebook is executed with `ipython <notebook>.ipynb`, which runs every cell
8+
# top-to-bottom and exits non-zero on the first error.
9+
# Any job failure blocks the PR.
10+
11+
on:
12+
pull_request:
13+
paths:
14+
- "examples/**"
15+
- "integrations/**"
16+
- "scripts/**"
17+
- "use-cases/**"
18+
- "guides/**"
19+
20+
permissions:
21+
contents: read
22+
23+
concurrency:
24+
group: test-notebooks-${{ github.ref }}
25+
cancel-in-progress: true
26+
27+
jobs:
28+
detect-changes:
29+
runs-on: ubuntu-latest
30+
outputs:
31+
folders: ${{ steps.detect.outputs.folders }}
32+
has_changes: ${{ steps.detect.outputs.has_changes }}
33+
has_secrets: ${{ steps.secrets.outputs.has_secrets }}
34+
steps:
35+
- uses: actions/checkout@v7
36+
with:
37+
fetch-depth: 0
38+
39+
- name: Detect changed notebook example folders
40+
id: detect
41+
run: |
42+
changed_files=$(git diff --name-only "origin/${{ github.base_ref }}...HEAD")
43+
44+
seen=()
45+
folders_json="["
46+
47+
while IFS= read -r file; do
48+
top=$(echo "$file" | cut -d'/' -f1)
49+
case "$top" in
50+
integrations)
51+
folder=$(echo "$file" | cut -d'/' -f1,2,3)
52+
;;
53+
examples|scripts|use-cases|guides)
54+
folder=$(echo "$file" | cut -d'/' -f1,2)
55+
;;
56+
*)
57+
continue
58+
;;
59+
esac
60+
61+
# Notebook examples only: the folder must contain at least one .ipynb.
62+
if [[ -z "$folder" ]] || [[ ! -d "$folder" ]]; then
63+
continue
64+
fi
65+
if ! ls "$folder"/*.ipynb >/dev/null 2>&1; then
66+
continue
67+
fi
68+
69+
already_seen=false
70+
for s in "${seen[@]}"; do
71+
[[ "$s" == "$folder" ]] && already_seen=true && break
72+
done
73+
74+
if [[ "$already_seen" == false ]]; then
75+
seen+=("$folder")
76+
[[ "${#seen[@]}" -gt 1 ]] && folders_json="$folders_json,"
77+
folders_json="$folders_json\"$folder\""
78+
fi
79+
done <<< "$changed_files"
80+
81+
folders_json="$folders_json]"
82+
83+
if [[ "${#seen[@]}" -eq 0 ]]; then
84+
echo "folders=[]" >> "$GITHUB_OUTPUT"
85+
echo "has_changes=false" >> "$GITHUB_OUTPUT"
86+
echo "No notebook example folders changed."
87+
else
88+
echo "folders=$folders_json" >> "$GITHUB_OUTPUT"
89+
echo "has_changes=true" >> "$GITHUB_OUTPUT"
90+
echo "Notebook folders to test: ${seen[*]}"
91+
fi
92+
93+
- name: Check whether Opik secrets are available
94+
id: secrets
95+
# secrets.* is not available in job-level `if:`, so resolve it here and expose a
96+
# boolean the run job gates on. Fork PRs get no secrets, so notebooks don't run.
97+
env:
98+
OPIK_API_KEY: ${{ secrets.OPIK_API_KEY }}
99+
run: |
100+
if [[ -n "$OPIK_API_KEY" ]]; then
101+
echo "has_secrets=true" >> "$GITHUB_OUTPUT"
102+
echo "Opik secrets available — notebooks will run live."
103+
else
104+
echo "has_secrets=false" >> "$GITHUB_OUTPUT"
105+
echo "No Opik secrets (fork PR) — notebook execution skipped."
106+
fi
107+
108+
run-notebooks:
109+
needs: detect-changes
110+
if: needs.detect-changes.outputs.has_changes == 'true' && needs.detect-changes.outputs.has_secrets == 'true'
111+
runs-on: ubuntu-latest
112+
timeout-minutes: 30
113+
strategy:
114+
fail-fast: false
115+
matrix:
116+
folder: ${{ fromJson(needs.detect-changes.outputs.folders) }}
117+
steps:
118+
- uses: actions/checkout@v7
119+
120+
- uses: astral-sh/setup-uv@v7
121+
with:
122+
version: "latest"
123+
enable-cache: true
124+
python-version: "3.12"
125+
126+
- name: Execute notebooks (live, real Opik credentials)
127+
uses: nick-fields/retry@v3
128+
env:
129+
OPIK_API_KEY: ${{ secrets.OPIK_API_KEY }}
130+
OPIK_WORKSPACE: ${{ vars.OPIK_WORKSPACE }}
131+
OPIK_ENVIRONMENT: ${{ vars.OPIK_ENVIRONMENT }}
132+
OPIK_EXAMPLES_MODEL: ${{ vars.OPIK_EXAMPLES_MODEL }}
133+
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
134+
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
135+
with:
136+
timeout_minutes: 25
137+
max_attempts: 2
138+
command: |
139+
cd "${{ matrix.folder }}"
140+
uv sync
141+
for nb in *.ipynb; do
142+
echo "Executing $nb ..."
143+
uv run --with ipython --with nbformat ipython "$nb"
144+
done

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,3 +23,6 @@ final_model/
2323
uv.lock
2424
.env
2525
poetry.lock
26+
27+
# Local planning/process artifacts (superpowers brainstorm specs & implementation plans) — kept local, never committed
28+
docs/superpowers/

‎CONTRIBUTING.md‎

Lines changed: 21 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -93,6 +93,8 @@ Then, either way:
9393
- Credentials loaded from environment variables only — no hardcoded keys
9494
- A `run.sh` that exports `OPIK_PROJECT_NAME` and can run the example end-to-end
9595

96+
> **Notebook examples are the exception.** A `.ipynb` guide ships `notebook.ipynb` + `pyproject.toml` + `README.md` and is executed live by [`test-notebooks.yml`](.github/workflows/test-notebooks.yml) — no `run.sh` and no dry-run. See [Notebook examples](#notebook-examples).
97+
9698
### README structure
9799

98100
Use the template's README as a guide. Required sections:
@@ -104,6 +106,8 @@ Use the template's README as a guide. Required sections:
104106

105107
### Dry-run mode
106108

109+
> Notebook examples are exempt — they require credentials and have no dry-run (see [Notebook examples](#notebook-examples)).
110+
107111
Every runnable example must work without credentials. The secrets-free CI `dry-run` job runs `bash run.sh` with no Opik or LLM keys set and expects a clean exit — it is the only execution signal a fork PR receives, so a working dry-run path is required, not optional. The standard pattern:
108112

109113
```python
@@ -136,6 +140,8 @@ Each example is a `uv` project: declare dependencies in its `pyproject.toml` (th
136140

137141
### run.sh
138142

143+
> Notebook examples don't use `run.sh` — they're executed by [`test-notebooks.yml`](.github/workflows/test-notebooks.yml). See [Notebook examples](#notebook-examples).
144+
139145
Every testable example must include a `run.sh` at its root. This file is what the CI matrix runs. Requirements:
140146

141147
- Start with `set -e` (fail fast on any error)
@@ -175,7 +181,18 @@ OPIK_PROJECT_NAME = os.environ.get("OPIK_PROJECT_NAME", "my-use-case")
175181
def my_function(): ...
176182
```
177183

178-
The compliance check accepts either pattern — it looks for `OPIK_PROJECT_NAME` in `run.sh` or in any `.py` file in the folder.
184+
The compliance check accepts any of these — it looks for `OPIK_PROJECT_NAME` exported in `run.sh`, or referenced in any `.py` or `.ipynb` file in the folder.
185+
186+
### Notebook examples
187+
188+
Some guides are Jupyter notebooks rather than runnable scripts. They follow a lighter contract:
189+
190+
- Ship `notebook.ipynb` + `pyproject.toml` + `README.md`. **No `run.sh`.**
191+
- Set `OPIK_PROJECT_NAME` in a notebook cell (e.g. `OPIK_PROJECT_NAME = "my-guide"`) and pass it via `@opik.track(project_name=...)`.
192+
- Commit with **outputs cleared** — no baked-in execution state (cleaner diffs, no stale/leaked run details).
193+
- Keep the install cell simple: `%pip install --quiet --upgrade opik` (works in Colab and locally). `pyproject.toml` stays the dependency source of truth and the hook for notebook CI.
194+
195+
A notebook teaches by logging real traces you watch render in Opik, so it needs credentials and has **no dry-run**. [`test-notebooks.yml`](.github/workflows/test-notebooks.yml) executes each changed notebook end-to-end with `ipython <notebook>.ipynb` (every cell runs; non-zero exit on the first error) — but only on same-repo PRs where Opik secrets are available. Fork PRs skip execution; the compliance check still runs. Because there's no secrets-free run, a notebook nobody edits isn't re-tested until it changes (a scheduled run for stable notebooks is a future addition).
179196

180197
### Opik workspace
181198

@@ -238,12 +255,12 @@ Before opening a PR, verify:
238255
- [ ] Folder name is lowercase with underscores (e.g. `my_example`, not `MyExample` or `my-example`)
239256
- [ ] `README.md` has all required sections
240257
- [ ] READMEs updated — the example's `README.md`, and for added/renamed/removed examples the bucket index and the root `README.md` table
241-
- [ ] Dry-run works with no credentials set — `bash run.sh` exits cleanly (this is exactly what CI's secrets-free job runs)
258+
- [ ] Dry-run works with no credentials set — `bash run.sh` exits cleanly (this is exactly what CI's secrets-free job runs) — *not applicable to notebook examples*
242259
- [ ] `uv run ruff check .` and `uv run ruff format --check .` are clean
243260
- [ ] No credentials or `.env` files committed
244261
- [ ] Dependencies declared in `pyproject.toml` (uv project); no `requirements.txt`
245-
- [ ] `run.sh` exists and starts with `set -e`
246-
- [ ] `OPIK_PROJECT_NAME` is set — exported in `run.sh` (scripts) or defined in `config.py` (use-cases/guides)
262+
- [ ] `run.sh` exists and starts with `set -e` — *or, for notebook examples, the folder ships a `.ipynb` (no `run.sh`) and notebook outputs are cleared*
263+
- [ ] `OPIK_PROJECT_NAME` is set — exported in `run.sh` (scripts), defined in `config.py` (use-cases/guides), or set in a notebook cell
247264
- [ ] Examples that call LLMs use litellm and read `OPIK_EXAMPLES_MODEL` in `config.py`
248265

249266
## Questions

‎README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@ Task-oriented examples for specific Opik workflows and patterns.
3232

3333
| | Description |
3434
|---|---|
35+
| [guides/annotation_queues_with_context](guides/annotation_queues_with_context/) | Structure RAG traces for Opik annotation queues — clean answer in output, context in metadata, full detail in child spans |
3536
| [guides/tracing_finetuned_models](guides/tracing_finetuned_models/) | Fine-tune a model, register it to the CometML Model Registry, then fetch and trace inference in Opik |
3637

3738
## Use Cases
@@ -41,6 +42,7 @@ End-to-end applications and domain-specific workflows.
4142
| | Description |
4243
|---|---|
4344
| [use-cases/call_summarizer](use-cases/call_summarizer/) | Streamlit app that summarises customer calls using an LLM, traced with Opik |
45+
| [use-cases/f1_radio_rag](use-cases/f1_radio_rag/) | Typer CLI walking the full Opik loop over F1 team-radio messages: RAG (ChromaDB) → eval dataset + test suite → optimization studio → prompt library |
4446
| [use-cases/governance_observability](use-cases/governance_observability/) | Instrument agents with governance metadata, derive composite metrics, and extract scores for oversight reporting |
4547

4648
## Scripts

‎guides/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ Task-oriented examples for doing something specific with Opik — combining Opik
44

55
| Guide | Description |
66
|---|---|
7+
| [annotation_queues_with_context/](./annotation_queues_with_context/) | Structure RAG traces for Opik annotation queues — clean answer in output, context in metadata, full detail in child spans |
78
| [tracing_finetuned_models/](./tracing_finetuned_models/) | Fine-tune a model, register it to the CometML Model Registry, then fetch and trace inference in Opik |
89

910
[Contribute one](../CONTRIBUTING.md).

‎guides/annotation_queues_with_context/README.md‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
# Structuring Traces for Annotation Queues
2+
3+
Shows how to structure a RAG pipeline's traces so they are immediately useful in Opik [annotation queues](https://www.comet.com/docs/opik/evaluation/advanced/annotation_queues) — a clean answer in `output`, supporting context in `metadata`, and full technical detail preserved in child spans.
4+
5+
## What this does
6+
7+
A trace gives you four distinct places to put data: `input`, `output`, `metadata`, and child `spans`. A common default is to return the whole pipeline dict — answer, retrieved documents, the built prompt — as the trace `output`, which buries the answer a reviewer needs to score. This example shows how to distribute the data instead:
8+
9+
- `input` — the user's question
10+
- `output` — the final answer only
11+
- `metadata` — retrieval context, set with `opik_context.update_current_trace()`
12+
- child `spans` — every sub-step decorated with `@opik.track`; full detail preserved
13+
14+
It also covers creating annotation queues programmatically and the post-hoc enrichment pattern for existing traces.
15+
16+
## Prerequisites
17+
18+
You need an Opik account to follow along — the value of this guide is watching the traces and the annotation queue render live in Opik.
19+
20+
| Variable | Description |
21+
|---|---|
22+
| `OPIK_API_KEY` | Opik API key |
23+
| `OPIK_WORKSPACE` | Opik workspace name |
24+
25+
No LLM API key required — the example uses a mock retriever and mock LLM.
26+
27+
## Running it
28+
29+
Open the notebook in Colab (badge below), or run it locally in a uv-managed environment:
30+
31+
```bash
32+
uv sync
33+
uv run --with jupyter jupyter lab
34+
```
35+
36+
Then open `annotation_queues_with_context.ipynb`.
37+
38+
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/comet-ml/opik-examples/blob/main/guides/annotation_queues_with_context/annotation_queues_with_context.ipynb)
39+
40+
## How it works
41+
42+
The notebook builds a small traced RAG pipeline and walks through three things:
43+
44+
1. **Structuring the trace.** `rag_pipeline()` calls `retrieve()` and `generate()` (each `@opik.track`, so they become child spans), returns only the answer as `output`, and attaches the retrieved context as `metadata` via `opik_context.update_current_trace()`. Input and output stay clean; the supporting detail is one layer down.
45+
2. **Creating a queue.** `client.create_traces_annotation_queue()` makes a review queue, `client.search_traces()` fetches the traces just logged, and `queue.add_traces()` adds them for review.
46+
3. **Post-hoc enrichment.** For traces already logged without context, `client.update_trace()` adds metadata after the fact; `client.flush()` commits the writes before the traces are added to a queue.

0 commit comments

Comments
 (0)