From 35d7d09ed27b0b5399b8a890e0d46b4a7553e2b2 Mon Sep 17 00:00:00 2001 From: LeoRoccoBreedt Date: Tue, 30 Jun 2026 14:34:15 +0200 Subject: [PATCH 1/4] feat: add annotation queues with context guide A guides-bucket notebook on structuring RAG traces for Opik annotation queues: clean answer in output, retrieval context in metadata via opik_context.update_current_trace(), and full detail preserved in child spans (type-tagged for UI rendering). Also covers creating a queue programmatically and the post-hoc enrichment pattern. Indexed in the root and guides README tables. --- README.md | 1 + guides/README.md | 1 + .../annotation_queues_with_context/README.md | 46 +++ .../annotation_queues_with_context.ipynb | 279 ++++++++++++++++++ .../pyproject.toml | 25 ++ 5 files changed, 352 insertions(+) create mode 100644 guides/annotation_queues_with_context/README.md create mode 100644 guides/annotation_queues_with_context/annotation_queues_with_context.ipynb create mode 100644 guides/annotation_queues_with_context/pyproject.toml diff --git a/README.md b/README.md index 268da64..6d0bfd0 100644 --- a/README.md +++ b/README.md @@ -28,6 +28,7 @@ Task-oriented examples for specific Opik workflows and patterns. | | Description | |---|---| +| [guides/annotation_queues_with_context](guides/annotation_queues_with_context/) | Structure RAG traces for Opik annotation queues — clean SME view with retrieval context in metadata | | [guides/tracing_finetuned_models](guides/tracing_finetuned_models/) | Fine-tune a model, register it to the CometML Model Registry, then fetch and trace inference in Opik | ## Use Cases diff --git a/guides/README.md b/guides/README.md index eb7a70e..ea2bd02 100644 --- a/guides/README.md +++ b/guides/README.md @@ -4,6 +4,7 @@ Task-oriented examples for doing something specific with Opik — combining Opik | Guide | Description | |---|---| +| [annotation_queues_with_context/](./annotation_queues_with_context/) | Structure RAG traces for Opik annotation queues — clean SME view with retrieval context in metadata | | [tracing_finetuned_models/](./tracing_finetuned_models/) | Fine-tune a model, register it to the CometML Model Registry, then fetch and trace inference in Opik | [Contribute one](../CONTRIBUTING.md). diff --git a/guides/annotation_queues_with_context/README.md b/guides/annotation_queues_with_context/README.md new file mode 100644 index 0000000..fa6ce5e --- /dev/null +++ b/guides/annotation_queues_with_context/README.md @@ -0,0 +1,46 @@ +# Structuring Traces for Annotation Queues + +Shows how to structure a RAG pipeline's traces so they are immediately useful in Opik [annotation queues](https://www.comet.com/docs/opik/evaluation/advanced/annotation_queues) — a clean answer in `output`, supporting context in `metadata`, and full technical detail preserved in child spans. + +## What this does + +A trace gives you four distinct places to put data: `input`, `output`, `metadata`, and child `spans`. A common default is to return the whole pipeline dict — answer, retrieved documents, the built prompt — as the trace `output`, which buries the answer a reviewer needs to score. This example shows how to distribute the data instead: + +- `input` — the user's question +- `output` — the final answer only +- `metadata` — retrieval context, set with `opik_context.update_current_trace()` +- child `spans` — every sub-step decorated with `@opik.track`; full detail preserved + +It also covers creating annotation queues programmatically and the post-hoc enrichment pattern for existing traces. + +## Prerequisites + +You need an Opik account to follow along — the value of this guide is watching the traces and the annotation queue render live in Opik. + +| Variable | Description | +|---|---| +| `OPIK_API_KEY` | Opik API key | +| `OPIK_WORKSPACE` | Opik workspace name | + +No LLM API key required — the example uses a mock retriever and mock LLM. + +## Running it + +Open the notebook in Colab (badge below), or run it locally in a uv-managed environment: + +```bash +uv sync +uv run --with jupyter jupyter lab +``` + +Then open `annotation_queues_with_context.ipynb`. + +[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/comet-ml/opik-examples/blob/main/guides/annotation_queues_with_context/annotation_queues_with_context.ipynb) + +## How it works + +The notebook builds a small traced RAG pipeline and walks through three things: + +1. **Structuring the trace.** `rag_pipeline()` calls `retrieve()` and `generate()` (each `@opik.track`, so they become child spans), returns only the answer as `output`, and attaches the retrieved context as `metadata` via `opik_context.update_current_trace()`. Input and output stay clean; the supporting detail is one layer down. +2. **Creating a queue.** `client.create_traces_annotation_queue()` makes a review queue, `client.search_traces()` fetches the traces just logged, and `queue.add_traces()` adds them for review. +3. **Post-hoc enrichment.** For traces already logged without context, `client.update_trace()` adds metadata after the fact; `client.flush()` commits the writes before the traces are added to a queue. diff --git a/guides/annotation_queues_with_context/annotation_queues_with_context.ipynb b/guides/annotation_queues_with_context/annotation_queues_with_context.ipynb new file mode 100644 index 0000000..9fdada9 --- /dev/null +++ b/guides/annotation_queues_with_context/annotation_queues_with_context.ipynb @@ -0,0 +1,279 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "a1b2c3d4", + "metadata": {}, + "source": [ + "# Structuring Traces for Annotation Queues\n", + "\n", + "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/comet-ml/opik-examples/blob/main/guides/annotation_queues_with_context/annotation_queues_with_context.ipynb)\n", + "\n", + "Opik's [annotation queues](https://www.comet.com/docs/opik/evaluation/advanced/annotation_queues) route traces to subject-matter experts (SMEs) for human review. Whether a reviewer can judge an answer quickly depends on how the trace is structured — and a trace gives you four distinct places to put data:\n", + "\n", + "| Field | What to put there |\n", + "|---|---|\n", + "| `input` | The user's question or request |\n", + "| `output` | The final answer only — not the full pipeline state |\n", + "| `metadata` | Supporting context a reviewer needs to verify quality (retrieved docs, sources) |\n", + "| child `spans` | Every sub-step (`@opik.track`), preserving full technical detail |\n", + "\n", + "A common default is to return the whole pipeline dict — answer, retrieved documents, the built prompt — as the trace `output`. That buries the answer a reviewer needs to score under pipeline internals. Keeping `output` to the user-facing answer and moving supporting context into `metadata` (with full detail in spans) keeps the review surface focused while losing nothing.\n", + "\n", + "**What you'll learn:**\n", + "\n", + "- How to structure a traced RAG pipeline so input and output stay clean for review\n", + "- How to surface retrieved context in metadata using `opik_context.update_current_trace()`\n", + "- How to create annotation queues programmatically and populate them with traces\n", + "- The post-hoc enrichment pattern for existing traces" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b2c3d4e5", + "metadata": {}, + "outputs": [], + "source": [ + "%pip install --quiet --upgrade opik" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c3d4e5f6", + "metadata": {}, + "outputs": [], + "source": [ + "import opik\n", + "\n", + "OPIK_PROJECT_NAME = \"annotation-queues\"\n", + "\n", + "# Reads OPIK_API_KEY and OPIK_WORKSPACE from the environment (or prompts); targets Opik Cloud\n", + "opik.configure(project_name=OPIK_PROJECT_NAME)" + ] + }, + { + "cell_type": "markdown", + "id": "d4e5f6a7", + "metadata": {}, + "source": [ + "## 1. Building the traced RAG pipeline\n", + "\n", + "The key principle: **separate the user-facing answer from internal pipeline state.**\n", + "\n", + "- `output` — the final answer only, not the full pipeline dict\n", + "- `metadata` — retrieval context for reviewers, set with `opik_context.update_current_trace()`\n", + "- child spans — every sub-step decorated with `@opik.track`; full detail preserved without crowding the output\n", + "\n", + "Tag each sub-step with `type=` on `@opik.track` — `\"tool\"` for retrieval, `\"llm\"` for the model call — so its span renders with the right treatment in the Opik UI.\n", + "\n", + "The example uses a mock retriever and a mock LLM so no API keys are required beyond Opik. Replace `generate()` with your actual LLM call (e.g. `openai.chat.completions.create`)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e5f6a7b8", + "metadata": {}, + "outputs": [], + "source": [ + "MOCK_CORPUS = {\n", + " \"What is our leave policy?\": [\n", + " {\"text\": \"Employees are entitled to 25 days of annual leave per year, plus public holidays.\", \"source\": \"HR Handbook v2.4, §3.1\"},\n", + " {\"text\": \"Leave requests must be submitted at least 2 weeks in advance via the HR portal.\", \"source\": \"HR Handbook v2.4, §3.2\"},\n", + " ],\n", + " \"How do I request flexible working?\": [\n", + " {\"text\": \"Flexible working requests should be submitted to your line manager using form FW-01.\", \"source\": \"Flexible Working Policy, §2\"},\n", + " {\"text\": \"Requests will be reviewed within 28 days and approved or declined in writing.\", \"source\": \"Flexible Working Policy, §4\"},\n", + " ],\n", + " \"What are the performance review criteria?\": [\n", + " {\"text\": \"Performance is assessed across four dimensions: delivery, collaboration, growth, and values.\", \"source\": \"Performance Framework 2024, §1\"},\n", + " {\"text\": \"Annual reviews take place in Q1. Mid-year check-ins are mandatory for all staff.\", \"source\": \"Performance Framework 2024, §3\"},\n", + " ],\n", + "}\n", + "\n", + "QUESTIONS = list(MOCK_CORPUS.keys())\n", + "\n", + "\n", + "@opik.track(type=\"tool\")\n", + "def retrieve(question: str) -> list[dict]:\n", + " return MOCK_CORPUS.get(question, [{\"text\": \"No relevant documents found.\", \"source\": \"N/A\"}])\n", + "\n", + "\n", + "@opik.track(type=\"llm\")\n", + "def generate(question: str, context_docs: list[dict]) -> str:\n", + " context_text = \" \".join(d[\"text\"] for d in context_docs)\n", + " return f\"Based on company policy: {context_text[:120]}...\"" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f6a7b8c9", + "metadata": {}, + "outputs": [], + "source": [ + "from opik import opik_context\n", + "\n", + "@opik.track(project_name=OPIK_PROJECT_NAME)\n", + "def rag_pipeline(question: str) -> str:\n", + " context_docs = retrieve(question)\n", + " answer = generate(question, context_docs)\n", + "\n", + " # Supporting context goes in metadata so reviewers can verify the answer without it crowding the output\n", + " opik_context.update_current_trace(\n", + " metadata={\n", + " \"retrieved_context\": [d[\"text\"] for d in context_docs],\n", + " \"sources\": [d[\"source\"] for d in context_docs],\n", + " }\n", + " )\n", + "\n", + " return answer\n", + "\n", + "\n", + "for question in QUESTIONS:\n", + " rag_pipeline(question)\n", + "\n", + "opik.flush_tracker()\n", + "print(f\"Logged {len(QUESTIONS)} traces to project '{OPIK_PROJECT_NAME}'\")" + ] + }, + { + "cell_type": "markdown", + "id": "a7b8c9d0", + "metadata": {}, + "source": [ + "## 2. Creating an annotation queue\n", + "\n", + "With traces structured correctly, we can create a queue and populate it programmatically. Queues can also be created in the Opik UI — the SDK approach is useful for CI pipelines and batch review workflows.\n", + "\n", + "See the [annotation queues docs](https://www.comet.com/docs/opik/evaluation/advanced/annotation_queues) for queue management options including sharing queues with SMEs." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b8c9d0e1", + "metadata": {}, + "outputs": [], + "source": [ + "client = opik.Opik()\n", + "\n", + "queue = client.create_traces_annotation_queue(\n", + " name=\"RAG Answer Quality Review\",\n", + " instructions=(\n", + " \"Score each answer for accuracy and relevance. \"\n", + " \"Expand the Metadata section to see the retrieved context used to generate the answer.\"\n", + " ),\n", + " feedback_definition_names=[\"relevance\", \"accuracy\"],\n", + ")\n", + "\n", + "traces = client.search_traces(project_name=OPIK_PROJECT_NAME)\n", + "queue.add_traces(traces)\n", + "\n", + "print(f\"Queue '{queue.name}' created\")\n", + "print(f\"Added {len(traces)} traces for review\")" + ] + }, + { + "cell_type": "markdown", + "id": "c9d0e1f2", + "metadata": {}, + "source": [ + "## 3. Post-hoc enrichment\n", + "\n", + "If traces were logged without context in metadata — or if the context comes from a dataset row rather than the live pipeline — you can enrich traces after the fact using `client.update_trace()`.\n", + "\n", + "> **Why the cell below logs a `may cause data loss` warning:** Opik batches writes, and this demo updates traces that were created seconds earlier in the same run. Updating a trace before its creation has been committed can drop the update. In a real post-hoc workflow the traces come from an earlier session and are already committed, so the warning doesn't apply. The `client.flush()` at the end commits the enrichment writes before the traces are added to a queue. See [batching and updates](https://www.comet.com/docs/opik/tracing/batching_and_updates) for the recommended patterns." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d0e1f2a3", + "metadata": {}, + "outputs": [], + "source": [ + "# Simulate a dataset that has ground-truth context for each question\n", + "dataset_context = {\n", + " \"What is our leave policy?\": \"Employees receive 25 days annual leave. Requests need 2 weeks notice.\",\n", + " \"How do I request flexible working?\": \"Submit form FW-01 to your line manager. Decisions issued within 28 days.\",\n", + " \"What are the performance review criteria?\": \"Four dimensions: delivery, collaboration, growth, and values.\",\n", + "}\n", + "\n", + "for trace in traces:\n", + " question = trace.input.get(\"question\", \"\") if isinstance(trace.input, dict) else \"\"\n", + " context = dataset_context.get(question)\n", + " if context:\n", + " client.update_trace(\n", + " trace_id=trace.id,\n", + " project_name=OPIK_PROJECT_NAME,\n", + " metadata={\"dataset_context\": context},\n", + " )\n", + "\n", + "client.flush()\n", + "print(\"Metadata enrichment complete\")" + ] + }, + { + "cell_type": "markdown", + "id": "e1f2a3b4", + "metadata": {}, + "source": [ + "## Summary\n", + "\n", + "Where data lives in a trace, and why:\n", + "\n", + "| Field | What to log | Why |\n", + "|---|---|---|\n", + "| `input` | The user's question or request | The reviewer needs to see what was asked |\n", + "| `output` | The final answer only | Keeps the review surface focused on what's being scored |\n", + "| `metadata` | Retrieval context, sources | Supporting detail a reviewer can check, without crowding the output |\n", + "| child `spans` | Every sub-step (`@opik.track`) | Full technical detail, preserved and inspectable |\n", + "\n", + "**Key takeaways:**\n", + "\n", + "1. Log only the final answer in `output` — never the full pipeline state.\n", + "2. Put retrieval context in `metadata` using `opik_context.update_current_trace(metadata=...)` — it stays out of the way but remains available to reviewers.\n", + "3. Decorate sub-steps with `@opik.track` — they become child spans, preserving full technical detail without crowding the output.\n", + "4. For existing traces, enrich with `client.update_trace()` and call `client.flush()` before adding to a queue." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f2a3b4c5", + "metadata": {}, + "outputs": [], + "source": [ + "try:\n", + " queue.delete()\n", + " print(\"Queue deleted\")\n", + "except Exception as e:\n", + " print(f\"Cleanup error: {e}\")" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.13" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/guides/annotation_queues_with_context/pyproject.toml b/guides/annotation_queues_with_context/pyproject.toml new file mode 100644 index 0000000..9b54118 --- /dev/null +++ b/guides/annotation_queues_with_context/pyproject.toml @@ -0,0 +1,25 @@ +[project] +name = "annotation-queues-with-context" +version = "0.1.0" +description = "Structuring RAG traces for Opik annotation queues." +readme = "README.md" +requires-python = ">=3.12,<3.14" +dependencies = [ + "opik>=2.0.74", +] + +[dependency-groups] +dev = ["ruff"] + +# WHY: notebook-only example — uv manages the env, no installable package. +[tool.uv] +package = false + +[tool.ruff] +line-length = 110 +target-version = "py312" +# WHY: ruff can't parse the notebook's cell schema; lint .py files only. +extend-exclude = ["*.ipynb"] + +[tool.ruff.lint] +select = ["E", "F", "I", "UP", "B"] From f5b6da3714c424885d52a36137058023ce96469a Mon Sep 17 00:00:00 2001 From: LeoRoccoBreedt Date: Tue, 30 Jun 2026 14:44:30 +0200 Subject: [PATCH 2/4] chore: gitignore local superpowers planning artifacts --- .gitignore | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.gitignore b/.gitignore index 305e854..d260914 100644 --- a/.gitignore +++ b/.gitignore @@ -23,3 +23,6 @@ final_model/ uv.lock .env poetry.lock + +# Local planning/process artifacts (superpowers brainstorm specs & implementation plans) — kept local, never committed +docs/superpowers/ From f9ba5a10bc2aad08dabcc8d897f994f69c3b01cb Mon Sep 17 00:00:00 2001 From: LeoRoccoBreedt Date: Wed, 1 Jul 2026 11:44:25 +0200 Subject: [PATCH 3/4] docs: align annotation-queues index blurbs with principle-first framing --- README.md | 2 +- guides/README.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 1424b83..f92af2e 100644 --- a/README.md +++ b/README.md @@ -31,7 +31,7 @@ Task-oriented examples for specific Opik workflows and patterns. | | Description | |---|---| -| [guides/annotation_queues_with_context](guides/annotation_queues_with_context/) | Structure RAG traces for Opik annotation queues — clean SME view with retrieval context in metadata | +| [guides/annotation_queues_with_context](guides/annotation_queues_with_context/) | Structure RAG traces for Opik annotation queues — clean answer in output, context in metadata, full detail in child spans | | [guides/tracing_finetuned_models](guides/tracing_finetuned_models/) | Fine-tune a model, register it to the CometML Model Registry, then fetch and trace inference in Opik | ## Use Cases diff --git a/guides/README.md b/guides/README.md index ea2bd02..6657051 100644 --- a/guides/README.md +++ b/guides/README.md @@ -4,7 +4,7 @@ Task-oriented examples for doing something specific with Opik — combining Opik | Guide | Description | |---|---| -| [annotation_queues_with_context/](./annotation_queues_with_context/) | Structure RAG traces for Opik annotation queues — clean SME view with retrieval context in metadata | +| [annotation_queues_with_context/](./annotation_queues_with_context/) | Structure RAG traces for Opik annotation queues — clean answer in output, context in metadata, full detail in child spans | | [tracing_finetuned_models/](./tracing_finetuned_models/) | Fine-tune a model, register it to the CometML Model Registry, then fetch and trace inference in Opik | [Contribute one](../CONTRIBUTING.md). From 5151dc6840c942e634545f740e2f53973a6b5cf8 Mon Sep 17 00:00:00 2001 From: LeoRoccoBreedt Date: Wed, 1 Jul 2026 11:48:13 +0200 Subject: [PATCH 4/4] fix: skip MCP setup prompt in opik.configure so the notebook runs headless in CI --- .../annotation_queues_with_context.ipynb | 11 ++--------- 1 file changed, 2 insertions(+), 9 deletions(-) diff --git a/guides/annotation_queues_with_context/annotation_queues_with_context.ipynb b/guides/annotation_queues_with_context/annotation_queues_with_context.ipynb index 9fdada9..2226dd8 100644 --- a/guides/annotation_queues_with_context/annotation_queues_with_context.ipynb +++ b/guides/annotation_queues_with_context/annotation_queues_with_context.ipynb @@ -44,14 +44,7 @@ "id": "c3d4e5f6", "metadata": {}, "outputs": [], - "source": [ - "import opik\n", - "\n", - "OPIK_PROJECT_NAME = \"annotation-queues\"\n", - "\n", - "# Reads OPIK_API_KEY and OPIK_WORKSPACE from the environment (or prompts); targets Opik Cloud\n", - "opik.configure(project_name=OPIK_PROJECT_NAME)" - ] + "source": "import opik\n\nOPIK_PROJECT_NAME = \"annotation-queues\"\n\n# Credentials come from OPIK_API_KEY / OPIK_WORKSPACE in the environment; targets Opik Cloud.\n# install_mcp=False keeps configure non-interactive (skips the MCP-server setup prompt) so it\n# runs unattended in CI and other headless environments.\nopik.configure(project_name=OPIK_PROJECT_NAME, install_mcp=False)" }, { "cell_type": "markdown", @@ -276,4 +269,4 @@ }, "nbformat": 4, "nbformat_minor": 5 -} +} \ No newline at end of file