feat: add annotation queues with context guide

.gitignore

@@ -23,3 +23,6 @@ final_model/
 uv.lock
 .env
 poetry.lock
+
+# Local planning/process artifacts (superpowers brainstorm specs & implementation plans) — kept local, never committed
+docs/superpowers/

README.md

@@ -31,6 +31,7 @@ Task-oriented examples for specific Opik workflows and patterns.
 
 | | Description |
 |---|---|
+| [guides/annotation_queues_with_context](guides/annotation_queues_with_context/) | Structure RAG traces for Opik annotation queues — clean answer in output, context in metadata, full detail in child spans |
 | [guides/tracing_finetuned_models](guides/tracing_finetuned_models/) | Fine-tune a model, register it to the CometML Model Registry, then fetch and trace inference in Opik |
 
 ## Use Cases

guides/README.md

@@ -4,6 +4,7 @@ Task-oriented examples for doing something specific with Opik — combining Opik
 
 | Guide | Description |
 |---|---|
+| [annotation_queues_with_context/](./annotation_queues_with_context/) | Structure RAG traces for Opik annotation queues — clean answer in output, context in metadata, full detail in child spans |
 | [tracing_finetuned_models/](./tracing_finetuned_models/) | Fine-tune a model, register it to the CometML Model Registry, then fetch and trace inference in Opik |
 
 [Contribute one](../CONTRIBUTING.md).

guides/annotation_queues_with_context/README.md

@@ -0,0 +1,46 @@
+# Structuring Traces for Annotation Queues
+
+Shows how to structure a RAG pipeline's traces so they are immediately useful in Opik [annotation queues](https://www.comet.com/docs/opik/evaluation/advanced/annotation_queues) — a clean answer in `output`, supporting context in `metadata`, and full technical detail preserved in child spans.
+
+## What this does
+
+A trace gives you four distinct places to put data: `input`, `output`, `metadata`, and child `spans`. A common default is to return the whole pipeline dict — answer, retrieved documents, the built prompt — as the trace `output`, which buries the answer a reviewer needs to score. This example shows how to distribute the data instead:
+
+- `input` — the user's question
+- `output` — the final answer only
+- `metadata` — retrieval context, set with `opik_context.update_current_trace()`
+- child `spans` — every sub-step decorated with `@opik.track`; full detail preserved
+
+It also covers creating annotation queues programmatically and the post-hoc enrichment pattern for existing traces.
+
+## Prerequisites
+
+You need an Opik account to follow along — the value of this guide is watching the traces and the annotation queue render live in Opik.
+
+| Variable | Description |
+|---|---|
+| `OPIK_API_KEY` | Opik API key |
+| `OPIK_WORKSPACE` | Opik workspace name |
+
+No LLM API key required — the example uses a mock retriever and mock LLM.
+
+## Running it
+
+Open the notebook in Colab (badge below), or run it locally in a uv-managed environment:
+
+```bash
+uv sync
+uv run --with jupyter jupyter lab
+```
+
+Then open `annotation_queues_with_context.ipynb`.
+
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/comet-ml/opik-examples/blob/main/guides/annotation_queues_with_context/annotation_queues_with_context.ipynb)
+
+## How it works
+
+The notebook builds a small traced RAG pipeline and walks through three things:
+
+1. **Structuring the trace.** `rag_pipeline()` calls `retrieve()` and `generate()` (each `@opik.track`, so they become child spans), returns only the answer as `output`, and attaches the retrieved context as `metadata` via `opik_context.update_current_trace()`. Input and output stay clean; the supporting detail is one layer down.
+2. **Creating a queue.** `client.create_traces_annotation_queue()` makes a review queue, `client.search_traces()` fetches the traces just logged, and `queue.add_traces()` adds them for review.
+3. **Post-hoc enrichment.** For traces already logged without context, `client.update_trace()` adds metadata after the fact; `client.flush()` commits the writes before the traces are added to a queue.

guides/annotation_queues_with_context/annotation_queues_with_context.ipynb

@@ -0,0 +1,272 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "a1b2c3d4",
+   "metadata": {},
+   "source": [
+    "# Structuring Traces for Annotation Queues\n",
+    "\n",
+    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/comet-ml/opik-examples/blob/main/guides/annotation_queues_with_context/annotation_queues_with_context.ipynb)\n",
+    "\n",
+    "Opik's [annotation queues](https://www.comet.com/docs/opik/evaluation/advanced/annotation_queues) route traces to subject-matter experts (SMEs) for human review. Whether a reviewer can judge an answer quickly depends on how the trace is structured — and a trace gives you four distinct places to put data:\n",
+    "\n",
+    "| Field | What to put there |\n",
+    "|---|---|\n",
+    "| `input` | The user's question or request |\n",
+    "| `output` | The final answer only — not the full pipeline state |\n",
+    "| `metadata` | Supporting context a reviewer needs to verify quality (retrieved docs, sources) |\n",
+    "| child `spans` | Every sub-step (`@opik.track`), preserving full technical detail |\n",
+    "\n",
+    "A common default is to return the whole pipeline dict — answer, retrieved documents, the built prompt — as the trace `output`. That buries the answer a reviewer needs to score under pipeline internals. Keeping `output` to the user-facing answer and moving supporting context into `metadata` (with full detail in spans) keeps the review surface focused while losing nothing.\n",
+    "\n",
+    "**What you'll learn:**\n",
+    "\n",
+    "- How to structure a traced RAG pipeline so input and output stay clean for review\n",
+    "- How to surface retrieved context in metadata using `opik_context.update_current_trace()`\n",
+    "- How to create annotation queues programmatically and populate them with traces\n",
+    "- The post-hoc enrichment pattern for existing traces"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b2c3d4e5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%pip install --quiet --upgrade opik"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c3d4e5f6",
+   "metadata": {},
+   "outputs": [],
+   "source": "import opik\n\nOPIK_PROJECT_NAME = \"annotation-queues\"\n\n# Credentials come from OPIK_API_KEY / OPIK_WORKSPACE in the environment; targets Opik Cloud.\n# install_mcp=False keeps configure non-interactive (skips the MCP-server setup prompt) so it\n# runs unattended in CI and other headless environments.\nopik.configure(project_name=OPIK_PROJECT_NAME, install_mcp=False)"
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d4e5f6a7",
+   "metadata": {},
+   "source": [
+    "## 1. Building the traced RAG pipeline\n",
+    "\n",
+    "The key principle: **separate the user-facing answer from internal pipeline state.**\n",
+    "\n",
+    "- `output` — the final answer only, not the full pipeline dict\n",
+    "- `metadata` — retrieval context for reviewers, set with `opik_context.update_current_trace()`\n",
+    "- child spans — every sub-step decorated with `@opik.track`; full detail preserved without crowding the output\n",
+    "\n",
+    "Tag each sub-step with `type=` on `@opik.track` — `\"tool\"` for retrieval, `\"llm\"` for the model call — so its span renders with the right treatment in the Opik UI.\n",
+    "\n",
+    "The example uses a mock retriever and a mock LLM so no API keys are required beyond Opik. Replace `generate()` with your actual LLM call (e.g. `openai.chat.completions.create`)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e5f6a7b8",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "MOCK_CORPUS = {\n",
+    "    \"What is our leave policy?\": [\n",
+    "        {\"text\": \"Employees are entitled to 25 days of annual leave per year, plus public holidays.\", \"source\": \"HR Handbook v2.4, §3.1\"},\n",
+    "        {\"text\": \"Leave requests must be submitted at least 2 weeks in advance via the HR portal.\", \"source\": \"HR Handbook v2.4, §3.2\"},\n",
+    "    ],\n",
+    "    \"How do I request flexible working?\": [\n",
+    "        {\"text\": \"Flexible working requests should be submitted to your line manager using form FW-01.\", \"source\": \"Flexible Working Policy, §2\"},\n",
+    "        {\"text\": \"Requests will be reviewed within 28 days and approved or declined in writing.\", \"source\": \"Flexible Working Policy, §4\"},\n",
+    "    ],\n",
+    "    \"What are the performance review criteria?\": [\n",
+    "        {\"text\": \"Performance is assessed across four dimensions: delivery, collaboration, growth, and values.\", \"source\": \"Performance Framework 2024, §1\"},\n",
+    "        {\"text\": \"Annual reviews take place in Q1. Mid-year check-ins are mandatory for all staff.\", \"source\": \"Performance Framework 2024, §3\"},\n",
+    "    ],\n",
+    "}\n",
+    "\n",
+    "QUESTIONS = list(MOCK_CORPUS.keys())\n",
+    "\n",
+    "\n",
+    "@opik.track(type=\"tool\")\n",
+    "def retrieve(question: str) -> list[dict]:\n",
+    "    return MOCK_CORPUS.get(question, [{\"text\": \"No relevant documents found.\", \"source\": \"N/A\"}])\n",
+    "\n",
+    "\n",
+    "@opik.track(type=\"llm\")\n",
+    "def generate(question: str, context_docs: list[dict]) -> str:\n",
+    "    context_text = \" \".join(d[\"text\"] for d in context_docs)\n",
+    "    return f\"Based on company policy: {context_text[:120]}...\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f6a7b8c9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from opik import opik_context\n",
+    "\n",
+    "@opik.track(project_name=OPIK_PROJECT_NAME)\n",
+    "def rag_pipeline(question: str) -> str:\n",
+    "    context_docs = retrieve(question)\n",
+    "    answer = generate(question, context_docs)\n",
+    "\n",
+    "    # Supporting context goes in metadata so reviewers can verify the answer without it crowding the output\n",
+    "    opik_context.update_current_trace(\n",
+    "        metadata={\n",
+    "            \"retrieved_context\": [d[\"text\"] for d in context_docs],\n",
+    "            \"sources\": [d[\"source\"] for d in context_docs],\n",
+    "        }\n",
+    "    )\n",
+    "\n",
+    "    return answer\n",
+    "\n",
+    "\n",
+    "for question in QUESTIONS:\n",
+    "    rag_pipeline(question)\n",
+    "\n",
+    "opik.flush_tracker()\n",
+    "print(f\"Logged {len(QUESTIONS)} traces to project '{OPIK_PROJECT_NAME}'\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a7b8c9d0",
+   "metadata": {},
+   "source": [
+    "## 2. Creating an annotation queue\n",
+    "\n",
+    "With traces structured correctly, we can create a queue and populate it programmatically. Queues can also be created in the Opik UI — the SDK approach is useful for CI pipelines and batch review workflows.\n",
+    "\n",
+    "See the [annotation queues docs](https://www.comet.com/docs/opik/evaluation/advanced/annotation_queues) for queue management options including sharing queues with SMEs."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b8c9d0e1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "client = opik.Opik()\n",
+    "\n",
+    "queue = client.create_traces_annotation_queue(\n",
+    "    name=\"RAG Answer Quality Review\",\n",
+    "    instructions=(\n",
+    "        \"Score each answer for accuracy and relevance. \"\n",
+    "        \"Expand the Metadata section to see the retrieved context used to generate the answer.\"\n",
+    "    ),\n",
+    "    feedback_definition_names=[\"relevance\", \"accuracy\"],\n",
+    ")\n",
+    "\n",
+    "traces = client.search_traces(project_name=OPIK_PROJECT_NAME)\n",
+    "queue.add_traces(traces)\n",
+    "\n",
+    "print(f\"Queue '{queue.name}' created\")\n",
+    "print(f\"Added {len(traces)} traces for review\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c9d0e1f2",
+   "metadata": {},
+   "source": [
+    "## 3. Post-hoc enrichment\n",
+    "\n",
+    "If traces were logged without context in metadata — or if the context comes from a dataset row rather than the live pipeline — you can enrich traces after the fact using `client.update_trace()`.\n",
+    "\n",
+    "> **Why the cell below logs a `may cause data loss` warning:** Opik batches writes, and this demo updates traces that were created seconds earlier in the same run. Updating a trace before its creation has been committed can drop the update. In a real post-hoc workflow the traces come from an earlier session and are already committed, so the warning doesn't apply. The `client.flush()` at the end commits the enrichment writes before the traces are added to a queue. See [batching and updates](https://www.comet.com/docs/opik/tracing/batching_and_updates) for the recommended patterns."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d0e1f2a3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Simulate a dataset that has ground-truth context for each question\n",
+    "dataset_context = {\n",
+    "    \"What is our leave policy?\": \"Employees receive 25 days annual leave. Requests need 2 weeks notice.\",\n",
+    "    \"How do I request flexible working?\": \"Submit form FW-01 to your line manager. Decisions issued within 28 days.\",\n",
+    "    \"What are the performance review criteria?\": \"Four dimensions: delivery, collaboration, growth, and values.\",\n",
+    "}\n",
+    "\n",
+    "for trace in traces:\n",
+    "    question = trace.input.get(\"question\", \"\") if isinstance(trace.input, dict) else \"\"\n",
+    "    context = dataset_context.get(question)\n",
+    "    if context:\n",
+    "        client.update_trace(\n",
+    "            trace_id=trace.id,\n",
+    "            project_name=OPIK_PROJECT_NAME,\n",
+    "            metadata={\"dataset_context\": context},\n",
+    "        )\n",
+    "\n",
+    "client.flush()\n",
+    "print(\"Metadata enrichment complete\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e1f2a3b4",
+   "metadata": {},
+   "source": [
+    "## Summary\n",
+    "\n",
+    "Where data lives in a trace, and why:\n",
+    "\n",
+    "| Field | What to log | Why |\n",
+    "|---|---|---|\n",
+    "| `input` | The user's question or request | The reviewer needs to see what was asked |\n",
+    "| `output` | The final answer only | Keeps the review surface focused on what's being scored |\n",
+    "| `metadata` | Retrieval context, sources | Supporting detail a reviewer can check, without crowding the output |\n",
+    "| child `spans` | Every sub-step (`@opik.track`) | Full technical detail, preserved and inspectable |\n",
+    "\n",
+    "**Key takeaways:**\n",
+    "\n",
+    "1. Log only the final answer in `output` — never the full pipeline state.\n",
+    "2. Put retrieval context in `metadata` using `opik_context.update_current_trace(metadata=...)` — it stays out of the way but remains available to reviewers.\n",
+    "3. Decorate sub-steps with `@opik.track` — they become child spans, preserving full technical detail without crowding the output.\n",
+    "4. For existing traces, enrich with `client.update_trace()` and call `client.flush()` before adding to a queue."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f2a3b4c5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "try:\n",
+    "    queue.delete()\n",
+    "    print(\"Queue deleted\")\n",
+    "except Exception as e:\n",
+    "    print(f\"Cleanup error: {e}\")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.13"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

guides/annotation_queues_with_context/pyproject.toml

@@ -0,0 +1,25 @@
+[project]
+name = "annotation-queues-with-context"
+version = "0.1.0"
+description = "Structuring RAG traces for Opik annotation queues."
+readme = "README.md"
+requires-python = ">=3.12,<3.14"
+dependencies = [
+    "opik>=2.0.74",
+]
+
+[dependency-groups]
+dev = ["ruff"]
+
+# WHY: notebook-only example — uv manages the env, no installable package.
+[tool.uv]
+package = false
+
+[tool.ruff]
+line-length = 110
+target-version = "py312"
+# WHY: ruff can't parse the notebook's cell schema; lint .py files only.
+extend-exclude = ["*.ipynb"]
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "UP", "B"]