feat: add annotation queues with context guide - #28
Merged
Conversation
A guides-bucket notebook on structuring RAG traces for Opik annotation queues: clean answer in output, retrieval context in metadata via opik_context.update_current_trace(), and full detail preserved in child spans (type-tagged for UI rendering). Also covers creating a queue programmatically and the post-hoc enrichment pattern. Indexed in the root and guides README tables.
LeoRoccoBreedt
added a commit
that referenced
this pull request
Jul 1, 2026
## What this adds A dedicated CI path for **notebook examples**, so a `.ipynb` guide no longer needs a `run.sh` wrapper. ### `test-notebooks.yml` (new) - Detects example folders changed in the PR that contain a `.ipynb`. - On same-repo PRs (Opik secrets available), executes each changed notebook end-to-end with `uv run --with ipython --with nbformat ipython <notebook>.ipynb` — every cell runs, non-zero exit on the first error — wrapped in `nick-fields/retry@v3` (25-min timeout, 2 attempts). - **No dry-run job.** Notebooks log live traces and need credentials, so fork PRs (no secrets) skip execution; the compliance check still runs. ### `compliance.yml` - A folder containing a `.ipynb` is exempt from the `run.sh` requirement (it's executed by `test-notebooks.yml`). - `OPIK_PROJECT_NAME` is accepted in a `.py` **or `.ipynb`**; the litellm/`OPIK_EXAMPLES_MODEL` check is likewise extended. ### `CONTRIBUTING.md` - New **Notebook examples** section documenting the lighter contract (notebook + pyproject + README, no run.sh, outputs cleared, credential-gated, no dry-run), with carve-outs in the existing run.sh/dry-run rules and the PR checklist. ### Design notes - **Discovery is change-detection** (mirrors `pr-test.yml`), so notebooks are auto-discovered with zero maintenance. Known gap: a notebook nobody edits isn't re-tested until it changes — a curated **scheduled** run for stable notebooks is a deliberate follow-up (needs idempotent notebooks first). - `ipython <notebook>` execution follows neptune-ai/scale-examples' working pattern; trivially swappable to `jupyter execute` later. Unblocks the notebook guide in #28, which then needs no `run.sh`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
A new
guides/notebook — annotation_queues_with_context — showing how to structure RAG traces so they're immediately useful in Opik annotation queues.The lesson
A trace gives you four distinct places to put data —
input,output,metadata, and childspans. Returning the whole pipeline dict asoutputburies the answer a reviewer needs to score. The guide shows how to distribute it instead:output— the final answer onlymetadata— retrieval context, viaopik_context.update_current_trace()spans— every sub-step (@opik.track, type-taggedtool/llmfor UI rendering)It also covers creating an annotation queue programmatically (
create_traces_annotation_queue→search_traces→add_traces) and the post-hoc enrichment pattern (update_trace+flush).Notes
pyproject.toml(uv) +README.mdper repo convention. Mock retriever/LLM — no LLM API key required.guides/README tables.Follow-ups (not in this PR)
search_tracesso the "added N traces" count reflects only the current run.