Sync skills pack from opik-mcp - #27
github-actions[bot] wants to merge 1 commit into
Conversation
![baz-reviewer[bot]](https://avatars.githubusercontent.com/in/933528?s=60&v=4){kind=small}
| `result.initial_score` → `result.score` on the metric; `result.details["stop_reason"]` and `["trials_completed"]`; `result.llm_calls`, `result.llm_cost_total` (may be `None` when the provider returns no cost — say "cost unavailable", don't invent one). **Report the validation score**, not the training score. A gain within run-to-run noise (rerun the baseline once if in doubt) is "no measurable improvement" — say so rather than shipping a lateral move, and do **not** save a new version for it. The common cause of a flat result: the answers depend on context the prompt can't contain (retrieval, tools, account data) — then the prompt isn't the bottleneck and the next step is `/opik-explain` on the worst items, not more trials. | ||
|
|
||
| ### 7. Save the winner (library prompts) and hand off | ||
| ```python | ||
| messages = result.prompt.get_messages() # optimizer ChatPrompt -> raw messages | ||
| new_version = client.create_chat_prompt( | ||
| name="<name>", messages=messages, project_name="<project>", | ||
| change_description=f"opik-optimize: {result.optimizer}, {result.metric_name} {result.initial_score:.2f} -> {result.score:.2f} (validation)", | ||
| ) |
There was a problem hiding this comment.
Reports selection score as held-out improvement
The optimization flow ranks trials on validation_dataset but treats result.score as improvement and saves the winner, so the same data selects and evaluates the prompt and can produce an improved claim without an untouched test set. Should we require the third-set reread before marking or saving it, or treat this result as selection-only and require that reread for the final claim?
Want Baz to fix this for you? Activate Fixer
Other fix methods
Prompt for AI Agents
Before applying, verify this suggestion against the current code. In
`skills/opik-optimize/SKILL.md` around lines 87-95, revise the optimization
result-handling and winner-saving logic so `validation_dataset` is treated only as
selection data, not final evidence. Require a third untouched evaluation split and a
fresh `evaluate()`-style reread of the baseline and winner before reporting `improved`
or saving a new prompt version; otherwise report the result as selection-only/no final
improvement and do not save it.
| messages=[LlmAsJudgeMessageWrite(role="USER", content="<the validated judge prompt using {{input}} and {{output}}>")], | ||
| variables={"input": "input", "output": "output"}, # field paths from step 2 |
There was a problem hiding this comment.
Raw production data reaches judge
The activation flow sends raw production input/output into the judge prompt, while tracing anonymizers only affect trace ingestion, so matching traces can disclose user-controlled secrets or PII to the judge provider — should we add a sensitivity gate or redaction before evaluation?
Want Baz to fix this for you? Activate Fixer
Other fix methods
Prompt for AI Agents
Before applying, verify this suggestion against the current code. In
`skills/opik-online-eval/SKILL.md` around lines 68-69, the LLM-as-judge rule creation
logic maps raw production `input` and `output` directly into the judge prompt without
protecting secrets or PII. Update the activation and mapping guidance to require an
explicit sensitivity check, redact or exclude sensitive fields, and use sanitized trace
fields before creating any evaluator; block creation when a safe mapping cannot be
established. Apply this requirement consistently to trace, span, thread, and Python
metric variants, and clarify that ingestion anonymizers alone do not protect data sent
to the judge provider.
| scored = client.search_traces(project_name="<project>", max_results=1, | ||
| filter_string='feedback_scores.refund_window_correct is_not_empty') | ||
| if scored: break |
There was a problem hiding this comment.
Historical score falsely reports rule live
The verification query stops at the first trace with the target feedback-score name, so a pre-rule trace can classify the result as live even when no post-creation trace exists — should we capture the creation boundary or emitted trace ID and require the score on that trace?
Want Baz to fix this for you? Activate Fixer
Other fix methods
Prompt for AI Agents
Before applying, verify this suggestion against the current code. In
`skills/opik-online-eval/SKILL.md` around lines 85-87, fix the verification logic so it
cannot treat a pre-existing `feedback_scores.<name>` as evidence that the newly created
rule has scored a trace. Capture the rule-creation boundary or identify a trace created
after rule creation, then require the target score on that newly observed trace before
breaking and classifying the rule as `live`; otherwise retain the unverified or error
path.
| | no open issues, last scan `<time>` | Diagnostics is working and found nothing. Continue to step 3. | | ||
| | enabled but has not scanned yet, or last scan older than a day | `write('agent_insights_job.trigger', {"project_name": "<project>"})`, report the Diagnostics page link, continue to step 3. No need to ask: a scan changes no data. | | ||
| | not enabled for this project, or turned off | Ask the user once, in one sentence: "Enable daily Diagnostics for `<project>`? First results take a few minutes." On yes: `enable`, then `trigger`, report the link, continue. On no: continue and say the shortlist was built without Diagnostics. | | ||
| | not available on this deployment | Continue, and say once that this deployment has no Diagnostics. Do not offer to enable it. | |
There was a problem hiding this comment.
Diagnostics workflow points to wrong step
The empty-Diagnostics instruction and line 81’s go to step 3 for the gap both loop back to the Diagnostics section, so candidate traces never populate the shortlist — should we update both references to step 4?
Want Baz to fix this for you? Activate Fixer
Other fix methods
Prompt for AI Agents
Before applying, verify this suggestion against the current code. In
`skills/opik-diagnose/SKILL.md` around lines 65-68 and 81, update the empty
Diagnostics-list and coverage-gap instructions to reference step 4, not step 3. These
paths should proceed to candidate trace collection so they do not loop back into the
Diagnostics section.
| suite = client.get_test_suite(name="<suite>", project_name="<project>") | ||
| prior = client.get_test_suite_experiments(name="<suite>", project_name="<project>") # newest first is not guaranteed — sort by created_at yourself | ||
| ``` | ||
| Baseline = the most recent prior experiment on this suite, unless the user names one. **No prior experiment → this run *is* the baseline** (step 3 still runs; status `baseline_created`). Two explicit experiments → skip step 3, go to step 4. |
There was a problem hiding this comment.
Explicit comparisons lack suite context
The two-experiment branch skips suite resolution, so find_experiment_items_for_dataset(...) lacks suite.name/project_name and the compare URL lacks suite.id; it also doesn't verify that both experiments share a suite/project, so the documented path cannot reliably join or link the runs. Could we resolve both experiments, derive and validate their common suite/project, then build the join and URL?
Want Baz to fix this for you? Activate Fixer
Other fix methods
Prompt for AI Agents
Before applying, verify this suggestion against the current code. In
`skills/opik-compare/SKILL.md` around lines 41-41, fix the two-existing-experiments
branch so it does not skip the metadata resolution required by the later item join and
compare-link logic. Resolve both experiments, derive and validate their common suite and
project, and use that context for `find_experiment_items_for_dataset(...)` and the
compare URL; stop with a clear blocker if the experiments belong to different suites or
projects.
| The suite's items say what to call: each item `description` written by `/opik-test` ends in `Entrypoint: <root span name>`. Grep the repo for that function, import it, and wrap it: | ||
|
|
||
| ```python | ||
| def task(item: dict) -> dict: | ||
| return {"input": item["input"], "output": str(entrypoint(item["input"]))} |
There was a problem hiding this comment.
Evaluation can trigger irreversible production side effects
The runner invokes the suite/user-selected entrypoint directly, so /opik-compare can execute credentialed mail-sending, data-writing, or purchasing code — should we require an independently verified test/dev target and an explicit mutation boundary before invocation?
Want Baz to fix this for you? Activate Fixer
Other fix methods
Prompt for AI Agents
Before applying, verify this suggestion against the current code. In
`skills/opik-compare/SKILL.md` around lines 46-50, the task adapter trusts the
suite-selected `Entrypoint` and invokes it directly, allowing credentialed production or
side-effecting functions to run. Refactor this workflow to require independent
verification that the entrypoint targets an isolated test/dev implementation, reject
production or unverified targets, and require an explicit mutation boundary confirming
that the runner cannot send, write, purchase, or otherwise mutate external state before
invoking any item.
| client.rest_client.experiments.execute_experiment( | ||
| dataset_name=suite.name, dataset_id=suite.id, | ||
| prompts=[{"model": "<model>", "messages": [...], "configs": {}, "prompt_versions": [{"id": "<prompt_version_id>"}]}], |
There was a problem hiding this comment.
Prompt candidates cannot run on suites
execute_experiment receives the TestSuite name/id as dataset fields even though it is dataset-scoped, so prompt candidates either fail dataset resolution or skip the suite’s pinned assertions — should we materialize the suite as the required Dataset with its pinned version, or use an API that explicitly accepts a TestSuite?
Want Baz to fix this for you? Activate Fixer
Other fix methods
Prompt for AI Agents
Before applying, verify this suggestion against the current code. In
`skills/opik-compare/SKILL.md` around lines 56-58, fix the prompt-candidate
`execute_experiment` logic because this dataset-scoped API is being given TestSuite
fields directly. Refactor it to resolve or materialize the suite as the required
Dataset, including the pinned suite version and its assertions, or use an API that
explicitly accepts a TestSuite; ensure prompt candidates remain evaluated against the
same versioned suite rather than bypassing its assertions.
| Same suite, same version, same judge model, same runs-per-item as the baseline — vary **only** the thing under test. | ||
| ```python | ||
| result = opik.run_tests( | ||
| test_suite=suite, # or suite.get_version_view("<baseline's version>") to pin |
There was a problem hiding this comment.
Compares different suite versions
The candidate run uses mutable current suite while the baseline may be an older snapshot, so suite changes can compare different items/assertions and report invalid regressions or deltas. Should we resolve and verify the baseline version_id/version_name, then pass suite.get_version_view(version_name) before reporting, since documented experiment-item responses expose no dataset_version?
Want Baz to fix this for you? Activate Fixer
Other fix methods
Prompt for AI Agents
Before applying, verify this suggestion against the current code. In
`skills/opik-compare/SKILL.md` around lines 67-67, update the candidate-run logic so it
does not execute against the mutable current `suite` when comparing with an older
baseline. Resolve the baseline’s suite version and pass
`suite.get_version_view(version_name)` (or an equivalent verified version ID) to
`opik.run_tests`; for two existing experiments, verify their version identifiers before
comparing. If version metadata is unavailable or mismatched, stop with a blocker instead
of reporting regressions or deltas, and document the same version check in the
comparability step.
User description
Generated by the scheduled sync. The skills in this repository are built from
comet-ml/opik-mcpundersrc/opik_mcp/skills/— review the diff and merge to publish.Content digest:
ff539046c81f706f71b3c3eab6c6929e3c8715e67d9a37d85e3cae40e102f906Do not hand-edit this branch; edit the skills upstream instead.
Generated description
Below is a concise technical summary of the changes proposed in this PR:
Sync the skills pack from
comet-ml/opik-mcpto version0.2.36, updating the source commit and content digest. Add new evaluation workflows for comparing runs, testing regressions, online evaluation, and prompt optimization, while enhancing diagnosis, evaluation, tracing, and project observability guidance.opik-compare,opik-test,opik-online-eval, andopik-optimizeto support regression capture, candidate-versus-baseline analysis, production scoring, and prompt improvement workflows.Modified files (6)
Latest Contributors(2)
opik-diagnoseandopik-evaluatewith richer project context, diagnostics handling, grounded scoring workflows, and clearer result reporting; correct project-aware span lookup in tracing and explanation guidance.Modified files (7)
Latest Contributors(2)
Customize your next review