[NA] [SDK] chore: better exception for evaluate_experiment with no te…#5855
Open
kriogenia wants to merge 1 commit intocomet-ml:mainfrom
Open
Conversation
![baz-reviewer[bot]](https://avatars.githubusercontent.com/in/933528?s=60&v=4){kind=small}
Comment on lines
578
to
582
| """Update the existing experiment with new evaluation metrics. You can use either `scoring_metrics` or `scorer_functions` to calculate | ||
| evaluation metrics. The scorer functions doesn't require `scoring_key_mapping` and use reserved parameters | ||
| to receive inputs and outputs from the task. | ||
| to receive inputs and outputs from the task. The experiment requires at least one test case. | ||
|
|
||
| Args: |
Contributor
There was a problem hiding this comment.
Docstring doesn't mention that exceptions.EmptyExperiment is raised when test_cases is empty — should we document this in the Raises section?
Finding type: Document behavior and context | Severity: 🟢 Low
Want Baz to fix this for you? Activate Fixer
Other fix methods
Prompt for AI Agents:
Before applying, verify this suggestion against the current code. In
sdks/python/src/opik/evaluation/evaluator.py around lines 578 to 582, the
evaluate_experiment docstring mentions that the experiment requires at least one test
case but does not document the new runtime behavior where the function raises
exceptions.EmptyExperiment if there are no test cases. Update the docstring by adding a
concise 'Raises' subsection (or one-line sentence in the existing docstring) that
states: exceptions.EmptyExperiment is raised when the experiment has no test traces/test
cases to evaluate. Keep the wording consistent with the existing Args style and place it
near the top of the docstring so callers know to catch or propagate this exception.
kriogenia
force-pushed
the
NA-meaningful-error-for-evaluate-experiment-with-no-test-case
branch
from
86aff04 to
2341fbe
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Details
Adds a custom and meaningful error to return when
evaluate_experimentis calledwith an experiment with no test cases. Previously the error was an index out of bounds.
Change checklist
Issues
AI-WATERMARK
AI-WATERMARK: [no]
Testing
Documentation
evaluate_experimentdocstring to signal the test case requirement.