Introduce the optunahub.benchmarks module
#73
Conversation
y0z
added
the
feature
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #73 +/- ##
==========================================
- Coverage 88.80% 88.67% -0.13%
==========================================
Files 6 8 +2
Lines 134 159 +25
==========================================
+ Hits 119 141 +22
- Misses 15 18 +3 ☔ View full report in Codecov by Sentry. |
nabenabe0928
left a comment
nabenabe0928
left a comment
There was a problem hiding this comment.
Thank you for the PR, I left some comments!
| trial._check_distribution(name, dist) | ||
| return self.evaluate(params) | ||
|
|
||
| def evaluate(self, params: dict[str, Any]) -> float | Sequence[float]: |
There was a problem hiding this comment.
Is there any reason why this method is not an abstract method?
There was a problem hiding this comment.
If you only want to support Optuna (e.g., implementing define-by-run objective), it is acceptable to override __call__ and not implement evaluate and search_space.
| raise NotImplementedError | ||
|
|
||
| @property | ||
| def search_space(self) -> dict[str, optuna.distributions.BaseDistribution]: |
There was a problem hiding this comment.
Is there any reason why this method is not an abstract method?
Co-authored-by: Shuhei Watanabe <[email protected]>
Co-authored-by: Shuhei Watanabe <[email protected]>
Co-authored-by: Shuhei Watanabe <[email protected]>
nabenabe0928
left a comment
nabenabe0928
left a comment
There was a problem hiding this comment.
Thank you for the update, LGTM!
3 participants
Motivation & Description of the changes
optunahub.benchmarksmodule as a new feature for benchmarking.