-
Notifications
You must be signed in to change notification settings - Fork 14
Add tutorial in docs about how to use benchmarks #110
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 3 commits
c15c661
7a45bc5
1ff9c29
c06582b
2d2f42a
5451c50
0dac627
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,128 @@ | ||
| How to Use Benchmarks | ||
| ===================== | ||
|
|
||
| In this tutorial, we will explain how to use benchmarks in optunahub. | ||
|
|
||
| Preparation | ||
| ----------- | ||
|
|
||
| First, ensure the necessary packages are installed by executing the following command: | ||
|
|
||
| .. code-block:: console | ||
|
|
||
| $ pip install optuna optunahub | ||
|
|
||
| Examples | ||
| -------- | ||
|
|
||
| We will use the `black-box optimization benchmarking (bbob) test suite <https://hub.optuna.org/benchmarks/bbob/>`__ in this tutorial. | ||
| This is a wrapper of `COCO (COmparing Continuous Optimizers) experiment <https://github.com/numbbo/coco-experiment>`__ library. | ||
| So you need to install COCO first. | ||
|
|
||
| .. code-block:: console | ||
|
|
||
| $ pip install coco-experiment | ||
|
|
||
|
|
||
| Test code is as follows: | ||
|
|
||
| .. code-block:: python | ||
|
|
||
| import optuna | ||
| import optunahub | ||
|
|
||
|
|
||
| bbob = optunahub.load_module("benchmarks/bbob") | ||
| sphere2d = bbob.Problem(function_id=1, dimension=2, instance_id=1) | ||
|
|
||
| study = optuna.create_study(directions=sphere2d.directions, sampler=optuna.samplers.TPESampler(seed=42)) | ||
| study.optimize(sphere2d, n_trials=100) | ||
|
|
||
| optuna.visualization.plot_optimization_history(study).show() | ||
|
|
||
| .. figure:: ./images/optimization_history.png | ||
| :alt: Optimization History | ||
| :align: center | ||
| :width: 800px | ||
|
|
||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Please add an example to use OptunaHub Benchmarks with optimizers other than Optuna, like https://medium.com/optuna/optunahub-benchmarks-a-new-feature-to-use-register-various-benchmark-problems-694401524ce0.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The doc of the |
||
| Constrained Problem | ||
| ^^^^^^^^^^^^^^^^^^^ | ||
|
|
||
| There are also some benchmarks that have constraints. | ||
|
|
||
| .. code-block:: python | ||
|
|
||
| import optuna | ||
| import optunahub | ||
| import matplotlib.pyplot as plt | ||
|
|
||
| bbob_constrained = optunahub.load_module("benchmarks/bbob_constrained") | ||
| constrained_sphere2d = bbob_constrained.Problem(function_id=1, dimension=2, instance_id=1) | ||
|
|
||
| study = optuna.create_study( | ||
| sampler=optuna.samplers.TPESampler( | ||
| constraints_func=constrained_sphere2d.constraints_func, | ||
| seed=42 | ||
| ), | ||
| directions=constrained_sphere2d.directions | ||
| ) | ||
| study.optimize(constrained_sphere2d, n_trials=100) | ||
|
|
||
| try: | ||
| print(study.best_trial.params, study.best_trial.value) | ||
| except Exception as e: | ||
| print(e) | ||
|
|
||
| optuna.visualization.plot_optimization_history(study).show() | ||
| plt.show() | ||
|
|
||
| .. figure:: ./images/optimization_history_constrained.png | ||
| :alt: Optimization History | ||
| :align: center | ||
| :width: 800px | ||
|
|
||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Please add an explanation of evaluate_constraints, which allows constrained optimization with optimizers other than Optuna by directly evaluating constraints with user-provided parameters. |
||
|
|
||
| Multi-objective Problem | ||
| ^^^^^^^^^^^^^^^^^^^^^^^ | ||
|
|
||
| You can also try multi-objective optimization by using the `the WFG Problem Collection <https://hub.optuna.org/benchmarks/wfg/>`__ module. | ||
| In order to use this module, you need to install `optproblems <https://pypi.org/project/optproblems/>`__ and `diversipy <https://pypi.org/project/diversipy/>`__ packages. | ||
|
|
||
| .. code-block:: console | ||
|
|
||
| $ pip install -U optproblems diversipy | ||
|
|
||
| Example is as follows: | ||
|
|
||
| .. code-block:: python | ||
|
|
||
| import optuna | ||
| import optunahub | ||
|
|
||
|
|
||
| wfg = optunahub.load_module("benchmarks/wfg") | ||
| wfg4 = wfg.Problem(function_id=4, n_objectives=2, dimension=3, k=1) | ||
|
|
||
| study_pareto = optuna.create_study( | ||
| study_name="ParetoFront", directions=wfg4.directions | ||
| ) | ||
| for x in wfg4.get_optimal_solutions(1000): | ||
| study_pareto.enqueue_trial(params={ | ||
| f"x{i}": x.phenome[i] for i in range(3) | ||
| }) | ||
| study_pareto.optimize(wfg4, n_trials=1000) | ||
|
|
||
| study_tpe = optuna.create_study( | ||
| study_name="TPESampler", | ||
| sampler=optuna.samplers.TPESampler(seed=42), directions=wfg4.directions | ||
| ) | ||
| study_tpe.optimize(wfg4, n_trials=1000) | ||
|
|
||
| optunahub.load_module("visualization/plot_pareto_front_multi").plot_pareto_front( | ||
| [study_pareto, study_tpe] | ||
| ).show() | ||
|
|
||
| .. figure:: ./images/pareto_front.png | ||
| :alt: Pareto Front | ||
| :align: center | ||
| :width: 800px | ||

There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please add some explanation of what OptunaHub Benchmarks is.
It is also helpful to add links to the Medium article (https://medium.com/optuna/optunahub-benchmarks-a-new-feature-to-use-register-various-benchmark-problems-694401524ce0) and how to implement original benchmarks (https://optuna.github.io/optunahub/generated/recipes/006_benchmarks_basic.html, https://optuna.github.io/optunahub/generated/recipes/007_benchmarks_advanced.html).