Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
128 changes: 128 additions & 0 deletions docs/source/tutorials_for_users/how_to_use_benchmarks.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
How to Use Benchmarks
=====================

In this tutorial, we will explain how to use benchmarks in optunahub.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
In this tutorial, we will explain how to use benchmarks in optunahub.
In this tutorial, we will explain how to use benchmarks in OptunaHub.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


Preparation
-----------

First, ensure the necessary packages are installed by executing the following command:

.. code-block:: console
$ pip install optuna optunahub
Examples
--------

We will use the `black-box optimization benchmarking (bbob) test suite <https://hub.optuna.org/benchmarks/bbob/>`__ in this tutorial.
This is a wrapper of `COCO (COmparing Continuous Optimizers) experiment <https://github.com/numbbo/coco-experiment>`__ library.
So you need to install COCO first.

.. code-block:: console
$ pip install coco-experiment
Test code is as follows:

.. code-block:: python
import optuna
import optunahub
bbob = optunahub.load_module("benchmarks/bbob")
sphere2d = bbob.Problem(function_id=1, dimension=2, instance_id=1)
study = optuna.create_study(directions=sphere2d.directions, sampler=optuna.samplers.TPESampler(seed=42))
study.optimize(sphere2d, n_trials=100)
optuna.visualization.plot_optimization_history(study).show()
.. figure:: ./images/optimization_history.png
:alt: Optimization History
:align: center
:width: 800px

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add an example to use OptunaHub Benchmarks with optimizers other than Optuna, like https://medium.com/optuna/optunahub-benchmarks-a-new-feature-to-use-register-various-benchmark-problems-694401524ce0.
image

Copy link
Member

@y0z y0z Sep 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The doc of the evaluate method is here.

Constrained Problem
^^^^^^^^^^^^^^^^^^^

There are also some benchmarks that have constraints.

.. code-block:: python
import optuna
import optunahub
import matplotlib.pyplot as plt
bbob_constrained = optunahub.load_module("benchmarks/bbob_constrained")
constrained_sphere2d = bbob_constrained.Problem(function_id=1, dimension=2, instance_id=1)
study = optuna.create_study(
sampler=optuna.samplers.TPESampler(
constraints_func=constrained_sphere2d.constraints_func,
seed=42
),
directions=constrained_sphere2d.directions
)
study.optimize(constrained_sphere2d, n_trials=100)
try:
print(study.best_trial.params, study.best_trial.value)
except Exception as e:
print(e)
optuna.visualization.plot_optimization_history(study).show()
plt.show()
.. figure:: ./images/optimization_history_constrained.png
:alt: Optimization History
:align: center
:width: 800px

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add an explanation of evaluate_constraints, which allows constrained optimization with optimizers other than Optuna by directly evaluating constraints with user-provided parameters.


Multi-objective Problem
^^^^^^^^^^^^^^^^^^^^^^^

You can also try multi-objective optimization by using the `the WFG Problem Collection <https://hub.optuna.org/benchmarks/wfg/>`__ module.
In order to use this module, you need to install `optproblems <https://pypi.org/project/optproblems/>`__ and `diversipy <https://pypi.org/project/diversipy/>`__ packages.

.. code-block:: console
$ pip install -U optproblems diversipy
Example is as follows:

.. code-block:: python
import optuna
import optunahub
wfg = optunahub.load_module("benchmarks/wfg")
wfg4 = wfg.Problem(function_id=4, n_objectives=2, dimension=3, k=1)
study_pareto = optuna.create_study(
study_name="ParetoFront", directions=wfg4.directions
)
for x in wfg4.get_optimal_solutions(1000):
study_pareto.enqueue_trial(params={
f"x{i}": x.phenome[i] for i in range(3)
})
study_pareto.optimize(wfg4, n_trials=1000)
study_tpe = optuna.create_study(
study_name="TPESampler",
sampler=optuna.samplers.TPESampler(seed=42), directions=wfg4.directions
)
study_tpe.optimize(wfg4, n_trials=1000)
optunahub.load_module("visualization/plot_pareto_front_multi").plot_pareto_front(
[study_pareto, study_tpe]
).show()
.. figure:: ./images/pareto_front.png
:alt: Pareto Front
:align: center
:width: 800px
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
3 changes: 2 additions & 1 deletion docs/source/tutorials_for_users/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,5 @@ Tutorials for Users
.. toctree::
:maxdepth: 1

plot-in-optuna-dashboard
plot-in-optuna-dashboard
how_to_use_benchmarks
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
Tutorial: How to Plot in Optuna-Dashboard
How to Plot in Optuna-Dashboard
=========================================

In this tutorial, we will explain how to display plots in `Optuna-Dashboard <https://optuna-dashboard.readthedocs.io/en/latest/index.html>`_.
Expand Down
Loading