Add ConfOpt Sampler

[rick12000]

Contributor Agreements

Please read the contributor agreements and if you agree, please click the checkbox below.

• I agree to the contributor agreements.

Tip

Please follow the Quick TODO list to smoothly merge your PR.

Motivation

Make ConfOpt available as a Sampler in Optuna.

Description of the changes

Overview:

Adds ConfOpt as a new Sampler.

Implementation Notes:

• Utilizes existing ConfOpt PyPI package's assets, requiring it as a dependancy and acting as a wrapper.
• Wrapper consists of:
  • Initializing QuantileConformalSearcher according to user inputs (with expert defaults).
  • sample_relative() call, which fits QuantileConformalSearcher to all currently available trials (user can specify whether to train on only completed, or also pruned trials) and selects next configuration to evaluate through objective function.
  • after_trial() call, which calls the QuantileConformalSearcher's update() method. This will perform a series of internal ConfOpt updates on the QuantileConformalSearcher state (namely conformal prediction adjustments and acquisition function updates).

In addition to default search, implementation supports (tested):

• Parallelization via n_jobs Optuna parameter in .optimize().
• Pruning.

Note, to allow for safe parallelization, a thread safe local copy of QuantileConformalSearcher is used in sample_relative(). Fit and predict method changes to the searcher's state are unimportant and don't need to persist between trials, so using a local copy per trial is harmless.

In after_trial(), we force a lock, so only one thread/process can update the true, non-copied, QuantileConformalSearcher object with after trial information (preserving single source of truth for searcher's permanent/cumulative states). The lock doesn't worsen parallelization performance, since the update takes fractions of a second.

TODO List towards PR Merge

Please remove this section if this PR is not an addition of a new package.
Otherwise, please check the following TODO list:

• Copy ./template/ to create your package
• Replace <COPYRIGHT HOLDER> in LICENSE of your package with your name
• Fill out README.md in your package
• Add import statements of your function or class names to be used in __init__.py
• (Optional) Add from __future__ import annotations at the head of any Python files that include typing to support older Python versions
• Apply the formatter based on the tips in README.md
• Check whether your module works as intended based on the tips in README.md

[y0z]

@not522, could you review this PR?

[not522]

Thank you for your PR!