OrthoBoSampler Implementation

package/samplers/orthobo/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Maxim Kirby.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/samplers/orthobo/README.md

@@ -0,0 +1,101 @@
+---
+author: Maxim Kirby
+title: 'ORTHOBO: Orthogonal Bayesian Hyperparameter Optimization'
+description: A drop-in Optuna sampler that reduces acquisition estimation noise in marginal Bayesian optimization using orthogonal score-function control variates, improving candidate ranking stability and optimization performance.
+tags: [sampler, bayesian-optimization, gaussian-process, variance-reduction, botorch]
+optuna_versions: [4.8.0]
+license: MIT License
+---
+
+## Abstract
+
+Standard Bayesian optimization handles uncertainty by taking random samples of model parameters, but this random sampling creates noise. This noise can cause the optimizer to accidentally rank bad configurations above good ones and steer the search in the wrong direction. This issue is especially noticeable in complex tasks with many variables or when computational limits restrict the number of samples you can take.
+
+This package implements OrthoBO, a framework introduced in [this paper from May 2026](https://arxiv.org/abs/2605.06454). OrthoBO uses an orthogonal correction to cancel out the noise caused by finite sampling. It keeps the estimates accurate while making the candidate rankings much more stable. The result is better optimization performance.
+
+OrthoBO is designed as a drop-in replacement for Optuna's default BoTorch sampler. It works with almost any standard Bayesian optimization problem and requires absolutely no changes to your objective function or search space.
+
+## APIs
+
+### `OrthoBoSampler`
+
+```python
+OrthoBoSampler(
+    n_startup_trials: int = 10,
+    mc_budget: int = 64,
+    use_orthogonal_correction bool = True,
+    seed: int | None = None,
+)
+```
+
+**Parameters:**
+
+- `n_startup_trials`:
+  Number of initial quasi-random (Sobol) trials before BO begins. Defaults to 10.
+- `mc_budget`:
+  The number of GP models sampled. Higher values reduce variance but increase compute time. Defaults to 64.
+- `use_orthogonal_correction`:
+  If True, applies the orthogonal score-function control variate for variance reduction (OrthoBO mode). If False, falls back to plain MC marginalisation over the hyperposterior (Naive Marginal BO mode). Defaults to True
+- `seed`:
+  Random seed for the Sobol startup sampler. Set for reproducibility. Defaults to None.
+
+## Installation
+
+```bash
+pip install torch botorch gpytorch optuna optuna-integration
+```
+
+## Example
+
+```python
+import optuna
+import optunahub
+
+def objective(trial: optuna.Trial) -> float:
+    x = trial.suggest_float("x", -5.0, 5.0)
+    y = trial.suggest_float("y", -5.0, 5.0)
+    return (x - 2) ** 2 + (y + 1) ** 2
+
+OrthoBoSampler = optunahub.load_module("samplers/orthobo").OrthoBoSampler
+
+sampler = OrthoBoSampler(n_startup_trials=10, mc_budget=64, use_orthogonal_correction=True)
+study = optuna.create_study(direction="minimize", sampler=sampler)
+study.optimize(objective, n_trials=50)
+
+print(f"Best value: {study.best_value:.4f}")
+print(f"Best params: {study.best_params}")
+```
+
+## Benchmark Results
+
+Mean best-so-far regret ± 1 std across 5 random seeds. OrthoBO (green) is
+compared against Vanilla BO (standard single-GP qLogEI, blue) and Naive
+Marginal BO (MC marginalisation without orthogonal correction, orange).
+
+### Hartmann-6 (6 dimensions)
+
+<p align="center">
+  <img src="images/Hartmann6.png" alt="Hartmann6" style="max-width: 100%; width: 600px;">
+</p>
+
+### Levy-16 (16 dimensions)
+
+<p align="center">
+  <img src="images/Levy16.png" alt="Levy16" style="max-width: 100%; width: 600px;">
+</p>
+
+OrthoBO's advantage is most pronounced on higher-dimensional problems where acquisition estimation noise has a larger effect on candidate rankings. On lower-dimensional problems (Hartmann-6) where the GP fits well with few observations, the methods converge to similar performance.
+
+## Reference
+
+```bibtex
+@misc{schröder2026orthoboorthogonalbayesianhyperparameter,
+      title={{ORTHOBO}: Orthogonal Bayesian Hyperparameter Optimization}, 
+      author={Maresa Schröder and Pascal Janetzky and Michael Klar and Stefan Feuerriegel},
+      year={2026},
+      eprint={2605.06454},
+      archivePrefix={arXiv},
+      primaryClass={cs.LG},
+      url={https://arxiv.org/abs/2605.06454}, 
+}
+```

package/samplers/orthobo/__init__.py

@@ -0,0 +1,4 @@
+from .sampler import OrthoBoSampler
+
+
+__all__ = ["OrthoBoSampler"]

package/samplers/orthobo/example.py

@@ -0,0 +1,28 @@
+from __future__ import annotations
+
+import optuna
+import optunahub
+
+
+optuna.logging.set_verbosity(optuna.logging.WARNING)
+
+
+def hartmann6(trial: optuna.Trial) -> float:
+    """Hartmann-6 benchmark. Global minimum: -3.32237 at ~(0.20, 0.15, 0.48, 0.27, 0.31, 0.66)."""
+    from botorch.test_functions import Hartmann
+    import torch
+
+    f = Hartmann(dim=6)
+    x = torch.tensor([trial.suggest_float(f"x{i}", 0.0, 1.0) for i in range(6)])
+    return f(x.unsqueeze(0)).item()
+
+
+if __name__ == "__main__":
+    OrthoBoSampler = optunahub.load_module("samplers/orthobo").OrthoBoSampler
+
+    sampler = OrthoBoSampler(n_startup_trials=10, mc_budget=64)
+    study = optuna.create_study(direction="minimize", sampler=sampler)
+    study.optimize(hartmann6, n_trials=30)
+
+    print(f"Best value : {study.best_value:.5f}  (global min: -3.32237)")
+    print(f"Best params: {study.best_params}")

package/samplers/orthobo/requirements.txt

@@ -0,0 +1,5 @@
+torch>=2.0.0
+botorch>=0.9.0
+gpytorch>=1.11
+optuna>=4.0.0
+optuna-integration>=0.1.0