Add BO termination conditions

[wgst]

Motivation

BO loops in BoFire currently run for a fixed number of iterations. In practice, especially for chemical experiments, where each iteration is very costly, we need a way to stop optimization early enough to reduce costs but still get a converged result.

This is an attempt to implement different termination conditions methods. The first method currently added in this PR is from Makarova, A. et al. (2022). The idea behind this method is to compute an upper bound on the simple regret of the current best recommendation using the GP posterior, and when this bound falls below a threshold ε_BO (derived from the observation noise variance), then we are confident that any further improvement will be smaller than the noise, and optimization can stop.

Two other termination methods - issues with implementing

1. EMSR Gap: Ishibashi, H. et al. (2023). This method requires computing an Expected Minimum Simple Regret gap between consecutive iterations, which needs per-iteration model snapshots or knowledge of which experiments belong to which iteration. I have not found any tracking functionalities within BoFire, which makes it hard to implement this method at the moment.

2. Probabilistic Regret Bound (PRB): Wilson, J. et al. (2024). This is a more probabilistic method, in which a statistically valid stopping rule is constructed using Matheron-rule pathwise trajectory samples (Thompson-style posterior samples via pathwise conditioning). Their implementation is based on TensorFlow and BO packages around it, and some crucial functionalities are not available in BoTorch.

Both methods could be good candidates for future work. Adding the method by Ishibashi et al should be fairly easy once the tracking architecture is settled. Method by Wilson could be a bit more problematic, but let's see.

Have you read the Contributing Guidelines on pull requests?

Yes

Test Plan

Added tests:

• data model (test_termination.py)
• evaluator (test_evaluator.py)
• mapper (test_mapper.py)

Best,
Wojciech

[Copilot]

Pull request overview

Adds support for Bayesian Optimization termination conditions to allow BO runs to stop early based on convergence criteria (initially: the UCB–LCB simple-regret upper bound inspired by Makarova et al., 2022). This integrates termination evaluation into the existing run() benchmark loop and adds a small termination submodule with data models, evaluators, and mapping utilities.

Changes:

• Introduces termination-condition data models (MaxIterationsTermination, AlwaysContinue, UCBLCBRegretTermination, CombiTerminationCondition).
• Adds a termination evaluator (UCBLCBRegretEvaluator) and mapper utilities to determine which evaluators are needed.
• Extends the benchmark runner to optionally stop early and to return a richer RunResult including termination metadata (plus tests).

Reviewed changes

Copilot reviewed 11 out of 13 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
bofire/data_models/termination/termination.py	New termination-condition data models + should_terminate logic.
bofire/data_models/termination/api.py	Exports termination-condition data models.
bofire/data_models/termination/init.py	Package init (empty).
bofire/termination/evaluator.py	Implements evaluator computing UCB–LCB regret bound + noise estimate.
bofire/termination/mapper.py	Maps conditions to evaluators; recursively gathers needed evaluators.
bofire/termination/api.py	Exports evaluator + mapping helpers.
bofire/termination/init.py	Package init (empty).
bofire/runners/run.py	Adds termination_condition support, early-stop logic, and new RunResult return type.
bofire/runners/api.py	Re-exports RunResult from runners API.
tests/bofire/data_models/test_termination.py	Unit tests for termination condition data models + serialization.
tests/bofire/termination/test_evaluator.py	Unit + integration tests for evaluator and run() early termination behavior.
tests/bofire/termination/test_mapper.py	Tests for termination mapper and evaluator discovery.
tests/bofire/termination/init.py	Test package init.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[jduerholt]

Hi @wgst,

thanks for the PR and sorry for the late answer. @bertiqwerty will provide a thorough and more detailed review. One general comment from my side:

I think it would be the best to incorparate the termination criteria not into the runner but into the strategy itself, we have the StepwiseStrategy that can be used to compose different atomic strategies into one including conditions when to execute which one (https://github.com/experimental-design/bofire/blob/main/bofire/data_models/strategies/stepwise/stepwise.py https://github.com/experimental-design/bofire/blob/main/bofire/strategies/stepwise/stepwise.py) I think it would be nice, to implement the stopping criteria as a condition for a certain stop. What do you think?

Best,

JOhannes

[jduerholt]

This one also already exisits in the StepwiseStrategy as a condition.

[wgst]

Good idea! I have now moved everything to StepwiseStrategy, so I hope it's in the right place now.

[bertiqwerty]

Thanks @wgst and @jduerholt. I agree with Johannes. Please use the StepwiseStrategy. If you need more guidance on how to do that, @wgst, let us have a call. I will have a closer look as soon as this refactored.

[bertiqwerty]

Please refactor into the StepwiseStrategy.

[bertiqwerty]

Sorry, for the delay. I will try to look into this next week.

[wgst]

Hi @jduerholt, and @bertiqwerty,

I implemented all four termination conditions that I intended to.

This includes:

• Makarova et al. (2022) method (UCBLCBRegretBoundCondition), which tracks an upper bound on the simple regret (the gap min UCB(evaluated) - min LCB(domain) from GP-UCB confidence bounds), and stops the BO loop whenever it falls below a threshold. The threshold can be a chosen fixed value (e.g., experimental error), the GP's estimated noise, or a cross-validation-based estimate.
• Ishibashi et al. (2023) method (ExpMinRegretGapCondition) that builds on Makarova et al. method by also evaluating the UCB-LCB regret bound, but it additionally introduces new terms, including the KL divergence between the predictive distribution of GP at the newest point before vs after conditioning on that new observation. The threshold can be adaptive (self-calibrating from the GP posterior variances and noise) or a median-based heuristic.
• Xie. et al. (2023) method (LogEIPCCondition), a cost-aware stopping for when the cost of experiments plays a crucial role. It uses a cost-adjusted acquisition function, LogEIC(x) = LogEI(x) − α·log c(x), and stops when no candidate's expected improvement is worth its evaluation cost. Currently, the LogEIC acquisition function is implemented within the BoFire (temporarily). However, I already submitted a BoTorch PR, so that the acquisition functions stay in BoTorch, not BoFire: Add LogExpectedImprovementPerCost acquisition function meta-pytorch/botorch#3304.
• Wilson, J. et al. (2024), the Probabilistic Regret Bound (PRB) method (ProbabilisticRegretBoundCondition), which utilizes Matheron-rule pathwise trajectory samples (posterior samples via pathwise conditioning), minimises each sampled path, and stops once a sequential Clopper–Pearson test certifies that P(regret ≤ ε) ≥ 1 − δ (where ε is the simple-regret tolerance and δ is the total risk budget, split between model risk and Monte-Carlo estimation risk, and both ε and δ are set by the user).

As suggested by you previously, I've also moved all the conditions to StepwiseStrategy.

Best,
Wojciech