Merge branch 'development' into v2.4.0

Author: mwever

README.md

@@ -38,8 +38,7 @@ We hope you enjoy this new user experience as much as we do. 🚀
 
 This instruction is for the installation on a Linux system, for Windows and Mac and further information see the [documentation](https://automl.github.io/SMAC3/latest/1_installation/).
 
-Create a new environment with python 3.10 and make sure swig is installed either on your system or
-inside the environment. We demonstrate the installation via anaconda in the following:
+Create a new environment with python 3.10. We demonstrate the installation via anaconda in the following:
 
 Create and activate environment:
 ```
@@ -62,7 +61,7 @@ make install-dev
 starting from 2.4.0, SMAC uses random forest from [sklearn](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html)
 instead of random forest from [pyrfr](https://pypi.org/project/pyrfr/) as the default surrogate model for HPO tasks.
 However, you could still use the old pyrfr surrogate model by calling `smac.facade.old.HyperparameterOptimizationRFRFacade`
-and `smac.facade.old.MultiFidelityRFRFacade`
+and `smac.facade.old.MultiFidelityRFRFacade`.
 
 To work with pyrfr, you need to first install gcc, gxx, and swig:
 ```

docs/1_installation.md

@@ -1,12 +1,11 @@
 # Installation
 ## Requirements
 
-SMAC is written in python3 and therefore requires an environment with python>=3.8.
-Furthermore, the Random Forest used in SMAC requires SWIG as a build dependency.
+SMAC is written in python3 and therefore requires an environment with python>=3.9.
 
 !!! info 
 
-    SMAC is tested on Linux and Mac machines with python >=3.8.
+    SMAC is tested on Linux and Mac machines with python >=3.9.
 
 
 ## SetUp
@@ -18,17 +17,6 @@ conda create -n SMAC python=3.10
 conda activate SMAC
 ```
 
-Now install swig either on the system level e.g. using the following command for Linux:
-```bash
-apt-get install swig
-```
-
-Or install swig inside of an already created conda environment using:
-
-```bash
-conda install gxx_linux-64 gcc_linux-64 swig
-```
-
 ## Install SMAC
 You can install SMAC either using PyPI or Conda-forge.
 
@@ -63,7 +51,31 @@ conda install smac
 
 Read [SMAC feedstock](https://github.com/conda-forge/smac-feedstock) for more details.
 
-## Windows (native or via WSL, experimental)
+## Running SMAC with pyrfr
+starting from 2.4.0, SMAC uses random forest from [sklearn](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html)
+instead of random forest from [pyrfr](https://pypi.org/project/pyrfr/) as the default surrogate model for HPO tasks.
+However, you could still use the old pyrfr surrogate model by calling `smac.facade.old.HyperparameterOptimizationRFRFacade`
+and `smac.facade.old.MultiFidelityRFRFacade`.
+
+The Random Forest used in SMAC requires SWIG as a build dependency. 
+You could install swig either on the system level e.g. using the following command for Linux:
+```bash
+apt-get install swig
+```
+
+Or install swig inside of an already created conda environment using:
+
+```bash
+conda install gxx_linux-64 gcc_linux-64 swig
+```
+
+And then install smac with the pyrfr option:
+```
+pip install smac[pyrfr]
+```
+
+
+### pyrfr on Windows (native or via WSL, experimental)
 
 SMAC can be installed under Windows in a WSL (Windows Subsystem for Linux). 
 You can find an instruction on how to do this here: [Experimental](./10_experimental.md).

docs/3_getting_started.md

@@ -84,6 +84,11 @@ scenario = Scenario(
 )
 ```
 
+!!! info "Deterministic Behavior"
+    The **`deterministic`** parameter controls whether the objective function is assumed to be noise-free.
+    * **For Deterministic Objectives (`deterministic=True`):** Use this when repeated evaluations of the same configuration produce identical results. In this mode, each configuration is evaluated only once, and the surrogate model does not include a noise term.
+    * **For Stochastic Objectives (`deterministic=False`):** Use this when the objective is noisy. Intensification parameters like `max_config_calls` and `n_seeds` must be set to enable repeated evaluations per configuration.
+
 !!! note
     If no `name` is given, a hash of the experiment is used. Running the same experiment again at a later time will result in exactly the same hash. This is important, because the optimization will warmstart on the preexisting evaluations, if not otherwise specified in the [Facade][smac.facade.abstract_facade].
 
@@ -150,4 +155,4 @@ it supports access to e.g. the runhistory with
 ````
 runhistory = smbo.runhistory()
 ````
-For more information check out the [API][smac.main.smbo].
+For more information check out the [API][smac.main.smbo].

docs/advanced_usage/13_adaptive_capping.md

@@ -0,0 +1,107 @@
+# Adaptive Capping
+
+Adaptive capping is a feature that can be used to speedup the evaluation of candidate configurations when the objective
+is to minimize runtime of an algorithm across a set of instances. The basic idea is to terminate unpromising candidates
+early and adapting the timeout for solving a single instance dynamically based on the incumbent's runtime and the.
+runtime already used by the challenging configuration.
+
+## Theoretical Background
+
+When comparing a challenger configuration with the current incumbent for a (sub-)set of instances, we already know how
+much cost (in terms of runtime) was incurred by the incumbent to solve the set of instances. As soon as the challenger
+configuration exceeds the cost of the incumbent, it is evident that the challenger will not become the new incumbent
+since the costs accumulate over time and are strictly positive, i.e., solving an instance cannot have negative runtime.
+
+Example:
+*Let the incumbent be evaluated for two instances with observed runtimes 3s and 4s. When a challenger configuration is
+evaluated and compared against the incumbent, it is first evaluated on a first instance. For example, we observe a
+runtime of 2s. As the challenger appears to be a promising configuration, its evaluation is intensified and the budget
+is doubled, i.e., the budget is increased to 2. For solving the second instance, adaptive capping will allow a timeout
+of 5s since the sum of runtimes for the incumbent is 7s and the challenger used up 2s for solving the first instance so
+far so that 5s remain until the costs of the incumbent are exceeded. Even if the challenger configuration would need 10s
+to solve the second instance, its execution would be aborted. In this example, by adaptive capping we thus save 5s of
+evaluation costs for the challenger to notice that it will not replace the current incumbent.*
+
+In combination with random online aggressive racing, we can further speedup the evaluation of challenger configurations
+as we increase the horizon for adaptive capping step by step with every step of intensification. Note that
+intensification will double the number of instances to which the challenger configuration (and eventually also the
+incumbent configuration) are applied to. Furthermore, to increase the trust into the current incumbent, the incumbent is
+regularly subject to intensification.
+
+
+## Setting up Adaptive Capping
+
+To achieve this, the user must take active care in the termination of their target function.
+The capped problem.train will receive a budget keyword argument, detailing the seconds allocated to the configuration.
+Below is an example of a capped problem that will return the used budget if the computation exceeds the budget.
+
+
+```python
+
+    class TimeoutException(Exception):
+        pass
+
+
+    @contextmanager
+    def timeout(seconds):
+        def handler(signum, frame):
+            raise TimeoutException(f"Function call exceeded timeout of {seconds} seconds")
+
+        # Set the signal handler for the alarm signal
+        signal.signal(signal.SIGALRM, handler)
+        signal.alarm(seconds)  # Schedule an alarm after the given number of seconds
+
+        try:
+            yield
+        finally:
+            # Cancel the alarm if the block finishes before timeout
+            signal.alarm(0)
+
+
+    class CappedProblem:
+        @property
+        def configspace(self) -> ConfigurationSpace:
+            ...
+
+        def train(self, config: Configuration, instance:str, budget, seed: int = 0) -> float:
+
+            try:
+                with timeout(int(math.ceil(budget))):
+                    start_time = time.time()
+                    ... # heavy computation
+                    runtime = time.time() - start_time
+                    return runtime
+            except TimeoutException as e:
+                print(f"Timeout for configuration {config} with runtime budget {budget}")
+                return budget # here the runtime is capped and we return the used budget.
+```
+
+In order to enable adaptive capping in smac, we need to create [problem instances](4_instances.md) to optimize over and specify a
+global runtime cutoff in the intensifier. Then we optimize as usual.
+
+
+```python
+from smac.intensifier import Intensifier
+from smac.scenario.scenario import Scenario
+
+scenario = Scenario(
+    capped_problem.configspace,
+    ...
+    instances=['1', '2', '3'], # add problem instances we want to solve
+    instance_features={'1': [1], '2': [2], '3': [3]} # in the absence of actual features add dummy features for identification
+)
+
+intensifier = Intensifier(
+scenario,
+runtime_cutoff=10 # specify an absolute runtime cutoff (sum over instances) never to be exceeded
+)
+
+smac = HyperparameterOptimizationFacade(
+    scenario,
+    capped_problem.train,
+    intensifier=intensifier,
+    ...
+)
+
+incumbent = smac.optimize()
+```

examples/1_basics/8_adaptive_capping.py

@@ -0,0 +1,98 @@
+"""Adaptive Capping
+# Flags: doc-Runnable
+
+Adaptive capping is often used in optimization algorithms, particularly in
+scenarios where the time taken to evaluate solutions can vary significantly.
+For more details on adaptive capping, consult the [info page adaptive capping](../../../advanced_usage/13_adaptive_capping.html).
+
+"""
+import math
+import time
+
+import signal
+from contextlib import contextmanager
+
+from smac.runhistory import InstanceSeedBudgetKey, TrialInfo
+
+import warnings
+
+from ConfigSpace import Categorical, Configuration, ConfigurationSpace, Float
+
+
+class TimeoutException(Exception):
+    pass
+
+@contextmanager
+def timeout(seconds):
+    def handler(signum, frame):
+        raise TimeoutException(f"Function call exceeded timeout of {seconds} seconds")
+
+    # Set the signal handler for the alarm signal
+    signal.signal(signal.SIGALRM, handler)
+    signal.alarm(seconds)  # Schedule an alarm after the given number of seconds
+
+    try:
+        yield
+    finally:
+        # Cancel the alarm if the block finishes before timeout
+        signal.alarm(0)
+
+
+class CappedProblem:
+    @property
+    def configspace(self) -> ConfigurationSpace:
+        cs = ConfigurationSpace(seed=0)
+        x0 = Float("x0", (0, 5), default=5, log=False)
+        x1 = Float("x1", (0, 7), default=7, log=False)
+        cs.add_hyperparameters([x0, x1])
+        return cs
+
+    def train(self, config: Configuration, instance:str, cutoff, seed: int = 0) -> float:
+        x0 = config["x0"]
+        x1 = config["x1"]
+
+        try:
+            with timeout(int(math.ceil(cutoff))):
+                runtime = 0.5 * x1 + 0.5 * x0 * int(instance)
+                time.sleep(runtime)
+                return runtime
+        except TimeoutException as e:
+            print(f"Timeout for configuration {config} with runtime cutoff {cutoff}")
+            return cutoff + 1
+
+
+if __name__ == '__main__':
+    from smac import AlgorithmConfigurationFacade
+    from smac import Scenario
+
+    capped_problem = CappedProblem()
+
+    scenario = Scenario(
+        capped_problem.configspace,
+        walltime_limit=3600,  # After 3600 seconds, we stop the hyperparameter optimization
+        n_trials=500,  # Evaluate max 500 different trials
+        instances=['1', '2', '3'],
+        instance_features={'1': [1], '2': [2], '3': [3]},
+        adaptive_capping=True,
+        runtime_cutoff=200 # We allow an algorithm at maximum 200s to solve all instances
+    )
+
+    # We want to run five random configurations before starting the optimization.
+    initial_design = AlgorithmConfigurationFacade.get_initial_design(scenario, n_configs=5)
+
+    # Create our SMAC object and pass the scenario and the train method
+    smac = AlgorithmConfigurationFacade(
+        scenario,
+        capped_problem.train,
+        initial_design=initial_design,
+        overwrite=True,
+    )
+
+    # Let's optimize
+    incumbent = smac.optimize()
+
+    # Get cost of default configuration
+    default_cost = smac.validate(capped_problem.configspace.get_default_configuration())
+
+    # Let's calculate the cost of the incumbent
+    incumbent_cost = smac.validate(incumbent)

examples/1_basics/8_warmstart.py

@@ -3,7 +3,7 @@
 
 With the ask and tell interface, we can support warmstarting SMAC. We can communicate rich
 information about the previous trials to SMAC using `TrialInfo` and `TrialValue` instances.
-For more details on ask and tell consult the [info page ask-and-tell](../../../advanced_usage/5_ask_and_tell).
+For more details on ask and tell consult the [info page ask-and-tell](../../../advanced_usage/5_ask_and_tell.html).
 """
 from __future__ import annotations
 

mkdocs.yaml

@@ -207,6 +207,7 @@ nav:
     - "advanced_usage/10_continue.md"
     - "advanced_usage/11_reproducibility.md"
     - "advanced_usage/12_optimizations.md"
+    - "advanced_usage/13_adaptive_capping.md"
   # Auto generated with docs/examples_runner.py
   - Examples: "examples/"
   # Auto generated with docs/api_generator.py