MO-SMAC merge #1222

jeroenrook · 2025-03-27T09:24:50Z

Multi-objective SMAC as described in https://doi.org/10.1162/evco_a_00371

This is a safety measure. Normally, everytime we update the runhistory, the objective bounds are updated so that the value to normalize should be inside the bound.

Created a helper method to create a set with preserved order from a list

Previously: random scalarization of MO costs bc ParEGO was the only MO algo. Now: separate function which can be overwritten.

Also, reset obtain to the kwargs (no magic numbers) Better debug message

Updates incumbents of runhistory automatically if updated

# Conflicts: # smac/acquisition/maximizer/local_search.py

…e retraining and forces to run the maximization procedure immediately to time the computation cost.

Early termination of unfruitful configurations.

…e intensifier class.

…eved from the RunHistoryEncoder (which could mean they are not normalized. Therefore the predictions are not either. Hence, we need to normalize the predictions of the objectives here before computing the hypervolumes.

…y play around with different implementations.

# Conflicts: # smac/acquisition/maximizer/local_search.py # smac/intensifier/abstract_intensifier.py

# Conflicts: # README.md # smac/intensifier/abstract_intensifier.py # smac/utils/multi_objective.py

jeroenrook · 2025-03-27T09:25:59Z

smac/acquisition/function/expected_hypervolume.py

rename file to hypervolume

jeroenrook · 2025-03-27T09:30:36Z

smac/utils/pareto_front.py

@@ -153,3 +156,62 @@ def sort_by_crowding_distance(
    config_with_crowding = sorted(config_with_crowding, key=lambda x: x[1], reverse=True)

    return [c for c, _ in config_with_crowding]
+
+def sort_by_hypervolume_contribution(


All functions seem not to be implemented

benjamc

In general:

add docstrings for classes and private functions
abstract intermediate update and sth else
check all TODOs
fix tests
write new tests, the coverage should be higher than before

benjamc · 2025-03-27T09:36:12Z

README.md

-We are excited to introduce the new major release and look forward to developing new features on the new code base. 
-We hope you enjoy this new user experience as much as we do. 🚀
-
+MO-SMAC is implemented directly into SMAC3. This repository is forked from the [SMAC3 repository](https://github.com/automl/SMAC3) and therefore contains references and copyright information to those authors. 


update README

benjamc · 2025-03-27T09:36:54Z

examples/2_multi_fidelity/1_mlp_epochs.py

+            objectives="accuracy",
+            # min_budget=1,  # Train the MLP using a hyperparameter configuration for at least 5 epochs
+            # max_budget=25,  # Train the MLP using a hyperparameter configuration for at most 25 epochs
+            n_workers=4,


test example

smac/acquisition/function/expected_hypervolume.py

benjamc · 2025-03-27T09:39:54Z

smac/acquisition/maximizer/local_search.py

@@ -123,7 +118,7 @@ def _maximize(
        # Sort according to acq value
        configs_acq.sort(reverse=True, key=lambda x: x[0])
        for a, inc in configs_acq:
-            inc.origin = "Acquisition Function Maximizer: Local Search"
+            inc.origin = "Local Search"


change back to AF Max --> make consistent for all (TODO @benjamc )

Fixed this already to make the pytests work again

benjamc · 2025-03-27T09:47:42Z

smac/facade/algorithm_configuration_facade.py

@@ -115,7 +116,12 @@ def get_intensifier(
        max_incumbents : int, defaults to 10
            How many incumbents to keep track of in the case of multi-objective.
        """
-        return Intensifier(
+        class NewIntensifier(intermediate_decision.NewCostDominatesOldCost,


rename intensifier LekkerMOACIntensifier

Combine with abstraction of the intensifier

benjamc · 2025-03-27T14:20:55Z

smac/acquisition/maximizer/local_search.py


        return init_points

+    def _create_sort_keys(self, costs: np.array) -> list[list[float]]:


What impact does the way of sorting have? Any intuition?

This is to get points to perform the local search with based on earlier runs. So it can make a difference. This code was not implemented by me btw. Only moved probably

benjamc · 2025-03-27T14:21:55Z

smac/intensifier/multi_objective_intensifier.py

+    #     # TODO adjust
+    #     raise NotImplementedError
+
+    def _cut_incumbents(self, incumbent_ids: list[int], all_incumbent_isb_keys: list[list[InstanceSeedBudgetKey]]) -> list[int]:


add docstring

benjamc · 2025-03-27T14:24:10Z

smac/intensifier/abstract_intensifier.py

+    #     if len(previous_incumbents) == len(new_incumbents):
+    #         if previous_incumbents == new_incumbents:
+    #             # No changes in the incumbents
+    #             self._remove_rejected_config(config_id) # This means that the challenger is not rejected!!


remove comments

benjamc · 2025-03-27T14:24:52Z

smac/intensifier/abstract_intensifier.py

@@ -612,20 +634,279 @@ def update_incumbents(self, config: Configuration) -> None:
        # Cut incumbents: We only want to keep a specific number of incumbents
        # We use the crowding distance for that
        if len(new_incumbents) > self._max_incumbents:
-            new_incumbents = sort_by_crowding_distance(rh, new_incumbents, all_incumbent_isb_keys)


sort_by_crowding_distance got replaced by what?

benjamc · 2025-03-27T14:25:15Z

smac/intensifier/abstract_intensifier.py

        elif len(previous_incumbents) < len(new_incumbents):
            # Config becomes a new incumbent; nothing is rejected in this case
-            self._remove_rejected_config(config_id)
+            self._remove_rejected_config(config)


We are not working on ids? Why?

jeroenrook · 2025-03-27T15:06:43Z

Several pytest fail. All of them yield an KeyError: TrialKey in smac/runhistory/runhistory.py:1116. Likely this happens due to the budgets being overwritten to None in the ISB keys when new runs are added.

jeroenrook

Extra note: After merge add additional functions such as separate surrogate models and runhistories (log) for the different objectives.

jeroenrook · 2025-03-27T15:16:44Z

examples/2_multi_fidelity/1_mlp_epochs.py

+            objectives="accuracy",
+            # min_budget=1,  # Train the MLP using a hyperparameter configuration for at least 5 epochs
+            # max_budget=25,  # Train the MLP using a hyperparameter configuration for at most 25 epochs
+            n_workers=4,


jeroenrook · 2025-03-27T15:19:30Z

smac/acquisition/maximizer/local_search.py

@@ -123,7 +118,7 @@ def _maximize(
        # Sort according to acq value
        configs_acq.sort(reverse=True, key=lambda x: x[0])
        for a, inc in configs_acq:
-            inc.origin = "Acquisition Function Maximizer: Local Search"
+            inc.origin = "Local Search"


Fixed this already to make the pytests work again

jeroenrook · 2025-03-27T15:21:20Z

smac/acquisition/maximizer/multi_objective_search.py

+    def _create_sort_keys(self, costs: np.array) -> list[list[float]]:
+        """Non-Dominated Sorting of Costs
+
+         In case of the predictive model returning the prediction for more than one objective per configuration


Update text to comply with workings of function

jeroenrook · 2025-03-27T15:39:59Z

smac/acquisition/maximizer/local_search.py


        return init_points

+    def _create_sort_keys(self, costs: np.array) -> list[list[float]]:


This is to get points to perform the local search with based on earlier runs. So it can make a difference. This code was not implemented by me btw. Only moved probably

jeroenrook · 2025-03-27T15:41:18Z

smac/facade/algorithm_configuration_facade.py

@@ -115,7 +116,12 @@ def get_intensifier(
        max_incumbents : int, defaults to 10
            How many incumbents to keep track of in the case of multi-objective.
        """
-        return Intensifier(
+        class NewIntensifier(intermediate_decision.NewCostDominatesOldCost,


Combine with abstraction of the intensifier

jeroenrook · 2025-03-27T15:43:54Z

smac/intensifier/intensifier.py

+        config_hash = get_config_hash(config)
+
+        # Do not compare very early in the process
+        if len(config_isb_keys) < 4:


Empirically. But the mixing is likely to overwrite this function anyway

jeroenrook · 2025-03-27T15:46:28Z

smac/intensifier/mixins/update_incumbent.py

+        isb_keys = self.get_incumbent_instance_seed_budget_keys(compare=True)
+
+        n_samples = 1000
+        if len(isb_keys) < 7:  # When there are only a limited number of trials available we run all combinations


I believe 7 is the miminum number you need to have a least 1000 distinct samples.

Carolin Benjamins and others added 30 commits January 9, 2023 12:57

Add MO facade with todos

f6ee35b

Add NoAggregatuonStrategy

2b97fca

Update aggregation strategy

556ad37

Limit value to bounds region

672389f

This is a safety measure. Normally, everytime we update the runhistory, the objective bounds are updated so that the value to normalize should be inside the bound.

Factor out creating a unique list

09160b7

Created a helper method to create a set with preserved order from a list

More debug logging

1359f19

Factor out sorting of costs

733f94d

Previously: random scalarization of MO costs bc ParEGO was the only MO algo. Now: separate function which can be overwritten.

Better docstring

3e015c0

Add MO acq maximizer

171958b

Update acq optimizer

0fe8e7d

Stop local search after max steps is reached

0059155

Also, reset obtain to the kwargs (no magic numbers) Better debug message

Abstract away population trimming and pareto front calculation

5b0a1bf

Add MO intensifier draft

a0bed50

Add comment

325cb5c

Add todos

227ceb7

Pass rh's incumbents to acquisition function

c320f04

Add incumbents data structure in runhistory

67eefec

Add property for incumbents

b297a98

Updates incumbents of runhistory automatically if updated

Add EHVI acq fun

6042bed

Update PHVI

a96172d

Add ACLib runner draft

75a2077

Merge branch 'development' into mosmac

4b2d101

# Conflicts: # smac/acquisition/maximizer/local_search.py

Native objective support

a5902d5

Fix typo

5e7d880

Initial modifications for mo facade

3cdf96a

Make the HV based acquisition functions work

087d7c8

Logic fix

1b20106

AClib runner

a057733

AClib runner fixes

6c0bcd1

MO utils initial expansion

71409ce

jeroenrook added 22 commits March 28, 2023 10:59

Added a minimal number of configuration that need to be yielded befor…

a26b7c9

…e retraining and forces to run the maximization procedure immediately to time the computation cost.

Remove sleep call used for testing

37ae763

Only compute Pareto fronts on the same subset of isb_keys.

9b85222

Early termination of unfruitful configurations.

Compute actual isb differences

8c114c0

Aclib runner

2bc7383

Reset counter when retrain is triggered

6ddc94c

Comparison on one config from the incumbent

24a749f

Make dask runner work

944425b

Added different intermediate update methods that can be mixed with th…

8496461

…e intensifier class.

Make normalization of costs in the mo setting a choice

da0bb6b

Generic HVI class

603182a

Decomposed the intensifier decision logic and created mixins to easil…

a109f48

…y play around with different implementations.

Changed the intensifier

17ce0a3

Commit everythin

fd317b0

csvs

b50db2b

Merge remote-tracking branch 'origin/main' into mosmac

38b22d4

# Conflicts: # smac/acquisition/maximizer/local_search.py # smac/intensifier/abstract_intensifier.py

README change

69d466b

README change

fdd33f6

Even bigger push

bf2a2f0

Merge remote-tracking branch 'origin/development' into mosmac-merge

1d71cf4

# Conflicts: # README.md # smac/intensifier/abstract_intensifier.py # smac/utils/multi_objective.py

Remove EHVI acquisition function

7d7290d

jeroenrook commented Mar 27, 2025

View reviewed changes

smac/acquisition/function/expected_hypervolume.py

Copy link

Contributor Author

jeroenrook Mar 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rename file to hypervolume

jeroenrook commented Mar 27, 2025

View reviewed changes

jeroenrook added 2 commits March 27, 2025 10:40

README

aec7609

Fix failing tests. Disentangle normalisation and aggregation

cb9eab6

benjamc requested changes Mar 27, 2025

View reviewed changes

Fix failing pytests

373dc08

jeroenrook commented Mar 27, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MO-SMAC merge #1222

MO-SMAC merge #1222

jeroenrook commented Mar 27, 2025

jeroenrook Mar 27, 2025

jeroenrook Mar 27, 2025

benjamc left a comment

benjamc Mar 27, 2025

benjamc Mar 27, 2025

jeroenrook Mar 27, 2025

benjamc Mar 27, 2025

jeroenrook Mar 27, 2025

benjamc Mar 27, 2025

jeroenrook Mar 27, 2025

benjamc Mar 27, 2025

jeroenrook Mar 27, 2025

benjamc Mar 27, 2025

benjamc Mar 27, 2025

benjamc Mar 27, 2025

benjamc Mar 27, 2025

jeroenrook commented Mar 27, 2025

jeroenrook left a comment

jeroenrook Mar 27, 2025

jeroenrook Mar 27, 2025

jeroenrook Mar 27, 2025

jeroenrook Mar 27, 2025

jeroenrook Mar 27, 2025

jeroenrook Mar 27, 2025

jeroenrook Mar 27, 2025


		return init_points

		def _create_sort_keys(self, costs: np.array) -> list[list[float]]:

MO-SMAC merge #1222

Are you sure you want to change the base?

MO-SMAC merge #1222

Conversation

jeroenrook commented Mar 27, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benjamc left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jeroenrook commented Mar 27, 2025

jeroenrook left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment