Forbidden relations of conditioned hyperparameters

[bbudescu]

Description

Using a relational constraint on the values that can be taken by two hyperparameters, one of them not being guaranteed to be sampled every time, causes a crash either when adding HPs to the config space, or when instantiating the Facade and sampling the initial configs.

Steps/Code to Reproduce

from ConfigSpace import (ConfigurationSpace, Configuration, Categorical, EqualsCondition, ForbiddenEqualsRelation,
                         ForbiddenAndConjunction, ForbiddenEqualsClause)
from smac import Scenario, HyperparameterOptimizationFacade, MultiFidelityFacade

a = Categorical('a', [2, 5, 10], ordered=True)
enable_a = Categorical('enable_a', [False, True])
cond_a = EqualsCondition(a, enable_a, True)

b = Categorical('b', [5, 10, 15], ordered=True)
forbid_a_b = ForbiddenEqualsRelation(a, b)
# forbid_a = ForbiddenEqualsClause(enable_a, True)
# forbid_a_b = ForbiddenAndConjunction(forbid_a, forbid_a_b)

cs = ConfigurationSpace()
cs.add([a, enable_a, cond_a, b, forbid_a_b])

scenario = Scenario(cs, deterministic=True, n_trials=200)

def train(config: Configuration, seed: int = 0, budget: int = 0) -> float:
    pass

smac = MultiFidelityFacade(scenario, train)

I've found that enabling the extra forbidden clause forbid_a in the commented line above, and replacing the original forbid_a_b by a ForbiddenAndConjunction helps when using HyperparameterOptimizationFacade but not with MultiFidelityFacade

Expected Results

No crash.

Actual Results

File "playground/playground_smac_bug.py", line 32, in <module>
    cs.add([a, enable_a, cond_a, b, forbid_a_b])
  File "/home/bogdan/anaconda3/envs/smac/lib/python3.10/site-packages/ConfigSpace/configuration_space.py", line 351, in add
    self._check_default_configuration()
  File "/home/bogdan/anaconda3/envs/smac/lib/python3.10/site-packages/ConfigSpace/configuration_space.py", line 915, in _check_default_configuration
    return Configuration(self, values=values)
  File "/home/bogdan/anaconda3/envs/smac/lib/python3.10/site-packages/ConfigSpace/configuration.py", line 126, in __init__
    self.check_valid_configuration()
  File "/home/bogdan/anaconda3/envs/smac/lib/python3.10/site-packages/ConfigSpace/configuration.py", line 160, in check_valid_configuration
    check_configuration(
  File "/home/bogdan/anaconda3/envs/smac/lib/python3.10/site-packages/ConfigSpace/util.py", line 611, in check_configuration
    if forbidden.is_forbidden_vector(vector):
  File "/home/bogdan/anaconda3/envs/smac/lib/python3.10/site-packages/ConfigSpace/forbidden.py", line 683, in is_forbidden_vector
    return self.left.to_value(left) == self.right.to_value(right)  # type: ignore
  File "/home/bogdan/anaconda3/envs/smac/lib/python3.10/site-packages/ConfigSpace/hyperparameters/hyperparameter.py", line 358, in to_value
    value: DType = self._transformer.to_value(np.array([vector]))[0]
  File "/home/bogdan/anaconda3/envs/smac/lib/python3.10/site-packages/ConfigSpace/hyperparameters/hp_components.py", line 174, in to_value
    raise ValueError(
ValueError: Got unexpected float value while trying to transform a vector representation into a value in [ 2  5 10].Expected integers but got [nan] (dtype: float64)

Versions

2.2.0

LE: setting default=True on init of enable_a helps avoid the issue happening at ConfigurationSpace.add() when checking the default configuration, and postpones it until a bunch of configs are sampled as vectors for the initial design (during the Facade initialization), where nans cause issues.

[bbudescu]

I forgot to mention that when stepping through the code, the nan values are used as replacements for variables that shouldn't be always sampled, e.g., a is assigned nan when enable_a is sampled as False.

[bbudescu]

Is this the right place to post this issue? Since this crash happens in ConfigurationSpace.add() then, perhaps, it's better to post this issue in the ConfigSpace repo. I was in doubt because after uncommenting the two above disabled lines (i.e., we add an extra condition for a to be enabled) makes the crash occur within the MultiFidelityFacade construction instead.

Here's the output after uncommenting the two lines:

File "playground_smac_bug.py", line 40, in <module>
    sm = MultiFidelityFacade(scenario, train)
  File "/home/bogdan/anaconda3/envs/smac/lib/python3.10/site-packages/smac/facade/abstract_facade.py", line 213, in __init__
    self._update_dependencies()
  File "/home/bogdan/anaconda3/envs/smac/lib/python3.10/site-packages/smac/facade/abstract_facade.py", line 443, in _update_dependencies
    self._config_selector._set_components(
  File "/home/bogdan/anaconda3/envs/smac/lib/python3.10/site-packages/smac/main/config_selector.py", line 104, in _set_components
    self._initial_design_configs = initial_design.select_configurations()
  File "/home/bogdan/anaconda3/envs/smac/lib/python3.10/site-packages/smac/initial_design/abstract_initial_design.py", line 135, in select_configurations
    configs += self._select_configurations()
  File "/home/bogdan/anaconda3/envs/smac/lib/python3.10/site-packages/smac/initial_design/random_design.py", line 15, in _select_configurations
    configs = self._configspace.sample_configuration(size=self._n_configs)
  File "/home/bogdan/anaconda3/envs/smac/lib/python3.10/site-packages/ConfigSpace/configuration_space.py", line 624, in sample_configuration
    cond_forbidden |= clause.is_forbidden_vector_array(valid_configs)
  File "/home/bogdan/anaconda3/envs/smac/lib/python3.10/site-packages/ConfigSpace/forbidden.py", line 547, in is_forbidden_vector_array
    forbidden_mask &= forbidden.is_forbidden_vector_array(arr)
  File "/home/bogdan/anaconda3/envs/smac/lib/python3.10/site-packages/ConfigSpace/forbidden.py", line 689, in is_forbidden_vector_array
    return self.left.to_value(left) == self.right.to_value(right)  # type: ignore
  File "/home/bogdan/anaconda3/envs/smac/lib/python3.10/site-packages/ConfigSpace/hyperparameters/hyperparameter.py", line 356, in to_value
    return self._transformer.to_value(vector)
  File "/home/bogdan/anaconda3/envs/smac/lib/python3.10/site-packages/ConfigSpace/hyperparameters/hp_components.py", line 174, in to_value
    raise ValueError(
ValueError: Got unexpected float value while trying to transform a vector representation into a value in [ 2  5 10].Expected integers but got [nan  1.  2. nan nan  0.  1.  1.  2. nan  1. nan  1.] (dtype: float64)

LE: added an issue in the right place (automl/ConfigSpace#396). Should I move all contents there?

[bbudescu]

fixed by automl/ConfigSpace#397