Better handling of n_retries

[hadarshavit]

Related issues: #1088 #1086 #1118

Better handling of n_retries: if the first n_retires configs returned from the acq maximizer were already sampled, sample random configs instead, and check that those configs were not sampled before. If this also fails, then simply sample random configuration without checking if they were sampled.

This handling is inspired by HEBO: https://github.com/huawei-noah/HEBO/blob/2cce0838dc16b53aa8955f3ead025268ddd18a00/HEBO/hebo/optimizers/hebo.py#L170

[daphne12345]

Hello @hadarshavit, thank you for the effort! It makes sense to me to sample a random configuration if n_retries is up but I don't understand the second part. Why do you want to run a configuration without knowing if it has been processed before? Isn't it likely that you sample a configuration that has already been run before?

[hadarshavit]

Hi @daphne12345, the idea is that if the user still wants to run a configuration (for example if there is a call to ask), and all other configurations were already processed, SMAC should still returns some configuration (because the user asked for one). This is the same mechanism HEBO uses, but also raising an exception can make sense in such case.

[benjamc]

Hi Hadar,
we are still stumbling about the second block. Assuming we tried every configuration possible, shouldn't we then stop the optimization? I agree that raising an exception could be the way to go here.
(The first block absolutely makes sense)

[hadarshavit]

I added an exception in case even the random sampling fails and a test for this case.

[daphne12345]

Hello Hadar, thank you for the update. I'm wondering why the ConfigurationSpaceExhautionException is only raised if random_configs_retries < self._max_new_config_tries. Should it not be raised in case we reach the max_new_config_tries also with the random_config_retries?