Re-implementing ALEBO in BOTorch

[shaheenahmedc]

Hi All,

Recently, I've been trying to re-implement ALEBO in BOTorch, to give me greater control over each step in the BO loop.

I now have a working prototype (after following the BOTorch tutorials), but I can't seem to get comparable performance to the ALEBO implementation in AX.

Here is a link to my implementation:
https://github.com/shaheenahmedc/ALEBO_BOTorch/blob/main/ALEBO%20re-implementation%20-%20BOTorch%20.ipynb

I'm testing performance on the Branin_100 function, which I've inherited from botorch.test_functions.synthetic.SyntheticTestFunction. Details are in the attached notebook.

While ALEBO can reach values of ~ 2 on Branin_100 in around 20 sample points (5 random, 15 sequential), my implementation usually only reaches values of 40 or so, for the same number of iterations.

Also, I run into memory issues sooner with my implementation. At 70 sample points (50 random, 20 sequential), the ALEBO AX implementation works great, with no memory issues, but my implementation asks for too much memory, and breaks (around the alebo_acqf_optimizer usually, if not always).

My questions are:

1. Have I re-implemented ALEBO correctly in BOTorch, in the above notebook?
2. Can you see why I may not be getting the same performance as ALEBO in AX?
3. Is there any reason I'm running out of memory sooner than the AX implementation?
4. Have I used the ALEBOInitializer , ALEBOGP, get_fitted_model, alebo_acqf_optimizer from the ALEBO AX code correctly?
5. Are there any other parts of the ALEBO AX code which you think could help in this BOTorch re-implementation?
6. I can sometimes get Cholesky errors. I tried increasing the noise to 1, which seems to catch most of them. Can you see any reason why my implementation might be causing these?
7. Do I need to normalise the outputs, as done in the notebook?

Perhaps the notebook can help others who are looking to implement embedding methods for HDBO in BOTorch. Keen to get feedback on this if so.

System Information:
BOTorch = 0.3.3
GPyTorch = 1.3.1
Torch = 1.7.1
Windows 10

Best wishes,

Shaheen

[Balandat]

Hi @shaheenahmedc, I haven't fully digested the example, but it looks like you are currently not normalizing the inputs domain to the unit hypercube? This is something that we typically do in BoTorch since the priors of our standard models are tuned to work well for that scale. You can see that Ax is doing this internally via the CenteredUnitX transform: https://github.com/facebook/Ax/blob/5df9de4a3a300e3e88a5c44e781e902ce94adbfa/ax/modelbridge/strategies/alebo.py#L27.

if you normalized your domain to [0,1]^d in BoTorch this should help a lot. It probably will also help mitigate / get rid of the numerical issues as you won't end up with crazy hyperparameters anymore.

As to the memory issues: ALEBO in Ax uses 10 random restarts by default: https://github.com/facebook/Ax/blob/5df9de4a3a300e3e88a5c44e781e902ce94adbfa/ax/models/torch/alebo.py#L644, which is half of the number you're using. This could contribute to that. What kind of machine are you running this on?

[shaheenahmedc]

Hi @Balandat, thanks kindly for the above comments!

My apologies for the late reply also.

Your comment regarding normalizing the input domain helped me spot an error in my implementation. I wasn't transforming the sample points, generated from ALEBO_Initializer.gen(), into the ranges for the Branin_100 function.

I'm working through the improvements now, but I think this has helped a lot. I'll try to upload the results next week.

I'd like to get the input normalization right too however.

If I'm not mistaken, does ALEBO require the input domain to be normalized to [-1,1]^D, with D being the original input dimensionality? In which case, would you still advise normalising sample points generated with ALEBO_Initializer.gen(), from [-1,1]^D to [0,1]^D?

With regards to new sample points, generated from alebo_acqf_optimizer, are they generated in [0,1]^d, if [0,1]^d are the bounds of the embedding space in ALEBO? (I could be wrong on this).

Reducing the random restarts also looks to have helped memory issues. I'm using a Ryzen 7, 16 GB RAM laptop.

Thanks again for the help.

[Balandat]

Glad my previous comment was helpful.

If I'm not mistaken, does ALEBO require the input domain to be normalized to [-1,1]^D, with D being the original input dimensionality? In which case, would you still advise normalising sample points generated with ALEBO_Initializer.gen(), from [-1,1]^D to [0,1]^D?

If ALEBO works in [-1, 1]^D I would just stick with that. Is seems like the bounds input to alebo_acqf_optimizer is actually just being ignored right now. I believe this is b/c we ensure upstream in the Ax model stack that the bounds are indeed [-1, 1]^D.