Cave + BOHB: Error in fanova

[PhMueller]

Hey Joshua,

thanks for your work on Cave!

While using it, I have stumbled upon some errors related to BOHB and Categorical HP. I am pretty sure they are not super relevant and I was able to find a workaround. But I thought it might be useful for you.

My ConfigSpace contains a categorical hyperparameter so running

cave $(pwd) --output cave_output_dir

threw the error which is described in #192

Full Error Message

NFO:fanova.visualizer.Visualizer:creating cave_report_2/analysis_data/-2146368420622879532/fanova/lr_weight_decay.png
ERROR:cave.cavefacade.CAVE:Error in fANOVA - please run with --pimp_no_fanova_pairs (this might be due to a known issue with ints and bools in categorical hyperparameters, see issue #192).
Traceback (most recent call last):
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/cave/cavefacade.py", line 324, in cave_fanova
  fanova = Fanova(self.runscontainer)
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/cave/analyzer/parameter_importance/fanova.py", line 35, in __init__
  self.parameter_importance("fanova")
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/cave/analyzer/parameter_importance/base_parameter_importance.py", line 60, in parameter_importance
  'bokeh': components(run.pimp.evaluator.plot_bokeh(show_plot=False))
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/pimp/evaluator/fanova.py", line 265, in plot_bokeh
  mean, std, grid = vis.generate_marginal(param_name, 100)
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/fanova/visualizer.py", line 276, in generate_marginal
  (m, v) = self.fanova.marginal_mean_variance_for_values(dim, [grid[i]])
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/fanova/fanova.py", line 367, in marginal_mean_variance_for_values
  sample[dimlist[i]] = valuesToPredict[i]
IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/cave/cavefacade.py", line 55, in wrap
  analyzer = f(self, *args, **kw)
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/cave/cavefacade.py", line 327, in cave_fanova
  raise IndexError("Error in fANOVA - please run with --pimp_no_fanova_pairs (this might be due to a known "
IndexError: Error in fANOVA - please run with --pimp_no_fanova_pairs (this might be due to a known issue with ints and bools in categorical hyperparameters, see issue #192).
Traceback (most recent call last):
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/cave/cavefacade.py", line 324, in cave_fanova
  fanova = Fanova(self.runscontainer)
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/cave/analyzer/parameter_importance/fanova.py", line 35, in __init__
  self.parameter_importance("fanova")
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/cave/analyzer/parameter_importance/base_parameter_importance.py", line 60, in parameter_importance
  'bokeh': components(run.pimp.evaluator.plot_bokeh(show_plot=False))
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/pimp/evaluator/fanova.py", line 265, in plot_bokeh
  mean, std, grid = vis.generate_marginal(param_name, 100)
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/fanova/visualizer.py", line 276, in generate_marginal
  (m, v) = self.fanova.marginal_mean_variance_for_values(dim, [grid[i]])
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/fanova/fanova.py", line 367, in marginal_mean_variance_for_values
  sample[dimlist[i]] = valuesToPredict[i]
IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/muelleph/miniconda3/envs/operation/bin/cave", line 8, in <module>
  sys.exit(entry_point())
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/cave/cave_cli.py", line 372, in entry_point
  cave.main_cli()
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/cave/cave_cli.py", line 297, in main_cli
  cave.analyze()
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/cave/utils/timing.py", line 11, in wrap
  result = f(*args, **kw)
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/cave/cavefacade.py", line 233, in analyze
  self.parameter_importance(self._get_dict(self.website, "Parameter Importance"))
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/cave/cavefacade.py", line 353, in parameter_importance
  self.cave_fanova(d=d)
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/cave/cavefacade.py", line 55, in wrap
  analyzer = f(self, *args, **kw)
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/cave/cavefacade.py", line 327, in cave_fanova
  raise IndexError("Error in fANOVA - please run with --pimp_no_fanova_pairs (this might be due to a known "
IndexError: Error in fANOVA - please run with --pimp_no_fanova_pairs (this might be due to a known issue with ints and bools in categorical hyperparameters, see issue #192).

However, using the parameter --pimp_no_fanova_pairs did not change anything.
Skipping the entire fanova package has worked. (using the cmd line argument --skip fanova)

I did a clean install of CAVE with (pip install --upgrade cave):
cave==1.4.0
PyImp==1.1.2

example_files.zip

[shukon]

Thanks for reporting. I'll probably have capacities to look into this in the second half of March (sorry, there is a number of deadlines before), maybe I can help out then.
In the meantime if it's important to you (or someone stumbling upon the issue), it's probably not due to #192 since you're using .json-files. You can still try to cut out individual parameters (maybe the none value of the transform parameter? Wouldn't know why but then again I'm currently not sure where the parsing happens and how), especially categorical hyperparameters were prone to break stuff in fANOVA iirc...