Test and document usage with RandomizedSearchCV

[jnothman]

Do we want to overload the use of set_grid, or use set_rv? Perhaps we can draw parameters from distrib if available, and grid otherwise.

How do we deal with multiple grids when make_randomized_search is called? (raise error?)

Can we generalise this interface?

Much the same can be done for skopt.BayesSearchCV except there multiple grids are allowed, and may have a number of iterations attached to them. Not sure how to specify that with the current searchgrid api.

[betatim]

Much the same can be done for skopt.BayesSearchCV except there multiple grids are allowed, and may have a number of iterations attached to them.

Do you think the weighting of different options is useful/needed? IMHO we could solve this by implementing weights for Categorical dimensions (in skopt). Instead of having a list of options.

It would require us being able to have a "tree" like search space, with some dimensions only existing for certain values of other dimensions. With the current setup with a list you can do this for one "level", but more gets hard/impossible.

Have you looked at hyperopt's way of specifying a search space?

[jnothman]

Yes, I realise implementing weights in categorical dimensions is an option... as long as that categorical draw entailed an entire search subspace as you say. So really it's about a generative sampling model. Which, yes, is what Hyperopt allows you to do (as well as having arbitrary relationships and constraints between parameters).

I suppose I'd be able to express what I need to in BayesSearchCV if your list of tuple format allowed the second value in the tuple to be a float proportion rather than an integer count of iterations.

[jnothman]

Or perhaps I mean that the tuple notation could be used to express the prior distribution over those alternatives.

[betatim]

Dunno, I'm not a fan of the tuple notation. Our code to interpret it is a constant source of confusion and bugs. So I've started telling people to be explicit for anything but the most trivial cases...

[jnothman]

Does allowing the grid to be a categorical-like thing fix those issues?

…

On 11 August 2017 at 16:53, Tim Head ***@***.***> wrote: Dunno, I'm not a fan of the tuple notation. Our code to interpret it is a constant source of confusion and bugs. So I've started telling people to be explicit for anything but the most trivial cases... — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEz65FF3_VxU3J8fxHANFJIzT7loF_8ks5sW_pbgaJpZM4Outzp> .

[jnothman]

i.e. a distribution over spaces

…

On 13 August 2017 at 22:23, Joel Nothman ***@***.***> wrote: Does allowing the grid to be a categorical-like thing fix those issues? On 11 August 2017 at 16:53, Tim Head ***@***.***> wrote: > Dunno, I'm not a fan of the tuple notation. Our code to interpret it is a > constant source of confusion and bugs. So I've started telling people to be > explicit for anything but the most trivial cases... > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > <#2 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AAEz65FF3_VxU3J8fxHANFJIzT7loF_8ks5sW_pbgaJpZM4Outzp> > . >

[betatim]

Does allowing the grid to be a categorical-like thing fix those issues?

I'd phrase it as having a dimension like so Categorical([a_space, a_different_space, ...]). Is that what you mean? If yes, then I also think this is probably the way to go. Hard bit is figuring out how to make a regressor that is happy with that so we can have shared knowledge between the two sub-spaces.