Add `learning_rate` hyper-parameter

[ablaom]

Currently, learning rates are buried in the optimiser provided to a model. This is awkward from the point of view of tuning the learning rate. One would like to specify a learning rate range like this,

option 1 MLJ.range(mljflux_model, :learning_rate, lower=1e-6, upper=1e-3, scale=:log10)

but instead is limited to

option 2

optimisers = [Adam(10^i) for i in -6:-3]
MLJ.range(mljflux_model, :optimiser, values=optimisers)

This is limiting for many reasons. Here are two:

• In option 2 I can't randomly search the entire continuous range I'm interested in (sample from the uniform distribution, on the log scale, as RandomSearch will do in option 1

• Plots don't get done on a log scale without intervention, in option 1, and the hyper-parameter labels mess up the standard plot(tuned_machine) output.

One solution is to add learning_rate as a new hyper-parameter with the understanding that, when specified, the specified optimiser is replaced with a copy in which the learning rate (first positional argument in most cases??) is changed to learning_rate.

Thoughts anyone?

[ablaom]

cc @LucasMatSP

[LucasMatSP]

For now, MLJFlux.jl supports any rule from Optimisers.jl, right? I wonder if the solution presented might restrict that, namely OptimiserChain and decays. Also, couldn't Optimisers.adjust! (link) be used for this under the hood?

[LucasMatSP]

Regardless of the solution we choose, we can't forget to document this clearly. Maybe even with some type of 'warning' block