new CCSA and MMA parameters: inner_gradients and always_improve

[stevengj]

This PR adds two new algorithm-specific parameters to the CCSA and MMA algorithms, one to control whether gradients are requested on inner iterations, and one to control how aggressive the algorithm is about updating the current point (see the new docs).

The default behavior is not changed, in order to not hurt existing code that uses these algorithms. For example, not computing the gradient on inner iterations actually hurts performance in the common case where it succeeds after a single inner iteration — since it has to then call the function again with gradients on the outer iteration — unless the caller optimizes their objective function (and constraints) to cache the current x point and not recompute the function (only compute the new gradient) if it is called twice in a row with the same x.

Closes #484. Closes #376.

[smartalecH]

Thanks @stevengj

Suppose we want to only perform one outer step per "stochastic sample" (and as many inner steps as needed to hit our convergence criteria.)

Is there an API to specify this behavior? In other words, can we specify the "max number of outer iterations"? We have a parameter for the max number of inner iterations, and all algorithms can specify a max number of function calls (which in this case includes outer and inner iterations). But I don't see a mechanism to specify a max number of outer iterations implicitly or explicitly.

[stevengj]

Good idea. It would be easy to add another algorithm parameter that makes the maxevals apply only to outer iterations.

Another option would be to count gradient evaluations an an eval, i.e. count function+gradient (outer iterations) as two evals and function-only (inner iterations) as one eval. (If you were benchmarking two algorithms against one another you would probably want to do this.) Or, equivalently (for better backwards compatibility), count function-only calls as half an evaluation.