Guidance on cache management

[johannahaffner]

Thanks for the great interface! It is really useful to me.

I'm trying to cache the pycutest problems during a docker build so I can speed up tests against pycutest. (I'm porting problems to JAX and use pycutest as the ground truth.) I'm running into memory issues though: the docker build breaks with an error that says that it cannot allocate memory, even if I am very strict and only select problems with up to ten variables, for example.

How large do you expect the pycutest cache to be for the full suite of CUTEst problems? Which parts and problems tend to be the most expensive? I noticed that some problems have extensive data, and I tried to restrict this by selecting only the academic ones, but this did not work either.

FWIW building the container does work if I don't try to cache anything.

[jfowkes]

Good question, it really depends on the problem. Some problems have rather large external data, even academic ones, for example some fitting problems that fit to a large number of points. This is one of the reasons we don't precompile all the problems into one large library, the other being that most problems have options to select variable number of dimensions. I believe @ragonneau has precompiled all the CUTEst problems so will hopefully be able to advise on how large the cache is.

[johannahaffner]

Thank you! Based on my preliminary experience building the container I expect it to be a few GB? (If indeed this is the cause of the memory failures I get when trying to precompile all of the problems.)

If so, then this would preclude the use of the container in a CI setting, and I think in that event I would anyway be fine tabling this issue for now.

Thanks for the advice regarding problem classification and problem size! This does make a lot of sense. I noticed that there are some problems with a lot of data attached, like the BRAIN series.

Have a good week! And thank you again for the excellent library.

[jfowkes]

Yes that sounds about right to me in terms of size, there are a lot of problems in CUTEst as it has been around for a long time!