[feature] Improve processing time of compatible packages

[ohanar]

What is your suggestion?

While testing out the new compatibility.py functionality in conan 2, I've found that the processing of compatible configurations is very slow. With my company's setting customization and notions of compatibility, we have something like ~1k compatible packages (cartesian products get large quickly), which ends up with minutes of processing time for a single package (should the binary be missing).

This seems to be mostly an issue of using many small queries to the server -- replacing a nearby conan repository with one across a major ocean has a dramatic impact. My guess is that the solution is to do a large batch request to the server, and then process the response locally.

Until this is resolved, we are resorting to hard-coding settings in profiles, like we did with conan v1.

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[memsharded]

Hi @ohanar

Thanks for the suggestion.
This is a known limitation of the client-server API, and we plan to improve it. But it requires changing the protocol and the servers, probably a new version of the API, and that is a ton of work that will require effort and time to be implemented in the server side.

At the moment, the recommendation is to keep the compatible packages limited to a few possibilities, like the default cppstd compatibility.py, or yes, hardcoding it in the profiles. Looking for thousands of combinations will not scale.

Assigning this to the 2.X roadmap milestone, but as commented above, it will take time.

[memsharded]

This already got an improvement on speed on Conan 2.0.13:

• Improved the http rest api speed (a lot, like reducing times in half)
• Caching the compatibility results (prioritizing the cache over remotes to lookup for compatibility), resulting in immediate resolution time when dependencies are already installed. This improvement is very useful specially for developer flows.