Workaround for undeclared `conda` dependency?

[bollwyvl]

Description

conda-recipe-manager has an undeclared, unbounded dependency on conda for its MatchSpec implementation, and the conda code organization makes it difficult to patch around heavy binary dependencies.

This makes it hard to bootstrap crm in lightweight/automation environments that don't fully support the conda ecosystem (or vice-versa conda-forge/conda-forge.github.io#2244).

At present, there is at least one alternate MatchSpec implementation from the conda top-level org, namely py-rattler.

Additional context

Discussed on:

• Make it easier to run CRM in a headless environment #337: heavy binary deps like graphviz and matplotlib have been removed
• Publish on PyPI? #469: conda is not distributed on PyPI, which generally makes it harder for Python-based downstreams such as grayskull to confidently depend on conda-recipe-manager

Design Ideas

• A relatively simple approach with import checks:

Details

• Move all imports of MatchSpec to an import of a local new matchspec.py or conda_compat.py
• either with ImportError (or slightly more efficient find_spec) checks, determine if conda is available
  • if not available, try finding alternate implementations
  • if any are found, provide that implementation

• A more scalable approach with entry points:

Details

• in pyproject.toml, declare a new entry_point, e.g. conda-recipe-manager.v0.matchspec with an initial implementation (similar to the above)
• declare an interface contract (potentially with abc, maybe just type hints) which includes an is_available check
• probably again in something like conda_compat.py, wrap the conda implementation in something that conforms and advertise this in pyproject.toml

[project.entry_points."conda-recipe-manager.v0.matchspec"]
conda = "conda_recipe_manager.conda_compat.matchspec:CondaMatchSpec"

• use entry_points(group="conda-recipe-manager.v0.matchspec") to find an available one when requested
• define [optional-dependencies] for different providers
  • if only for reference purposes, having the lower bound of conda would probably be good

This would allow third-party packages to add new implementations, either directly, or via some plugin package, e.g. crm-py-rattler, e.g.

[project.entry_points."conda-recipe-manager.v0.matchspec"]
py-rattler = "crm_py_rattler.matchspec:PyRattlerMatchSpec"

In either event:

• add an environment variable, CLI flag, or other means to force a specific implementation
• if no implementation is found, add a helpful message and raise a dedicated exception type
  • in the CLI, probably don't show a full traceback

[schuylermartin45]

conda-recipe-manager has an undeclared, unbounded dependency on conda

Unbounded, yes. Undeclared, only if you are talking about the sad state of our pyproject.toml. I will admit is not up to date with the meta.yaml and environment.yaml files, but both of those include conda.

Personally, I'm not a huge fan of dynamic imports. I'd rather we have one solution that is well-tested for everyone. I've found that that usually provides a better user experience.

I'm especially concerned by just how umm...for lack of a better term "notorious" the conda::MatchSpec implementation is. I worry if we support both, there will be difficult to detect differences in the feature sets that may cause painful runtime issues.

I picked conda::MatchSpec as it was the "canonical" option for the ecosystem at the time.

Let me throw out 2 other approaches:

1. We move to use py-rattler::MatchSpec full-stop.
I think this is on the table as rattler is a Conda-org project. I admittedly have not looked at its implementation so I don't know if there are any pitfalls in hard-pivoting to using it.

As much as I subscribe to the book of "don't break the user", realistically, I don't think there are a lot of projects out there that use CRM's dependency work. I get the impression the broader OSS/Conda Community sees CRM only as "the thing that converts V0 to V1".

CRM powers a number of Anaconda projects, but I'm more than comfortable fixing those. I just happen to know the author of those projects.

2. We talk to the conda Team and see if we can break-out MatchSpec
I'm sure CRM is not the only project that includes conda to use MatchSpec. This is probably a more difficult ask than it sounds, but maybe there's an opportunity to decrease dependency monoliths for other Conda Community projects(?)

[bollwyvl]

Hooray, a pure python matchspec (PyPI available) in a separate package would indeed be an ideal outcome, which I think we mentioned on one of the earlier discussions. I've definitely used other extracted features like conda-index in isolation.

pitfalls

Yep; different things are always different. An even more basal, implementation-agnostic EBNF and dumb-text-files-based conformance suite or something that could be the "final exam" for CEPs would be an ideal outcome, and different implementations could test against that.

Shucks, different implementations could even be pitted against each other with e.g. hypothesis.

the thing that converts V0 to V1

Indeed, that's what I mostly want.

[schuylermartin45]

TIL the "break-up conda into a library" idea has had some traction in the past and may be up for debate again soon: conda/conda#13789

[maresb]

Dropping by from conda/conda-lock#899. I've been on a long battle to tame the madness in the conda-lock codebase. We solve the MatchSpec problem by vendoring Conda. This is not ideal, is a huge effort to maintain, and explodes our dependencies. (It's not our only such problem, since we also vendor Poetry 🙈.)

It would be amazing for us to have a simpler way to depend on MatchSpec. The idea of switching to py-rattler::MatchSpec is very tempting. It'd increase my confidence a lot to see both implementations rigorously tested against each other. Thanks @bollwyvl for always nudging us in the right direction.