Incompatible with most of conga-forge ecosystem now

[timkpaine]

we really need to finish: #217
as we are now broadly incompatible with the entire conda-forge ecosystem which has long-since moved on.

xref: #90

[h-vetinari]

ray-project/ray#51673 was closed, let's hope ray-project/ray#58661 is more successful

[aslonnie]

ray-project/ray#51673 was closed, let's hope ray-project/ray#58661 is more successful

reading #90, I have a question:

is enabling using local libs to compile required? or is it that upgrading the library usages in ray upstream is already enough?

[h-vetinari]

is enabling using local libs to compile required?

If ray can be built without the presence of abseil/protobuf etc., then of course you would not inherit the constraints that arise from compiling against a given version (though ray would have to make sure that the vendored symbols are not visible in its own library, otherwise things might easily blow up).

I don't know ray's build well enough to know if that's feasible (and it is indeed the case that conda-forge prefers not vendoring libraries), but I'm assuming it's not, or at least not easy?

[aslonnie]

I don't think ray exposes any symbol. ray is built as a python package. the libraries that are vendored are statically linked. this is required by the manylinux standard.

maybe I missed something or misunderstood something, could some one explain to me in detail where the "incompatible with most of conda-forge ecosystem" comes from exactly? like what specifically will go wrong in conda-forge world with having a copy of some c++ code (and with some custom patches) and build it into ray package if it is all ray internal implementation?

[aslonnie]

cc @mattip

[aslonnie]

after some quick research, it seems that the root of issue is that conda-forge recommends dynamic linking even for python packages, and conda-forge grpcio package is dynamically linking grpc++

[aslonnie]

I feel that the main thing that ray needs to do is to hide its symbols.

and dynamically linking libraries like grpc++ in conda-forge world seems nice to have but optional?

[h-vetinari]

maybe I missed something or misunderstood something, could some one explain to me in detail where the "incompatible with most of conda-forge ecosystem" comes from exactly?

Because ray currently depends on libgrpc

ray-packages-feedstock/recipe/recipe.yaml

Line 98 in d14673d

- libgrpc

which adds constraints to ray's runtime dependencies through its run-export. In turn, any given grpc version is tightly coupled with some abseil & protobuf version (due to the way these packages usually get migrated together).

like what specifically will go wrong in conda-forge world with having a copy of some c++ code (and with some custom patches) and build it into ray package if it is all ray internal implementation?

It can conceivably work, but aside from being fraught technically, it's also not a sustainable practice on an ecosystem level. If common libraries like zlib, fmt, zstd, abseil etc. get vendored and then revendored, and nestedly revendored, you end up with a huge mess. This is why conda-forge very strongly demands dynamic linking in general. Definitely more than "nice to have but optional".

Ray could claim an exception here, but historically this has not worked well (c.f. #136, ray-project/ray#35383). It's not just about hiding the symbols, but also about matching the actual ABI of packages, and that may not be the case if ray built with vendored grpc gets used with a grpcio that uses a different abseil/grpc/protobuf version resp. ABI under the hood, for example.

The real culprit here is bazel. From a compiler/linker perspective it's trivial to link against an existing grpc install, but bazel refuses to work with anything it doesn't explicitly control (not counting completely insane work-arounds that take 1000+ lines to link to a single library). Hermetic builds and aggressive caching are nice, but completely antithetical to distribution (the packages we publish are our cache). However, that's not a problem google faces itself, so chances aren't great that this will change soon or at all. Projects using bazel suffer from this, especially if they have complicated dependencies. The solution is to implement escape hatches like bazelbuild/bazel#20947 in bazel, or (more likely) to switch to another build system.

[aslonnie]

The real culprit here is bazel.

fwiw, the bazel issue can be worked out. it might need a lot of efforts upstream, but it is technically possible.. for example, when compiling _raylet.so, the python headers are actually always from the systems, rather than from source.

ray currently depends on libgrpc

that seems like a declaration of a build-time dependency? ray upstream building does not seem to need libgrpc from the systems? why is it required in conda-forge?

It's not just about hiding the symbols, but also about matching the actual ABI of packages

from my point of view, the more concerning issue is the divergence from upstream. ray's main use case is not through conda/mamba install from conda-forge's repos, but pip install from pypi, and for pip install users, they do not need to install libgrpc++ onto their systems first, and ray upstream will never demand users to install the right version of libgrpc++ onto their systems first. ray upstream is designed to be manylinux conformant. it is currently on manylinux2014, which says that the binary bits cannot dynamically link to libraries other than the designated list. future manylinux ones are less strict, but no matter what, libraries like libgrpc++ cannot be dynamically linked in a standard python wheel. one cannot expect a common linux distro can ship with libgrpc++ by default.

honestly, I feel that applying https://github.com/conda-forge/cfep/blob/main/cfep-18.md to python packages is a misread.. I understand the pros and cons stated in the doc, but it is not going to be easily sustainable, as it is basically fighting against python binary wheels packaging standards.

[h-vetinari]

but it is not going to be easily sustainable, as it is basically fighting against python binary wheels packaging standards.

The conda ecosystem has been doing it for 10+ years. Wheels are strictly inferior as a technology, because in order to make them capable enough to do what conda does, you'd have to standardize a lot of things around build systems, compilers etc., which not even the C/C++ ecosystems ever got sorted out, and python people generally didn't want to touch with a 10-foot pole. Wheels are the result of this compromise - just stuff everything you need in there, so that you can ship anything precompiled at all. It was the best that could be achieved for overall python-land, but it would be impossible to run conda-forge this way. There's a whole host of problems that come with wheels and PyPI, so the fact that we're doing this integration work is what enables a bunch of important use cases in the first place.

So it's exactly the other way around. It especially applies to python packages, which shouldn't vendor libraries we already provide in high quality out of the box. You cannot expect a vanilla user system to have libgrpc (etc.), but you can do so for conda-forge.

Again, this is not a matter of religious strictness - if there are strong technical reasons, static libraries can be an acceptable compromise. But even then, ray has shown itself susceptible to breakage in such situations, so this needs thorough testing.

The real culprit here is bazel.

fwiw, the bazel issue can be worked out. it might need a lot of efforts upstream, but it is technically possible.

I'd be thrilled if you can pull this off. So far the gap between "technically possible" and "lots of effort" has never been closed, except in specific examples like the commit of the protobuf-feedstock I linked (which itself was piggybacking off of what tensorflow provides). Matti tried very hard too. I don't doubt that it's possible, but it's a non-trivial effort.