Add `numba` and `llvmlite` `musl` wheels for `alpine` support

[mpierrau]

Feature request

Hi, I was wondering if it is possible to add MUSL wheels for numba and llvmlite since lots of people use CICD images based on Alpine. This distribution uses different a implementation of libc: musl (see Alpine Linux has switched to musl libc | Alpine Linux ), and is incompatible with manylinux wheels.

I know that for example the shap library is waiting for llvmlite and numba to release MUSL wheels.

Thanks!

[esc]

@mpierrau quick question about this: where are they distributed, can they be uploaded to PyPi too? How many artifacts are we talking about here?

[esc]

@mpierrau we just discussed this during our issue triage and it is unlikely that this will happen anytime soon due to a maintainer resource shortage. Would suggest to leave this open anyway, in case that resources become available over the coming months or years.

[mpierrau]

I have no experience from distributing Python packages, but from what I can gather, the musl wheels are also uploaded to PyPi, and it's one wheel artifact per architecture you wish to support. The number of architectures supported for musl are in general the same as for manylinux (from looking in my lockfiles for other packages which do distribute musl wheels to PyPi).

It looks like cibuildwheel supports building musl wheels for CPython 3.6 - 3.12, for the following architectures x86_64, aarch64, i686, s390x, ppc64le.

[esc]

@mpierrau thank you for digging into this, much appreciated.

Thoughts:

a) We have space issues on PyPi with llvmlite. Our wheels contain LLVM itself and we regularly run out of space on PyPi because of this. Adding another 4 (or 8?) wheels per release may increase the size requirement so we'll probably need additional checks before uploading, if there is enough space left. This isn't something that many or even most Python packages face so tooling here -- for example to check for space -- is very rudimentary and sometimes not reliable. We usually only notice that we run out of space when a release fails to upload and then need to request more space on PyPi which can sometimes take several days to process from the PyPi side.

b) As you have noticed, llvmlite is quite unique in how it must be built, hence we don't (and maybe even can't) use cibuildwheel but instead have a very custom solution consisting of several bare metal machines that we maintain ourselves. Thus, adding musl wheels will require significantly more custom tooling and work than for an average package. It's not known how long doing this will take and may include multiple rabbit holes. I would estimate it takes between 3-4 weeks of developer time, so this is in the "nice to have" category for me as you have been the only person asking about this so far.

Of course, this could receive a higher priority if there is a significant ask (hair on fire need) for this from the community.

[mpierrau]

@esc Thanks for the reply and the insight to your situation. It is indeed a "nice to have", and my hair currently isn't on fire, so I'll patiently wait and come back to you with a bump if it spontaneously ignites in the future :)

[esc]

@esc Thanks for the reply and the insight to your situation. It is indeed a "nice to have", and my hair currently isn't on fire, so I'll patiently wait and come back to you with a bump if it spontaneously ignites in the future :)

🔥🔥🔥

[esc]

@esc Thanks for the reply and the insight to your situation. It is indeed a "nice to have", and my hair currently isn't on fire, so I'll patiently wait and come back to you with a bump if it spontaneously ignites in the future :)

🔥🔥🔥

No, but seriously, this is laborious and may come with challanges, but not technically impossible or not feasible. Maybe as musl wheels become more widely used, well see a need to have them for Numba too.

PS: do you know if NumPy provides these already?

[mpierrau]

No, but seriously, this is laborious and may come with challanges, but not technically impossible or not feasible. Maybe as musl wheels become more widely used, well see a need to have them for Numba too.

My impression from lurking in some major package issue boards is that the requests for musl wheels are gaining traction, but I don't have the pulse on whether it's requested by your community.

PS: do you know if NumPy provides these already?

NumPy started providing musl wheels in their 1.25.0 release (June 2023), this is an issue where some pro's and con's were discussed, especially this comment had some good points:

Is there enough use of musl to justify it?

I'd say yes. Musl is used by Alpine Linux, which is quite popular for use in Docker containers, because of the small-sized images it produces (thanks to Musl being so much smaller than Glibc). We get regular requests, probably more than for any other platform. And several Pandas and SciPy devs seem to want this.

Can pip can distinguish the linked libraries?

Yes, there is a separate musllinux wheel spec.

Does this affect OpenBLAS maintenance?

Only insofar as a separate build of OpenBLAS must be done for this. Which is available - hopefully maintenance of that will be smooth sailing.

Do downstream projects also need to link with musl?

Yes - but a separate wheel tag and Linux distro takes care of that. However, what you may be getting at is that one needs a complete collection of musllinux wheels for all the packages one needs. Patchy support isn't very useful. So it'd be good to see at least Pandas, SciPy, scikit-learn, and Matplotlib wheels. My impression is that that will happen if NumPy leads the way.

Does the MIT license add anything?

No worries there, all good on the licensing front.

Could static linking be useful?

That's actually a common reason for using Musl too. It's not useful for NumPy I'd say, but in general static linking with Musl is a great way to produce binaries that run on any distro (because no dependency on the distro-shipped libc).

At some point CI stability might become a problem.

That's always a worry, but more so for nonstandard hardware than for something that runs on plain Linux x86-64 and aarch64 instances. So for Musl it should be pretty good compared to our other candidates for inclusion in our zoo of wheels.

In addition, it looks like Pandas are also considering it, but are waiting for PyArrow, whom
are working on it.

I asked the same question in scikit-learn and they replied that:

For the record this was discussed at yesterday's monthly dev meeting and for now the consensus seems to be that the extra maintenance effort and CI costs are not necessarily worth it.
[...]
the maintainers might reconsider that decision (not to provide musllinux wheels on pypi.org) based on expressed user feedback e.g. next year.

SciPy are supporting providing them since this fall and so is matplotlib (just to give you an idea).