Speedup wheels builds by using pysam wheels

[nh13]

I am not sure why we're compiling a pysam wheels from source: https://github.com/fulcrumgenomics/pybwa/actions/runs/13866976722/job/38807978316

My wheels understanding is not deep enough, but could it be related to: pysam-developers/pysam#1132 (comment)

Calling @jmarshall: "Help me, Obi-Wan Kenobi, you're my only hope"

[jmarshall]

You are correct that on x86-64 manylinux you're having pretty much exactly the same problem as in that comment.

Pybwa's .github/workflows/wheels.yml uses CIBW_MANYLINUX_X86_64_IMAGE: manylinux2014, so your x86-64 builds cannot use pysam's manylinux_2_28 wheels (because that is a newer baseline requiring more recent pip, glibc, etc than manylinux2014 implies) hence must build pysam from the sdist. Pysam has never built manylinux2014 wheels — from 0.17.0 we built manylinux_2_x for variously x = 5, 12, 17, and 24. When I revamped the wheel building for 0.22.0 I continued that pattern with the then-supported (and still currently-supported) manylinux_2_28. We do build one Python 3.8 x86-64 manylinux2014 wheel as described in pysam-developers/pysam#1132 (comment). So it could be instructive to add Py3.8 to your build matrix or to use CIBW_MANYLINUX_X86_64_IMAGE: manylinux_2_28 and observe that these builds now use pre-built pysam wheels.

We try not to build an overwhelming number of pysam wheels, so have to date been reluctant to build manylinux2014 wheels in general as well. Especially as we will soon drop Python 3.6 and 3.7, that might be revisited but pysam has not had many reports of difficulties caused by the lack of manylinux2014 wheels.

Pysam does not build musllinux wheels at all, so your musllinux builds are also building pysam themselves. I considered building musllinux wheels when revamping this in 0.22.0, but we had not previously built them, no-one had asked for them, and I didn't want to inflate the number of wheels being built. This should be revisited, and pysam is much more likely to start building musllinux wheels than manylinux2014 wheels.

Pybwa's workflow uses CIBW_MANYLINUX_AARCH64_IMAGE: manylinux_2_28 just like pysam, so your aarch64 manylinux builds do just download a pre-built pysam wheel. Similarly your macOS builds just download a pre-built pysam wheel.

So there are two categories of build here that are not finding a pre-built pysam wheel to download:

• The musllinux builds. The way to fix this will be for pysam to start building musllinux wheels. Thanks for raising this; I'll aim to turn this on in the next pysam release.

• The manylinux2014 builds. You might consider joining pysam living in the future and using manylinux_2_28 on x86-64, as you already do on aarch64. Note that the cibuildwheel default will likely move in this direction (see Update default image away from manylinux2014 pypa/cibuildwheel#1772), though the timing is not yet determined. (Alternatively I suppose pysam could consider adding another five wheels by building x86-64 wheels for both manylinux_2_28 and a legacy manylinux2014.)

[nh13]

I have no issue getting rid of manylinux2014, which seems to have done the trick for x86-64, on manylinux. Done here: #63.

It gets particularly slow (1+ hour!) for musllinux on aarch64. I am curious why pysam does not build both intel and arm builds (see the cibuildwheels example, but only x86-64. Selfishly, adding wheels builds for both muslinux and aarch64 support in pysam would help here, but not the biggest of deals.

[jmarshall]

Look on PyPI to see which pysam wheels exist.

We build the ARM wheels on Cirrus CI because when this was set up that was the only CI vendor that really offered native ARM runners. Now that GitHub does as well, we may move this to GH actions, mainly because it's a bit annoying that Cirrus does not support PyPI Trusted Publishing.

Your aarch64 musllinux builds are spectacularly slow because they're running emulated on x86-64 and doing a full pysam build via emulation. So this will improve in future when pysam provides musllinux wheels, and you could probably improve it drastically now by running natively on ubuntu-2x.04-arm instead of via emulation.