Handle meson based python-wheel

[th0ma7]

Description

Handle meson based python-wheel

MESON

• Add fortran support for DSM >= 7
• Add per-dependency level meson cross-file generation (spksrc.cross-meson-crossfile.mk)
• Enforce emptying ENV when using meson cross-file (using $(ENV_MESON))
  NOTE: Due to a bug LDFLAGS is being kept in environment as it otherwise fails to set rpath properly when having multiple run path definitions.

CMAKE

• Add fortran support for DSM >= 7
• Isolate per-dependency level cmake toolchain-file generation (spksrc.cross-cmake-toolchainfile.mk)
  NOTE: While meson refers to cross-file, cmake refers to toolchain file.
• Enforce emptying ENV when using meson cross-file (using $(ENV_CMAKE))

wheel-install

• Avoid copying wheels at every wheel build to reduce overall compile time

python31*-wheels

• Add meson-python wheels to python311
• Add meson-python wheels to python312
• Add meson-python wheels to python313

Fixes

• META: Python 3.12 package status #6421 (comment)

Checklist

• Build rule all-supported completed successfully
• New installation of package completed successfully
• Package upgrade completed successfully (Manually install the package again)
• Package functionality was tested
• Any needed documentation is updated/created

Type of change

• Bug fix
• New Package
• Package update
• Includes small framework changes
• This change requires a documentation update (e.g. Wiki)

[th0ma7]

Note that this is not ready for testing yet... still early WIP, simply ported in a new PR my pending changes from my local branch.

[hgy59]

additional info:

1. Supprted DSM

   building for comcerto2k-7.1 shows error:
   meson.build:28:4: ERROR: Problem encountered: NumPy requires GCC >= 8.4

   DSM < 7 will not be supported and we have to add to numpy Makefile something like:

   # numpy requires gcc >= 8.4
   REQUIRED_MIN_DSM = 7.0
   UNSUPPORTED_ARCHS = comcerto2k
   

2. cython not found
   when adding this to cross/numpy/Makefile:
   ENV += PATH=$(WORK_DIR)/crossenv-default/build/bin:$(PATH)
   it can find cython and sucessfully builds the wheel for x64

   it still fails for evansport, aarch64 and armv7:
   IMHO it should not use native/python312 but python in the build crossenv
   and additionally it must use python header files of cross python
   currently evansport and armv7 have errors like
   /spksrc/native/python312/work-native/install/usr/local/include/python3.12/pyport.h:586:2: error: #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
   (pyport.h does no match the 32-bit target platform)
   and aarch64 fails with:
   ../numpy/_core/src/umath/loops_autovec.dispatch.c.src:107:43: internal compiler error: Segmentation fault

[th0ma7]

additional info:

1. Supprted DSM
   building for comcerto2k-7.1 shows error:
   meson.build:28:4: ERROR: Problem encountered: NumPy requires GCC >= 8.4
   DSM < 7 will not be supported and we have to add to numpy Makefile something like:

   # numpy requires gcc >= 8.4
   REQUIRED_MIN_DSM = 7.0
   UNSUPPORTED_ARCHS = comcerto2k
   

Yup, on my radar, will add that indeed. Although I wonder if numpy 1.26.x might still work with older DSM considering the new meson I'm trying to build-up.

2. cython not found
   when adding this to cross/numpy/Makefile:
   ENV += PATH=$(WORK_DIR)/crossenv-default/build/bin:$(PATH)
   it can find cython and sucessfully builds the wheel for x64
   it still fails for evansport, aarch64 and armv7:
   IMHO it should not use native/python312 but python in the build crossenv

That should be the case from spksrc.python-module-meson.mk... Although up to now I was still on the per depend fully-generated meson cross-file (which looks like working now). So I haven't reconvene on that part just yet.

and additionally it must use python header files of cross python

The per depend fully-generated meson cross-file should now fix that. I had encountered that same issue with cmake long ago where the $(WORK_DIR)/tc-vars.cmake would be use in conjunction of per depend C|CPP|CXX|LDFLAGS. Issue was that:

1. The generic $(WORK_DIR)/tc-vars.cmake cannot handle per depends *FLAGS as there may be additional ones defined using ADDITIONAL_*FLAGS variables in any of the dependencies in cross/<meh>. And cmake cross file does not handle ${var} of that kind.
2. So to fix that when cmake is called it fetches the content of the generic $(WORK_DIR)/tc-vars.cmake and create a $(WORK_DIR)/$(PKG_DIR)/<arch>.cmake file and add all FLAGS information to it so it is fully complete and no longer dependent on env flags.
3. To that effect, with a fully defined per dependency cmake cross file the build process is called with an empty environment similarly as native. This ensures that the host is set to the targeted architecture while the build remains unchanged (i.e. the docker arch we're building on which gets auto-detected).

Long story short, the meson had never received such enhancement as things we're working just fine. But that's no longer true with python wheels whereas with the python virtual environment things gets totally confused for meson. Thus the need to have a fully functional meson cross-file defining all library and include path properly.

have a look at current tc-vars.cmake|meson and you'll notice no flags being set. When invoking a cmake build you'll have a fully compliant cross-file within the corresponding $(PKG_DIR).

This is now mostly working with meson, still have a few things to go through but getting there.

currently evansport and armv7 have errors like
/spksrc/native/python312/work-native/install/usr/local/include/python3.12/pyport.h:586:2: error: #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
(pyport.h does no match the 32-bit target platform)
and aarch64 fails with:
../numpy/_core/src/umath/loops_autovec.dispatch.c.src:107:43: internal compiler error: Segmentation fault

That's a current known issue with numpy. We need to force settin the long bit accordingly for big or little endian into the meson cross-file such as:

[properties]
longdouble_format = 'IEEE_DOUBLE_BE'

EDIT: with regards to cython (which I just hit) there must be an issue with the PATH although there may be a way to set them in the meson native file such as:

[binaries]
python = '/usr/bin/python3'
cython = '/usr/bin/cython'

Anyhow, as usual thnx for your feedback, and work slowly progressing on this.

EDIT2: It turns out that I have yet to empty the env now when invoking meson build, which is not the case yet. Although it does work for regular meson builds it wont for python-based meson wheel builds. next on my todo.

[th0ma7]

@hgy59 having a proof of concept that does build sucessfully for both aarch64 and x64 using latest numpy 2.2.3.

Although struggling with armv7 and evansport... I'll check if I can make 1.26.4 to work instead for the moment.

[mreid-tt]

@th0ma7, was looking at the errors which remain:

error: #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."

And found the following which may be useful if you've not already considered:

• Building cython based module with numpy dependency in SDK
• Compilation error with out-of-tree builds with 0.24.0a1: LONG_BIT definition appears wrong for platform pyodide/pyodide#4118

Hope they can assist...

[th0ma7]

I believe I now have something functional, but unmaintainable as-is.

The good

Using normal python -m pip wheel ... and passing proper meson-python flags:

--config-settings=setup-args="--cross-file=$(MESON_CROSS_TOOLCHAIN_PKG)" \
--config-settings=setup-args="--native-file=$(MESON_NATIVE_FILE)" \
--config-settings=install-args="--tags=runtime,python-runtime" \
--config-settings=build-dir="$(MESON_BUILD_DIR)" \
--no-build-isolation

I can now sucesfully cross-compile for arm7, evansport and x64 for DSM-7.1

The bad

There is a know bug in gcc<=10 with aarch64 that makes the compiler to segfault. I tried pretty much every possible alternatives of flags/disabling in the code but I wasn't able to workaround it.
Bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=97696
Potential workaround that I was not able to make use of: https://github.com/google/highway/pull/1932/files
@hgy59 maybe you can help on this which would be really nice to have a working patch.

The ugly

Part of my crusade at making meson-python build to work, I ended-up at one point to reproduce the normal meson+ninja build. Surprisingly, this ended-up allowing me to sucesfully build numpy for aarch64... Missing is then the (re)packaging in wheel format, which hapens to be the exact same process as "The good" as long as I re-use the exact same builddir (which I ended-up figuring out tonight).

This really is ugly, but does work. This last commit a2068f5 was not tested on previously working x64, evansport and armv7. I'll let this rest for tonight. Good news is, we're probably much closer now... just need to tie-up the remaining loose-ends.

[hgy59]

@th0ma7 the wheels created from python/*/Makefile are not yet added to wheelhouse/requirements.txt.
And I guess it should be added to wheelhouse/requirements-cross.txt too.

The wheelhouse/requirements.txt is used by the install_python_wheels function in spksrc.service.installer.functions

[th0ma7]

Thnx for catching this up, will include this.

I'm also looking at how to install numpy in the crossenv... I got an idea on how i could reuse the newly cross-compiled numpy wheel so it gets installed into the cross portion of the crossenv so it can then be made available for other wheels that depends on it.

Lastly, also looking at adding flexibility to have different vendor managed meson (other than numpy use case where the source package provides its own modified meson.py) and skipping that meson+ninja part when no vendor managed meson is provided (i.e. being the default use case)

All in all, taking shape but will require a few more spare cycles before reaching the finishing line...

[hgy59]

@th0ma7 another small issue popped up:
The wheel for charset-normalizer is generated as charset_normalizer-3.4.1-py3-none-any.whl and also the WHEEL file shows, it is pure python:

Wheel-Version: 1.0
Generator: setuptools (75.8.0)
Root-Is-Purelib: true
Tag: py3-none-any

The original wheels in the index (pypi) are cross compiled (like charset_normalizer-3.4.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl).
maybe our crossenv build needs a flag or something to create a cross compiled wheel for charset-normalizer.

[hgy59]

@th0ma7 I have successfully built python311-wheels with added python/numpy and python/numpy_1.26 for aarch64-7.1 and armv7-7.1.

It would be interresting to validate whether such wheels created with gcc 8.5 will run under DSM 6. I guess if the *.so files within the wheels do not reference GLBC > 2.20 functions, it might work.

My background: I am trying to build a final homeassistant package with support for DSM 6. This will be homeassistant 2024.3.3 that depends on numpy 1.26.0. This version is available in the index for x86_64 and aarch64 only, and I will have to build it at least for armv7 and evansport (i686).
To support armv7, I will have to cross build additionally ha-av==10.1.1 (this is not so easy since it depends on ffmpeg libraries).

To support armv7 in homeassistant 2025.1.4, it will be av==13.1.0 that depends on ffmpeg libraries too

[th0ma7]

@th0ma7 another small issue popped up:
The wheel for charset-normalizer is generated as charset_normalizer-3.4.1-py3-none-any.whl and also the WHEEL file shows, it is pure python:

Wheel-Version: 1.0
Generator: setuptools (75.8.0)
Root-Is-Purelib: true
Tag: py3-none-any

The original wheels in the index (pypi) are cross compiled (like charset_normalizer-3.4.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl).
maybe our crossenv build needs a flag or something to create a cross compiled wheel for charset-normalizer.

Maybe this is similar to msgpack where it can fit in both?

[th0ma7]

@th0ma7 I have successfully built python311-wheels with added python/numpy and python/numpy_1.26 for aarch64-7.1 and armv7-7.1.

It would be interresting to validate whether such wheels created with gcc 8.5 will run under DSM 6. I guess if the *.so files within the wheels do not reference GLBC > 2.20 functions, it might work.

My background: I am trying to build a final homeassistant package with support for DSM 6. This will be homeassistant 2024.3.3 that depends on numpy 1.26.0. This version is available in the index for x86_64 and aarch64 only, and I will have to build it at least for armv7 and evansport (i686).
To support armv7, I will have to cross build additionally ha-av==10.1.1 (this is not so easy since it depends on ffmpeg libraries).

To support armv7 in homeassistant 2025.1.4, it will be av==13.1.0 that depends on ffmpeg libraries too

That's a long shot! Not sure how i can help you though. I could reinstall my armv7 using a 6.2.4 image to try it out if that helps?

[th0ma7]

I got some pretty cool code locally that allows installing cross-compiled numpy wheel into the crossenv to allow building scipy and others... But I faced one major major major problem, gcc version.

For numpy, with aarch64, it mandatory requires gcc>=10 otherwise it segfaults. I was able to workaround that up until needing to include OpenBLAS which re-trigerred this segfault bug with gcc for numpy. And without it I cannot build scipy, thus I cannot build scikit-learn, thus the domino effect.

@hgy59 All in all, this would require bumping our minimal version to DSM-7.2.

EDIT: I'll sleep on it... and probably upload my new code online to safeguard it just in case even though it will fail to build.

[th0ma7]

Good news, I was able to create a workaround patch for aarch64 ... a few loose ends but looking much better now.

[th0ma7]

@hgy59 and @mreid-tt It may look like stagnating but after spending numerous hours on this I finally made a major leap forward which now allows using default pip wheel methodology to build meson type wheels. Hopefully this will now allow me to include lapack and open the way to scipy, and thus things should start rolling at that point.

This has definitively been taking way longer than anticipated but I believe things will now start to shape nicely 🤞

[th0ma7]

@hgy59 and @mreid-tt I'll let github-action run and if no remaining issues are found I'll merge. I'll do the remaining changes mentioned in the description from another PR helping tracking issues between PR if any.

Feel free to let me know if you see anything needing adjustments prior from me merging things. thnx.

[th0ma7]

@hgy59 and @mreid-tt just briging your attention that the ability to include wheel requirement definition directly within the WHEELS variable is now functional once again. This can allow avoiding using multiple requirement files for every gcc versions and ease code maintenance. Although, I don't suggest moving away from using requirement files, but use this for special cases.

Lastly, note that you cal add the pure:, abi3: or cross: prefix. With none defaults to cross.

[hgy59]

@th0ma7 it looks like it failed to download https://download.savannah.gnu.org/releases/acl/acl-2.3.1.tar.xz

will cancel, delete the download cache an rerun all jobs (not only failed) should fix it.

EDIT:
second attempt seems ok, no download error in Prepare Job...

[th0ma7]

Now merged. For your infromation, @hgy59 and @mreid-tt note that I will have a reduced availability for a few months. I'll look into doing the other changes mentioned in this PR's description although this could take a little longer.

In the meantime let me know if you find something not working out with this PR, thnx.

[hgy59]

@th0ma7 thanks for the huge work!

shall we now publish Python 3.13 (and update of Python 3.12)?

[th0ma7]

@th0ma7 thanks for the huge work!

A pleasure, and sorry for taking that long! This was more complex than anticipated.

shall we now publish Python 3.13 (and update of Python 3.12)?

Sure, although i would not suggest not migrating to 3.13 now (feels like we had that chat but don't recall the ending)

Also, if you want to help out on this effort, feel free to move all python module/wheel based entries from cross to the new python folder :)

[hgy59]

Sure, although i would not suggest not migrating to 3.13 now (feels like we had that chat but don't recall the ending)

already tried py313 with homeassistant >= 2025.2.x, but several wheels fail to cross compile...

[hgy59]

Also, if you want to help out on this effort, feel free to move all python module/wheel based entries from cross to the new python folder :)

sure, but my cycles are limitted too, in the next few weeks...

[th0ma7]

thnx @hgy59 for uploading py312-313. I just noticed the py313 changelog, not that it matter too much:

Version 3.13.2-2
1. Initial Python 3.13.1 package release

and likewise for py312:

Version 3.12.9-2
1. Initial Python 3.12.8 package release