[WIP] Handle meson based python-wheel

[th0ma7]

Description

Handle meson based python-wheel

Fixes:

• META: Python 3.12 package status #6421 (comment)

Checklist

• Build rule all-supported completed successfully
• New installation of package completed successfully
• Package upgrade completed successfully (Manually install the package again)
• Package functionality was tested
• Any needed documentation is updated/created

Type of change

• Bug fix
• New Package
• Package update
• Includes small framework changes
• This change requires a documentation update (e.g. Wiki)

[th0ma7]

Note that this is not ready for testing yet... still early WIP, simply ported in a new PR my pending changes from my local branch.

[hgy59]

additional info:

1. Supprted DSM

   building for comcerto2k-7.1 shows error:
   meson.build:28:4: ERROR: Problem encountered: NumPy requires GCC >= 8.4

   DSM < 7 will not be supported and we have to add to numpy Makefile something like:

   # numpy requires gcc >= 8.4
   REQUIRED_MIN_DSM = 7.0
   UNSUPPORTED_ARCHS = comcerto2k
   

2. cython not found
   when adding this to cross/numpy/Makefile:
   ENV += PATH=$(WORK_DIR)/crossenv-default/build/bin:$(PATH)
   it can find cython and sucessfully builds the wheel for x64

   it still fails for evansport, aarch64 and armv7:
   IMHO it should not use native/python312 but python in the build crossenv
   and additionally it must use python header files of cross python
   currently evansport and armv7 have errors like
   /spksrc/native/python312/work-native/install/usr/local/include/python3.12/pyport.h:586:2: error: #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
   (pyport.h does no match the 32-bit target platform)
   and aarch64 fails with:
   ../numpy/_core/src/umath/loops_autovec.dispatch.c.src:107:43: internal compiler error: Segmentation fault

[th0ma7]

additional info:

1. Supprted DSM
   building for comcerto2k-7.1 shows error:
   meson.build:28:4: ERROR: Problem encountered: NumPy requires GCC >= 8.4
   DSM < 7 will not be supported and we have to add to numpy Makefile something like:

   # numpy requires gcc >= 8.4
   REQUIRED_MIN_DSM = 7.0
   UNSUPPORTED_ARCHS = comcerto2k
   

Yup, on my radar, will add that indeed. Although I wonder if numpy 1.26.x might still work with older DSM considering the new meson I'm trying to build-up.

2. cython not found
   when adding this to cross/numpy/Makefile:
   ENV += PATH=$(WORK_DIR)/crossenv-default/build/bin:$(PATH)
   it can find cython and sucessfully builds the wheel for x64
   it still fails for evansport, aarch64 and armv7:
   IMHO it should not use native/python312 but python in the build crossenv

That should be the case from spksrc.python-module-meson.mk... Although up to now I was still on the per depend fully-generated meson cross-file (which looks like working now). So I haven't reconvene on that part just yet.

and additionally it must use python header files of cross python

The per depend fully-generated meson cross-file should now fix that. I had encountered that same issue with cmake long ago where the $(WORK_DIR)/tc-vars.cmake would be use in conjunction of per depend C|CPP|CXX|LDFLAGS. Issue was that:

1. The generic $(WORK_DIR)/tc-vars.cmake cannot handle per depends *FLAGS as there may be additional ones defined using ADDITIONAL_*FLAGS variables in any of the dependencies in cross/<meh>. And cmake cross file does not handle ${var} of that kind.
2. So to fix that when cmake is called it fetches the content of the generic $(WORK_DIR)/tc-vars.cmake and create a $(WORK_DIR)/$(PKG_DIR)/<arch>.cmake file and add all FLAGS information to it so it is fully complete and no longer dependent on env flags.
3. To that effect, with a fully defined per dependency cmake cross file the build process is called with an empty environment similarly as native. This ensures that the host is set to the targeted architecture while the build remains unchanged (i.e. the docker arch we're building on which gets auto-detected).

Long story short, the meson had never received such enhancement as things we're working just fine. But that's no longer true with python wheels whereas with the python virtual environment things gets totally confused for meson. Thus the need to have a fully functional meson cross-file defining all library and include path properly.

have a look at current tc-vars.cmake|meson and you'll notice no flags being set. When invoking a cmake build you'll have a fully compliant cross-file within the corresponding $(PKG_DIR).

This is now mostly working with meson, still have a few things to go through but getting there.

currently evansport and armv7 have errors like
/spksrc/native/python312/work-native/install/usr/local/include/python3.12/pyport.h:586:2: error: #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
(pyport.h does no match the 32-bit target platform)
and aarch64 fails with:
../numpy/_core/src/umath/loops_autovec.dispatch.c.src:107:43: internal compiler error: Segmentation fault

That's a current known issue with numpy. We need to force settin the long bit accordingly for big or little endian into the meson cross-file such as:

[properties]
longdouble_format = 'IEEE_DOUBLE_BE'

EDIT: with regards to cython (which I just hit) there must be an issue with the PATH although there may be a way to set them in the meson native file such as:

[binaries]
python = '/usr/bin/python3'
cython = '/usr/bin/cython'

Anyhow, as usual thnx for your feedback, and work slowly progressing on this.

EDIT2: It turns out that I have yet to empty the env now when invoking meson build, which is not the case yet. Although it does work for regular meson builds it wont for python-based meson wheel builds. next on my todo.

[th0ma7]

@hgy59 having a proof of concept that does build sucessfully for both aarch64 and x64 using latest numpy 2.2.3.

Although struggling with armv7 and evansport... I'll check if I can make 1.26.4 to work instead for the moment.

[mreid-tt]

@th0ma7, was looking at the errors which remain:

error: #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."

And found the following which may be useful if you've not already considered:

• Building cython based module with numpy dependency in SDK
• Compilation error with out-of-tree builds with 0.24.0a1: LONG_BIT definition appears wrong for platform pyodide/pyodide#4118

Hope they can assist...

[th0ma7]

I believe I now have something functional, but unmaintainable as-is.

The good

Using normal python -m pip wheel ... and passing proper meson-python flags:

--config-settings=setup-args="--cross-file=$(MESON_CROSS_TOOLCHAIN_PKG)" \
--config-settings=setup-args="--native-file=$(MESON_NATIVE_FILE)" \
--config-settings=install-args="--tags=runtime,python-runtime" \
--config-settings=build-dir="$(MESON_BUILD_DIR)" \
--no-build-isolation

I can now sucesfully cross-compile for arm7, evansport and x64 for DSM-7.1

The bad

There is a know bug in gcc<=10 with aarch64 that makes the compiler to segfault. I tried pretty much every possible alternatives of flags/disabling in the code but I wasn't able to workaround it.
Bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=97696
Potential workaround that I was not able to make use of: https://github.com/google/highway/pull/1932/files
@hgy59 maybe you can help on this which would be really nice to have a working patch.

The ugly

Part of my crusade at making meson-python build to work, I ended-up at one point to reproduce the normal meson+ninja build. Surprisingly, this ended-up allowing me to sucesfully build numpy for aarch64... Missing is then the (re)packaging in wheel format, which hapens to be the exact same process as "The good" as long as I re-use the exact same builddir (which I ended-up figuring out tonight).

This really is ugly, but does work. This last commit a2068f5 was not tested on previously working x64, evansport and armv7. I'll let this rest for tonight. Good news is, we're probably much closer now... just need to tie-up the remaining loose-ends.

[hgy59]

@th0ma7 the wheels created from python/*/Makefile are not yet added to wheelhouse/requirements.txt.
And I guess it should be added to wheelhouse/requirements-cross.txt too.

The wheelhouse/requirements.txt is used by the install_python_wheels function in spksrc.service.installer.functions

[th0ma7]

Thnx for catching this up, will include this.

I'm also looking at how to install numpy in the crossenv... I got an idea on how i could reuse the newly cross-compiled numpy wheel so it gets installed into the cross portion of the crossenv so it can then be made available for other wheels that depends on it.

Lastly, also looking at adding flexibility to have different vendor managed meson (other than numpy use case where the source package provides its own modified meson.py) and skipping that meson+ninja part when no vendor managed meson is provided (i.e. being the default use case)

All in all, taking shape but will require a few more spare cycles before reaching the finishing line...

[hgy59]

@th0ma7 another small issue popped up:
The wheel for charset-normalizer is generated as charset_normalizer-3.4.1-py3-none-any.whl and also the WHEEL file shows, it is pure python:

Wheel-Version: 1.0
Generator: setuptools (75.8.0)
Root-Is-Purelib: true
Tag: py3-none-any

The original wheels in the index (pypi) are cross compiled (like charset_normalizer-3.4.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl).
maybe our crossenv build needs a flag or something to create a cross compiled wheel for charset-normalizer.

[hgy59]

@th0ma7 I have successfully built python311-wheels with added python/numpy and python/numpy_1.26 for aarch64-7.1 and armv7-7.1.

It would be interresting to validate whether such wheels created with gcc 8.5 will run under DSM 6. I guess if the *.so files within the wheels do not reference GLBC > 2.20 functions, it might work.

My background: I am trying to build a final homeassistant package with support for DSM 6. This will be homeassistant 2024.3.3 that depends on numpy 1.26.0. This version is available in the index for x86_64 and aarch64 only, and I will have to build it at least for armv7 and evansport (i686).
To support armv7, I will have to cross build additionally ha-av==10.1.1 (this is not so easy since it depends on ffmpeg libraries).

To support armv7 in homeassistant 2025.1.4, it will be av==13.1.0 that depends on ffmpeg libraries too

[th0ma7]

@th0ma7 another small issue popped up:
The wheel for charset-normalizer is generated as charset_normalizer-3.4.1-py3-none-any.whl and also the WHEEL file shows, it is pure python:

Wheel-Version: 1.0
Generator: setuptools (75.8.0)
Root-Is-Purelib: true
Tag: py3-none-any

The original wheels in the index (pypi) are cross compiled (like charset_normalizer-3.4.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl).
maybe our crossenv build needs a flag or something to create a cross compiled wheel for charset-normalizer.

Maybe this is similar to msgpack where it can fit in both?

[th0ma7]

@th0ma7 I have successfully built python311-wheels with added python/numpy and python/numpy_1.26 for aarch64-7.1 and armv7-7.1.

It would be interresting to validate whether such wheels created with gcc 8.5 will run under DSM 6. I guess if the *.so files within the wheels do not reference GLBC > 2.20 functions, it might work.

My background: I am trying to build a final homeassistant package with support for DSM 6. This will be homeassistant 2024.3.3 that depends on numpy 1.26.0. This version is available in the index for x86_64 and aarch64 only, and I will have to build it at least for armv7 and evansport (i686).
To support armv7, I will have to cross build additionally ha-av==10.1.1 (this is not so easy since it depends on ffmpeg libraries).

To support armv7 in homeassistant 2025.1.4, it will be av==13.1.0 that depends on ffmpeg libraries too

That's a long shot! Not sure how i can help you though. I could reinstall my armv7 using a 6.2.4 image to try it out if that helps?

[th0ma7]

I got some pretty cool code locally that allows installing cross-compiled numpy wheel into the crossenv to allow building scipy and others... But I faced one major major major problem, gcc version.

For numpy, with aarch64, it mandatory requires gcc>=10 otherwise it segfaults. I was able to workaround that up until needing to include OpenBLAS which re-trigerred this segfault bug with gcc for numpy. And without it I cannot build scipy, thus I cannot build scikit-learn, thus the domino effect.

@hgy59 All in all, this would require bumping our minimal version to DSM-7.2.

EDIT: I'll sleep on it... and probably upload my new code online to safeguard it just in case even though it will fail to build.

[th0ma7]

Good news, I was able to create a workaround patch for aarch64 ... a few loose ends but looking much better now.

[th0ma7]

@hgy59 and @mreid-tt It may look like stagnating but after spending numerous hours on this I finally made a major leap forward which now allows using default pip wheel methodology to build meson type wheels. Hopefully this will now allow me to include lapack and open the way to scipy, and thus things should start rolling at that point.

This has definitively been taking way longer than anticipated but I believe things will now start to shape nicely 🤞

[th0ma7]

@hgy59 and @mreid-tt progress being made:

• numpy (and thnx @hgy59 for your few additions to enable lapack from openblas)
• scipy
• scikit-learn ... next. Any other missing using meson-python that also depends on numpy + scipy ?

Afterwards, missing are:

• adding fortran to cmake portion of the framework
• confirm wetter using python -m pip or python -m build ... thinking of keeping both with a flag option, TBD.
• cleanups & testing

Then things should be good for merging :)

[hgy59]

@th0ma7 another wheel that requires numpy is pandas (HA 2025.1.4 depends on pandas==2.2.3)

python312/crossenv/requirements-pandas.txt would look like:

pip==24.3.1
setuptools==75.8.0
wheel==0.45.1
#
build:build==1.2.2
build:Cython==3.0.11
build:meson==1.2.1
build:meson-python==0.13.1
build:pip-tools==7.4.1
#build:versioneer[toml]
build:versioneer==0.29
#
wheelhouse:numpy==2.2.3

[hgy59]

Some pending issues (maybe for another PR)

• When a requirement-file is missing in WHEELS = missing-requirements.txt it is silently ignored (I expect an error like file ... not found)
• If a package depends on cross compiled wheels (python/* or cross/*) and does not have other requirements (i.e. does not have WHEELS = ... defined) then the wheels are not added to the package (those are in wheelhouse but not in install/..../shared/wheelhouse, and whl names are not adjusted)
• when using package names like numpy_v2 (in python/numpy/Makefile) a specificy crossenv is not used (I suggest that the file must be named requirements-numpy_v2.txt instead of requirements-numpy.txt ) If this is by intention it should be documented somewhere... or we need another solution to include multiple versions of such wheels

[hgy59]

@th0ma7 another wheel that requires numpy is pandas (HA 2025.1.4 depends on pandas==2.2.3)

python312/crossenv/requirements-pandas.txt would look like:

pip==24.3.1
setuptools==75.8.0
wheel==0.45.1
#
build:build==1.2.2
build:Cython==3.0.11
build:meson==1.2.1
build:meson-python==0.13.1
build:pip-tools==7.4.1
#build:versioneer[toml]
build:versioneer==0.29
#
wheelhouse:numpy==2.2.3

I tried this, and the crossenv is created as expected 🎉 - but it fails to build:
I use numpy==2.2.0 as required for HA

===>  Package list for /spksrc/diyspk/pandas_ha/work-x64-7.1/crossenv-pandas/build:
Package            Version
------------------ -------
build              1.2.2
click              8.1.8
Cython             3.0.11
meson              1.2.1
meson-python       0.13.1
packaging          24.2
pip                23.2.1
pip-tools          7.4.1
pyproject_hooks    1.2.0
pyproject-metadata 0.9.0
setuptools         75.8.0
versioneer         0.29
wheel              0.45.1
===>  Package list for /spksrc/diyspk/pandas_ha/work-x64-7.1/crossenv-pandas/cross:
Package    Version
---------- -------
numpy      2.2.0
pip        23.2.1
setuptools 75.8.0
wheel      0.45.1
===>  Generating
env make --no-print-directory meson_pkg_toolchain > /spksrc/diyspk/pandas_ha/work-x64-7.1/pandas-2.2.3/x64-toolchain.meson 2>/dev/null;
===>  Compiling for pandas
===>  Installing for pandas
find /spksrc/diyspk/pandas_ha/work-x64-7.1/install//var/packages/pandas-wheel/target/../ \! -type d -printf '%P\n' | sed 's?^target/??g' | sort > /spksrc/diyspk/pandas_ha/work-x64-7.1/pandas.plist.tmp
===>  - crossenv: [/spksrc/diyspk/pandas_ha/work-x64-7.1/crossenv-pandas/bin/activate]
===>  - meson: [/spksrc/diyspk/pandas_ha/work-x64-7.1/crossenv-pandas/build/bin/meson]
===>  - python: [/spksrc/diyspk/pandas_ha/work-x64-7.1/crossenv-pandas/bin/cross-python]
===>  _PYTHON_HOST_PLATFORM="x86_64-pc-linux-gnu" PATH=/spksrc/diyspk/pandas_ha/work-x64-7.1/crossenv-pandas/cross/bin:/spksrc/diyspk/pandas_ha/work-x64-7.1/crossenv-pandas/build/bin:/spksrc/diyspk/pandas_ha/work-x64-7.1/crossenv-pandas/bin:/spksrc/distrib/cargo/bin:/home/.local/share/pnpm:/scripts:/home/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/games /spksrc/diyspk/pandas_ha/work-x64-7.1/crossenv-pandas/bin/cross-python -m pip wheel . --config-settings=setup-args="-Dbuildtype=release" --config-settings=setup-args="--cross-file=/spksrc/diyspk/pandas_ha/work-x64-7.1/pandas-2.2.3/x64-toolchain.meson" --config-settings=builddir="/spksrc/diyspk/pandas_ha/work-x64-7.1/pandas-2.2.3/builddir" --no-build-isolation --wheel-dir /spksrc/diyspk/pandas_ha/work-x64-7.1/wheelhouse --verbose
Processing /spksrc/diyspk/pandas_ha/work-x64-7.1/pandas-2.2.3
  Preparing metadata (pyproject.toml): started
  Running command Preparing metadata (pyproject.toml)
  + meson setup /spksrc/diyspk/pandas_ha/work-x64-7.1/pandas-2.2.3 /spksrc/diyspk/pandas_ha/work-x64-7.1/pandas-2.2.3/builddir -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --vsenv -Dbuildtype=release --cross-file=/spksrc/diyspk/pandas_ha/work-x64-7.1/pandas-2.2.3/x64-toolchain.meson --native-file=/spksrc/diyspk/pandas_ha/work-x64-7.1/pandas-2.2.3/builddir/meson-python-native-file.ini
  The Meson build system
  Version: 1.2.1
  Source dir: /spksrc/diyspk/pandas_ha/work-x64-7.1/pandas-2.2.3
  Build dir: /spksrc/diyspk/pandas_ha/work-x64-7.1/pandas-2.2.3/builddir
  Build type: cross build
  Project name: pandas
  Project version: 2.2.3
  C compiler for the host machine: /spksrc/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/bin/x86_64-pc-linux-gnu-gcc (gcc 8.5.0 "x86_64-pc-linux-gnu-gcc (GCC) 8.5.0")
  C linker for the host machine: /spksrc/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/bin/x86_64-pc-linux-gnu-gcc ld.bfd 2.30
  C++ compiler for the host machine: /spksrc/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/bin/x86_64-pc-linux-gnu-g++ (gcc 8.5.0 "x86_64-pc-linux-gnu-g++ (GCC) 8.5.0")
  C++ linker for the host machine: /spksrc/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/bin/x86_64-pc-linux-gnu-g++ ld.bfd 2.30
  Cython compiler for the host machine: cython (cython 3.0.11)
  C compiler for the build machine: cc (gcc 12.2.0 "cc (Debian 12.2.0-14) 12.2.0")
  C linker for the build machine: cc ld.bfd 2.40
  C++ compiler for the build machine: c++ (gcc 12.2.0 "c++ (Debian 12.2.0-14) 12.2.0")
  C++ linker for the build machine: c++ ld.bfd 2.40
  Cython compiler for the build machine: cython (cython 3.0.11)
  Build machine cpu family: x86_64
  Build machine cpu: x86_64
  Host machine cpu family: x86_64
  Host machine cpu: x86_64
  Target machine cpu family: x86_64
  Target machine cpu: x86_64
  Program python found: YES (/spksrc/diyspk/pandas_ha/work-x64-7.1/crossenv-pandas/bin/cross-python)

  ../pandas/meson.build:1:15: ERROR: Command `/spksrc/diyspk/pandas_ha/work-x64-7.1/crossenv-pandas/bin/cross-python -c '
  import os
  import numpy as np
  try:
      # Check if include directory is inside the pandas dir
      # e.g. a venv created inside the pandas dir
      # If so, convert it to a relative path
      incdir = os.path.relpath(np.get_include())
  except Exception:
      incdir = np.get_include()
  print(incdir)
       '` failed with status 1.

  A full log can be found at /spksrc/diyspk/pandas_ha/work-x64-7.1/pandas-2.2.3/builddir/meson-logs/meson-log.txt
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /spksrc/diyspk/pandas_ha/work-x64-7.1/crossenv-pandas/cross/bin/python /spksrc/spk/python312/work-x64-7.1/install/var/packages/python312/target/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmp1bmpbgw8
  cwd: /spksrc/diyspk/pandas_ha/work-x64-7.1/pandas-2.2.3
  Preparing metadata (pyproject.toml): finished with status 'error'
error: metadata-generation-failed


× Encountered error while generating package metadata.
╰─> See above for output.

with further details in meson-log.txt:

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.12 from "/spksrc/diyspk/pandas_ha/work-x64-7.1/crossenv-pandas/cross/bin/python"
  * The NumPy version is: "2.2.0"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: libgfortran.so.5: cannot open shared object file: No such file or directory

I guess this is an known issue (numpy is lacking fortran libraries) and numpy wheels are not yet fully working.