BLD: add option to specify numpy header location #61095

OldManYellsAtCloud · 2025-03-10T13:17:09Z

In some cases the numpy module might not be usable during build-time, especially when cross-compiling. (E.g. when compiling for arm32 on a x86-64 machine, the arm32 module is not usable at build time).

This makes meson fail, as it isn't able to figure out the location of numpy headers.

To allow an alternative way to find these headers, introduce a meson build option, where the location of the numpy headers can be specified.

In case numpy module cannot be loaded for some reason to query the include folder location, fall back to the value of this meson option.

closes BUG: ValueError: Buffer dtype mismatch, expected 'intp_t' but got 'long long' on ARMv7 32 bit #55305 (Replace xxxx with the GitHub issue number)
Tests added and passed if fixing a bug or adding a new feature
All code checks passed.
Added type annotations to new arguments/methods/functions.
Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

WillAyd · 2025-03-10T17:03:26Z

pandas/meson.build

@@ -4,17 +4,24 @@ incdir_numpy = run_command(
        '-c',
        '''
 import os
-import numpy as np
+


This custom code to get the include path from NumPy was implemented as a workaround prior to Meson adding first class support, which I believe landed in 1.4

Rather than continue to patch this, I'd prefer if we bump our minimum Meson to 1.4 and use Meson's built-in NumPy resolution mechanisms

I'm also wary of pandas itself adding an option for this; you may want to check upstream if Meson supports NumPy for cross compilation, and if not ask for it to be implemented there rather than here

I started to rework it to use meson's numpy support - it does look nicer.

I need a day or two to stare at this a bit more though before committing. Meson supports numpy via pkgconfig and numpy-config. Numpy's pkgconfig file is not usable for cross-compiling. numpy-config looks more promising, but I couldn't make it work yet for cross-compiling - at this time I think it's numpy's issue, but want to touch some grass to think it through, maybe I find something if I stop looking at it for a short time. If it's numpy issue, than I would contact numpy about this before changing the meson scripts in pandas, otherwise it could make things more difficult than they are. Will be back.

Nope, unfortunately it requires a change in numpy to make it better - a way to get the includes folder without actually importing the numpy module (it's closely related to numpy/numpy#18209 ).

I think that currently, using meson's numpy resolution support would cause a regression for cross-compiling, it would make things harder. If this PR isn't merged, that's fine - the ultimate goal was to inform you about the root cause of the linked bug, and that this is one (but certainly not the only) possible solution.

@rgommers do you have any thoughts on this issue?

But it's the same absolute path as you need to pass in as a build option, like in this PR, right? I was suggesting generating this numpy.pc on the fly. It's maybe a couple lines more code in your build script, but it avoids adding this build option to every package that depends on NumPy - it seems like a win to me.

If that really doesn't work then this PR is okay too (it's what we've done in SciPy too after all).

Hmmm... technically it is possible. Let me do a poc, and see what the Yocto folks say about it. Will be back.

Going around the same idea, eventually I found an even simpler solution, which I think won't find a lot of resistance in Yocto - we actually don't necessarily need the full absolute path for cross-compiling, the relative path to the sysroot is perfectly enough, and that's a static path. That 1-liner sed needs to be done only once - I don't expect this to be seen controversial at all, will just submit it during the week as a run of the mill patch. Thanks for the hint.

I have pushed the version that uses Meson's own dependency resolution instead of the custom code.

Sounds like good news, thanks!

we actually don't necessarily need the full absolute path for cross-compiling, the relative path to the sysroot is perfectly enough

Out of interest: how does the translation from sysroot path to numpy header location happen?

Yocto has a standard variable that points to the current site-packages dir (without the sysroot path). The change I have just tested is essentially
sed -i "s:\${pcfiledir}:${PYTHON_SITEPACKAGES_DIR}/numpy/_core/lib/pkgconfig:g" [...]/numpy.pc

meson.build

WillAyd · 2025-03-16T20:14:03Z

I don't think the build failures showing now are related to this change, but rather a longstanding issue where build warnings were being suppressed with our Meson 1.2.1 pin. #60681 should have the more comprehensive fix for that

For now, you can just remove any setup calls with -Csetup-args="--werror"

OldManYellsAtCloud · 2025-03-16T20:23:50Z

Sure. Though there are still at least 1 or 2 that were broken by this - though if the list would get a bit cleaner, that wouldn't be a problem of course.

WillAyd · 2025-03-16T22:08:12Z

For the pre-commit error I think you just need to run ./scripts/generate_pip_deps_from_conda.py and commit the reuslt.

I thought the pre-commit hook would do this for you, so maybe it was skipped. If that doesn't work can help take a closer look

OldManYellsAtCloud · 2025-03-17T08:36:06Z

You are right, that did the trick. But I'm not really sure why the wheel* workflows weren't kicked off

OldManYellsAtCloud · 2025-03-17T10:29:22Z

I see the problem - it's about the PKG_CONF_PATH env var, which should be visible for each process, that is building pandas (or wants to find numpy via meson for any other reasons). But github actions is fighting against me. I will experiment how to pass env vars back and forth between actions and workflows, but in a test repo, and hopefully will be back.

rgommers · 2025-03-17T16:53:33Z

Ah yes, cibuildwheel-within-GHA is odd layering. I think this does what you want:

https://github.com/scipy/scipy/blob/689ebb9d0332b025d604b755febb68fb0d017a40/.github/workflows/wheels.yml#L125

Instead of querying the include folder's location from NumPy during building, use Meson's built-in dependency resolution for NumPy. With this change during build-time Meson first tries to query the dependency details using numpy-config (which, in turn essentially uses the same method as the original code this commit replaces), and in case that fails for some reason, it tries to discover NumPy resources using pkg-config - which, beside being a good fail-over mechanism, has the added benefit of somewhat simpler cross-compiling, as querying the include folder location from NumPy module is only usable for cross-compiling only in some corner cases, while pkg-config is a bit more universal. Signed-off-by: Gyorgy Sarvari <[email protected]>

OldManYellsAtCloud · 2025-03-18T12:57:13Z

I dunno. I tried it, but i doesn't seem to have any effect. It looks the environment variables are passed correctly, but they disappear in thin air by the time meson is executed. At this time I suspect that might be running into the same issue as mesonbuild/meson-python#604 - and it would need a clean build folder (but today I'm not brave enough to do rm -rf on random build machines), but that's only a desperate last thought, and I'm just afraid to accept the fact that it is cursed.

If you have any ideas, suggestions, please go for it, I'm all ears. Personally I have to put this to rest on my end for at least a day.

WillAyd · 2025-03-18T15:23:15Z

I might have missed a prior conversation but does NumPy definitely distribute the pkg-config file when installed via conda? Checking my local machine in an environment with 1.26.4 there is no directory for <site_packages>/numpy/_core/lib

lithomas1 · 2025-03-18T15:40:27Z

.github/workflows/docbuild-and-upload.yml

@@ -40,11 +40,18 @@ jobs:
      with:
        fetch-depth: 0

+    - name: Determine NumPy pkg-config location
+      run: |
+        echo "PKG_CONFIG_PATH=$(python -c 'import site; print(site.getsitepackages()[0])')/numpy/_core/lib/pkgconfig" >> $GITHUB_ENV


I feel like sprinkling PKG_CONFIG_PATH in every workflow seems a bit hacky.
Is it possible for us to upstream something to meson/numpy to autodetect this better so that we can avoid this?

Can you also add a comment (if this is necessary and not upstreamable) explaining what this environment variable does?

(At a quick glance, it is not entirely obvious to me what the ordering should be - i.e. should this go before or after the conda env is activated, and the significance of the hard-coded path inside numpy, so if you could add an explanation that would help me and the other core team members that aren't as familiar with pkgconfig a lot)

Is it possible for us to upstream something to meson/numpy to autodetect this better so that we can avoid this?

As far as Meson is concerned providing the pkg_config_path is a standard argument to setup/configure, so I don't think there is anything left that could be done in that library. For NumPy, Ralf would know best, but I am not sure that it would be able to install the pkg-config file to the normal location on a user's disk given that would be outside the typical installation architecture of a Python package

I feel like sprinkling PKG_CONFIG_PATH in every workflow seems a bit hacky

It's only needed for cross-compiling, so it should not be present/needed in other workflows.

With numpy 2.0, which provides numpy-config, this is all you need in meson.build (and nothing in CI jobs):

dependency('numpy')

OldManYellsAtCloud · 2025-03-18T16:22:10Z

When using meson's numpy resolution, there are two ways one can go in my understanding:

Use pkg-config. NumPy uses a non-standard location for the .pc file, I guess mostly because there is no standard location for .pc files provided by Python modules. This needs the PKG_CONFIG_PATH to be set to this custom path, otherwise pkg-config can't find it. I believe that there is always a numpy.pc file, though maybe depending on the version, the location might be different? (thinking this because of your comment) That unfortunately wouldn't make things easier. On the other hand I'm not too familiar with NumPy, I'm just trying to make it cross-compile - if you tell me that this is not right, I will believe it without second guessing.
Use /usr/bin/numpy-config. For this, meson needs to know where this executable/script is - it could be added to a meson*native file as a binary which that could be passed to meson (at least in some workflows this file seems to exist already, though I didn't check where it is coming from exactly), which would make the pkg-config method superfluous.

I do realize that currently it looks like a hack, and in the past days I only wanted to see it work at least once, to have at least a baseline - I suspect that half of the env-declarations are useless, and could be removed. But unfortunately I still have no idea why it doesn't work - and unless some miracle happens and I have some revelation, I think I might have to give this up - unfortunately I am not able to learn the ins and outs of a non-trivial CI setup due to time constraints :(

rgommers · 2025-03-18T19:48:01Z

in an environment with 1.26.4 there is no directory

It's new in 2.0.0

rgommers · 2025-03-18T19:50:34Z

meson.build

    default_options: ['buildtype=release', 'c_std=c11', 'warning_level=2'],
 )

 fs = import('fs')
 py = import('python').find_installation(pure: false)
 tempita = files('generate_pxi.py')
 versioneer = files('generate_version.py')
-
+numpy_dep = dependency('numpy', method: 'auto')


This should use required: false as long as numpy 1.26.x is still supported

rgommers · 2025-03-18T19:53:42Z

@OldManYellsAtCloud thank you for the effort, this was helpful to understand the limitations of the .pc file. This PR is largely still in the right direction. Let's see if we can finish it off. @WillAyd do you want to take a stab at it, or do you want me to?

WillAyd · 2025-03-19T20:18:29Z

I don't have the time personally to work on this at the moment, but am happy to review any changes that come through. I do think this is close

OldManYellsAtCloud mentioned this pull request Mar 10, 2025

BUG: ValueError: Buffer dtype mismatch, expected 'intp_t' but got 'long long' on ARMv7 32 bit #55305

Open

3 tasks

mroeschke requested review from WillAyd and lithomas1 March 10, 2025 16:46

mroeschke added the Build Library building on various platforms label Mar 10, 2025

WillAyd requested changes Mar 10, 2025

View reviewed changes

OldManYellsAtCloud force-pushed the main branch from ff1b9fa to 563ea31 Compare March 16, 2025 19:07

OldManYellsAtCloud requested a review from MarcoGorelli as a code owner March 16, 2025 19:07

WillAyd reviewed Mar 16, 2025

View reviewed changes

meson.build Show resolved Hide resolved

OldManYellsAtCloud force-pushed the main branch from 563ea31 to ee76552 Compare March 16, 2025 19:41

OldManYellsAtCloud requested a review from mroeschke as a code owner March 16, 2025 19:41

OldManYellsAtCloud force-pushed the main branch 2 times, most recently from d62c160 to 9867106 Compare March 16, 2025 20:06

OldManYellsAtCloud force-pushed the main branch from 9867106 to e247c1a Compare March 16, 2025 20:21

OldManYellsAtCloud force-pushed the main branch 2 times, most recently from c514975 to 49dac18 Compare March 16, 2025 20:35

OldManYellsAtCloud force-pushed the main branch from 49dac18 to d613efa Compare March 17, 2025 08:29

OldManYellsAtCloud force-pushed the main branch 5 times, most recently from 83cf21d to 74e5be0 Compare March 17, 2025 10:22

OldManYellsAtCloud force-pushed the main branch 2 times, most recently from 1338a90 to d9fb595 Compare March 18, 2025 12:07

OldManYellsAtCloud force-pushed the main branch from d9fb595 to 557a93d Compare March 18, 2025 12:11

lithomas1 reviewed Mar 18, 2025

View reviewed changes

OldManYellsAtCloud closed this Mar 18, 2025

rgommers reviewed Mar 18, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BLD: add option to specify numpy header location #61095

BLD: add option to specify numpy header location #61095

OldManYellsAtCloud commented Mar 10, 2025

WillAyd Mar 10, 2025

WillAyd Mar 10, 2025

OldManYellsAtCloud Mar 11, 2025

OldManYellsAtCloud Mar 16, 2025

WillAyd Mar 16, 2025

rgommers Mar 16, 2025

OldManYellsAtCloud Mar 16, 2025

OldManYellsAtCloud Mar 16, 2025

rgommers Mar 16, 2025

OldManYellsAtCloud Mar 16, 2025

WillAyd commented Mar 16, 2025

OldManYellsAtCloud commented Mar 16, 2025

WillAyd commented Mar 16, 2025

OldManYellsAtCloud commented Mar 17, 2025

OldManYellsAtCloud commented Mar 17, 2025

rgommers commented Mar 17, 2025

OldManYellsAtCloud commented Mar 18, 2025

WillAyd commented Mar 18, 2025

lithomas1 Mar 18, 2025 •

edited

Loading

lithomas1 Mar 18, 2025

WillAyd Mar 18, 2025

rgommers Mar 18, 2025

OldManYellsAtCloud commented Mar 18, 2025

rgommers commented Mar 18, 2025

rgommers Mar 18, 2025

rgommers commented Mar 18, 2025

WillAyd commented Mar 19, 2025

BLD: add option to specify numpy header location #61095

BLD: add option to specify numpy header location #61095

Conversation

OldManYellsAtCloud commented Mar 10, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WillAyd commented Mar 16, 2025

OldManYellsAtCloud commented Mar 16, 2025

WillAyd commented Mar 16, 2025

OldManYellsAtCloud commented Mar 17, 2025

OldManYellsAtCloud commented Mar 17, 2025

rgommers commented Mar 17, 2025

OldManYellsAtCloud commented Mar 18, 2025

WillAyd commented Mar 18, 2025

lithomas1 Mar 18, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

OldManYellsAtCloud commented Mar 18, 2025

rgommers commented Mar 18, 2025

Choose a reason for hiding this comment

rgommers commented Mar 18, 2025

WillAyd commented Mar 19, 2025

lithomas1 Mar 18, 2025 •

edited

Loading