Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BLD: add option to specify numpy header location #61095

Closed
wants to merge 1 commit into from
Closed

BLD: add option to specify numpy header location #61095

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 2 additions & 4 deletions .github/actions/build_pandas/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,8 @@ runs:
- name: Build Pandas
run: |
if [[ ${{ inputs.editable }} == "true" ]]; then
pip install -e . --no-build-isolation -v --no-deps \
-Csetup-args="--werror"
pip install -e . --no-build-isolation -v --no-deps
else
pip install . --no-build-isolation -v --no-deps \
-Csetup-args="--werror"
pip install . --no-build-isolation -v --no-deps
fi
shell: bash -el {0}
1 change: 1 addition & 0 deletions .github/workflows/code-checks.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ on:
env:
ENV_FILE: environment.yml
PANDAS_CI: 1
PKG_CONFIG_PATH: "${{env.PYTHONHOME}}/numpy/_core/lib/pkgconfig"

permissions:
contents: read
Expand Down
7 changes: 7 additions & 0 deletions .github/workflows/docbuild-and-upload.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,11 +40,18 @@ jobs:
with:
fetch-depth: 0

- name: Determine NumPy pkg-config location
run: |
echo "PKG_CONFIG_PATH=$(python -c 'import site; print(site.getsitepackages()[0])')/numpy/_core/lib/pkgconfig" >> $GITHUB_ENV
Copy link
Member

@lithomas1 lithomas1 Mar 18, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like sprinkling PKG_CONFIG_PATH in every workflow seems a bit hacky.
Is it possible for us to upstream something to meson/numpy to autodetect this better so that we can avoid this?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you also add a comment (if this is necessary and not upstreamable) explaining what this environment variable does?

(At a quick glance, it is not entirely obvious to me what the ordering should be - i.e. should this go before or after the conda env is activated, and the significance of the hard-coded path inside numpy, so if you could add an explanation that would help me and the other core team members that aren't as familiar with pkgconfig a lot)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible for us to upstream something to meson/numpy to autodetect this better so that we can avoid this?

As far as Meson is concerned providing the pkg_config_path is a standard argument to setup/configure, so I don't think there is anything left that could be done in that library. For NumPy, Ralf would know best, but I am not sure that it would be able to install the pkg-config file to the normal location on a user's disk given that would be outside the typical installation architecture of a Python package

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like sprinkling PKG_CONFIG_PATH in every workflow seems a bit hacky

It's only needed for cross-compiling, so it should not be present/needed in other workflows.

With numpy 2.0, which provides numpy-config, this is all you need in meson.build (and nothing in CI jobs):

dependency('numpy')

- name: Set up Conda
uses: ./.github/actions/setup-conda

- name: Build Pandas
uses: ./.github/actions/build_pandas
env:
CIBW_ENVIRONMENT: "PKG_CONFIG_PATH=${{env.PKG_CONFIG_PATH}}"
PKG_CONFIG_PATH: ${{env.PKG_CONFIG_PATH}}

- name: Extra installs
# https://pytest-qt.readthedocs.io/en/latest/troubleshooting.html#github-actions-azure-pipelines-travis-ci-and-gitlab-ci-cd
Expand Down
19 changes: 11 additions & 8 deletions .github/workflows/unit-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,9 @@ defaults:
run:
shell: bash -el {0}

env:
PKG_CONFIG_PATH: "${{env.PYTHONHOME}}/numpy/_core/lib/pkgconfig"

jobs:
ubuntu:
runs-on: ${{ matrix.platform }}
Expand Down Expand Up @@ -244,10 +247,10 @@ jobs:
run: |
/opt/python/cp313-cp313/bin/python -m venv ~/virtualenvs/pandas-dev
. ~/virtualenvs/pandas-dev/bin/activate
python -m pip install --no-cache-dir -U pip wheel setuptools meson[ninja]==1.2.1 meson-python==0.13.1
python -m pip install --no-cache-dir -U pip wheel setuptools meson[ninja]==1.4.0 meson-python==0.13.1
python -m pip install numpy -Csetup-args="-Dallow-noblas=true"
python -m pip install --no-cache-dir versioneer[toml] cython python-dateutil pytest>=7.3.2 pytest-xdist>=3.4.0 hypothesis>=6.84.0
python -m pip install --no-cache-dir --no-build-isolation -e . -Csetup-args="--werror"
python -m pip install --no-cache-dir --no-build-isolation -e .
python -m pip list --no-cache-dir
PANDAS_CI=1 python -m pytest -m 'not slow and not network and not clipboard and not single_cpu' pandas --junitxml=test-data.xml
concurrency:
Expand Down Expand Up @@ -282,9 +285,9 @@ jobs:
run: |
/opt/python/cp313-cp313/bin/python -m venv ~/virtualenvs/pandas-dev
. ~/virtualenvs/pandas-dev/bin/activate
python -m pip install --no-cache-dir -U pip wheel setuptools meson-python==0.13.1 meson[ninja]==1.2.1
python -m pip install --no-cache-dir -U pip wheel setuptools meson-python==0.13.1 meson[ninja]==1.4.0
python -m pip install --no-cache-dir versioneer[toml] cython numpy python-dateutil pytest>=7.3.2 pytest-xdist>=3.4.0 hypothesis>=6.84.0
python -m pip install --no-cache-dir --no-build-isolation -e . -Csetup-args="--werror"
python -m pip install --no-cache-dir --no-build-isolation -e .
python -m pip list --no-cache-dir

- name: Run Tests
Expand Down Expand Up @@ -353,10 +356,10 @@ jobs:
- name: Build Environment
run: |
python --version
python -m pip install --upgrade pip setuptools wheel meson[ninja]==1.2.1 meson-python==0.13.1
python -m pip install --upgrade pip setuptools wheel meson[ninja]==1.4.0 meson-python==0.13.1
python -m pip install --pre --extra-index-url https://pypi.anaconda.org/scientific-python-nightly-wheels/simple numpy
python -m pip install versioneer[toml] python-dateutil tzdata cython hypothesis>=6.84.0 pytest>=7.3.2 pytest-xdist>=3.4.0 pytest-cov
python -m pip install -ve . --no-build-isolation --no-index --no-deps -Csetup-args="--werror"
python -m pip install -ve . --no-build-isolation --no-index --no-deps
python -m pip list

- name: Run Tests
Expand Down Expand Up @@ -395,10 +398,10 @@ jobs:
- name: Build Environment
run: |
python --version
python -m pip install --upgrade pip setuptools wheel numpy meson[ninja]==1.2.1 meson-python==0.13.1
python -m pip install --upgrade pip setuptools wheel numpy meson[ninja]==1.4.0 meson-python==0.13.1
python -m pip install --pre --extra-index-url https://pypi.anaconda.org/scientific-python-nightly-wheels/simple cython
python -m pip install versioneer[toml] python-dateutil pytz tzdata hypothesis>=6.84.0 pytest>=7.3.2 pytest-xdist>=3.4.0 pytest-cov
python -m pip install -ve . --no-build-isolation --no-index --no-deps -Csetup-args="--werror"
python -m pip install -ve . --no-build-isolation --no-index --no-deps
python -m pip list

- name: Run Tests
Expand Down
8 changes: 8 additions & 0 deletions .github/workflows/wheels.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,10 @@ jobs:
with:
python-version: '3.11'

- name: Determine NumPy pkg-config location
run: |
echo "PKG_CONFIG_PATH=$(python -c 'import site; print(site.getsitepackages()[0])')/numpy/_core/lib/pkgconfig" >> $GITHUB_ENV

- name: Build sdist
run: |
python -m pip install build
Expand Down Expand Up @@ -140,6 +144,10 @@ jobs:
shell: bash -el {0}
run: echo "sdist_name=$(ls ./dist)" >> "$GITHUB_ENV"

- name: Determine NumPy pkg-config location
run: |
echo "PKG_CONFIG_PATH=$(python -c 'import site; print(site.getsitepackages()[0])')/numpy/_core/lib/pkgconfig" >> $GITHUB_ENV

# Python version used to build sdist doesn't matter
# wheel will be built from sdist with the correct version
- name: Unzip sdist (macOS)
Expand Down
2 changes: 1 addition & 1 deletion ci/deps/actions-310-minimum_versions.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ dependencies:
# build dependencies
- versioneer
- cython>=0.29.33
- meson=1.2.1
- meson=1.4.0
- meson-python=0.13.1

# test dependencies
Expand Down
2 changes: 1 addition & 1 deletion ci/deps/actions-310.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ dependencies:
# build dependencies
- versioneer
- cython>=0.29.33
- meson=1.2.1
- meson=1.4.0
- meson-python=0.13.1

# test dependencies
Expand Down
2 changes: 1 addition & 1 deletion ci/deps/actions-311-downstream_compat.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ dependencies:
# build dependencies
- versioneer
- cython>=0.29.33
- meson=1.2.1
- meson=1.4.0
- meson-python=0.13.1

# test dependencies
Expand Down
2 changes: 1 addition & 1 deletion ci/deps/actions-311-numpydev.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ dependencies:

# build dependencies
- versioneer
- meson=1.2.1
- meson=1.4.0
- meson-python=0.13.1
- cython>=0.29.33

Expand Down
2 changes: 1 addition & 1 deletion ci/deps/actions-311-pyarrownightly.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ dependencies:

# build dependencies
- versioneer
- meson=1.2.1
- meson=1.4.0
- cython>=0.29.33
- meson-python=0.13.1

Expand Down
2 changes: 1 addition & 1 deletion ci/deps/actions-311.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ dependencies:
# build dependencies
- versioneer
- cython>=0.29.33
- meson=1.2.1
- meson=1.4.0
- meson-python=0.13.1

# test dependencies
Expand Down
2 changes: 1 addition & 1 deletion ci/deps/actions-312.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ dependencies:
# build dependencies
- versioneer
- cython>=0.29.33
- meson=1.2.1
- meson=1.4.0
- meson-python=0.13.1

# test dependencies
Expand Down
2 changes: 1 addition & 1 deletion ci/deps/actions-pypy-39.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ dependencies:
# build dependencies
- versioneer
- cython>=0.29.33
- meson=1.2.1
- meson=1.4.0
- meson-python=0.13.1

# test dependencies
Expand Down
1 change: 1 addition & 0 deletions doc/source/whatsnew/v3.0.0.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,7 @@ Other enhancements
- Support passing a :class:`Iterable[Hashable]` input to :meth:`DataFrame.drop_duplicates` (:issue:`59237`)
- Support reading Stata 102-format (Stata 1) dta files (:issue:`58978`)
- Support reading Stata 110-format (Stata 7) dta files (:issue:`47176`)
- Use Meson's native NumPy dependency resolution method for building (:issue:`61095`)

.. ---------------------------------------------------------------------------
.. _whatsnew_300.notable_bug_fixes:
Expand Down
2 changes: 1 addition & 1 deletion environment.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ dependencies:
# build dependencies
- versioneer
- cython~=3.0.5
- meson=1.2.1
- meson=1.4.0
- meson-python=0.13.1

# test dependencies
Expand Down
4 changes: 2 additions & 2 deletions meson.build
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,15 @@ project(
'cython',
version: run_command(['generate_version.py', '--print'], check: true).stdout().strip(),
license: 'BSD-3',
meson_version: '>=1.2.1',
meson_version: '>=1.4.0',
default_options: ['buildtype=release', 'c_std=c11', 'warning_level=2'],
)

fs = import('fs')
py = import('python').find_installation(pure: false)
tempita = files('generate_pxi.py')
versioneer = files('generate_version.py')

numpy_dep = dependency('numpy', method: 'auto')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should use required: false as long as numpy 1.26.x is still supported


add_project_arguments('-DNPY_NO_DEPRECATED_API=0', language: 'c')
add_project_arguments('-DNPY_NO_DEPRECATED_API=0', language: 'cpp')
Expand Down
9 changes: 7 additions & 2 deletions pandas/_libs/meson.build
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,12 +149,17 @@ if get_option('buildtype') == 'debug'
endif

foreach ext_name, ext_dict : libs_sources
dependencies = [numpy_dep]
if ext_dict.has_key('deps')
dependencies += ext_dict.get('deps')
endif

py.extension_module(
ext_name,
ext_dict.get('sources'),
cython_args: cython_args,
include_directories: [inc_np, inc_pd],
dependencies: ext_dict.get('deps', ''),
include_directories: [inc_pd],
dependencies: dependencies,
subdir: 'pandas/_libs',
install: true,
)
Expand Down
9 changes: 7 additions & 2 deletions pandas/_libs/tslibs/meson.build
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,12 +29,17 @@ if get_option('buildtype') == 'debug'
endif

foreach ext_name, ext_dict : tslibs_sources
dependencies = [numpy_dep]
if ext_dict.has_key('deps')
dependencies += ext_dict.get('deps')
endif

py.extension_module(
ext_name,
ext_dict.get('sources'),
cython_args: cython_args,
include_directories: [inc_np, inc_pd],
dependencies: ext_dict.get('deps', ''),
include_directories: [inc_pd],
dependencies: dependencies,
subdir: 'pandas/_libs/tslibs',
install: true,
)
Expand Down
6 changes: 4 additions & 2 deletions pandas/_libs/window/meson.build
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@ py.extension_module(
'aggregations',
['aggregations.pyx'],
cython_args: ['-X always_allow_keywords=true'],
include_directories: [inc_np, inc_pd],
include_directories: [inc_pd],
dependencies: [numpy_dep],
subdir: 'pandas/_libs/window',
override_options: ['cython_language=cpp'],
install: true,
Expand All @@ -12,7 +13,8 @@ py.extension_module(
'indexers',
['indexers.pyx'],
cython_args: ['-X always_allow_keywords=true'],
include_directories: [inc_np, inc_pd],
include_directories: [inc_pd],
dependencies: [numpy_dep],
subdir: 'pandas/_libs/window',
install: true,
)
Expand Down
21 changes: 0 additions & 21 deletions pandas/meson.build
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,24 +1,3 @@
incdir_numpy = run_command(
py,
[
'-c',
'''
import os
import numpy as np
try:
# Check if include directory is inside the pandas dir
# e.g. a venv created inside the pandas dir
# If so, convert it to a relative path
incdir = os.path.relpath(np.get_include())
except Exception:
incdir = np.get_include()
print(incdir)
''',
],
check: true,
).stdout().strip()

inc_np = include_directories(incdir_numpy)
inc_pd = include_directories('_libs/include')

fs.copyfile('__init__.py')
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# See https://github.com/scipy/scipy/pull/12940 for the AIX issue.
requires = [
"meson-python>=0.13.1",
"meson>=1.2.1,<2",
"meson>=1.4.0,<2",
"wheel",
"Cython~=3.0.5", # Note: sync with setup.py, environment.yml and asv.conf.json
# Force numpy higher than 2.0rc1, so that built wheels are compatible
Expand Down
2 changes: 1 addition & 1 deletion requirements-dev.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
pip
versioneer[toml]
cython~=3.0.5
meson[ninja]==1.2.1
meson[ninja]==1.4.0
meson-python==0.13.1
pytest>=7.3.2
pytest-cov
Expand Down
Loading