PyPi package doesn't install header files in the header file search path.

[jonathanunderwood]

Use case: I maintain a python package which includes an extension module, and which uses the py3c headers, which, up to now, I have bundled with the project. Now that version 1.0 of py3c is available on PyPi, I had hoped that it would be possible to stop bundling the py3c headers and add py3c to the install_requires of my packages setup.py.

Unfortunately, however, pip install py3c puts the headers in a place which is not on the header file search path:

$ ls venv/include/site/python3.6/py3c/
capsulethunk.h  compat.h    py3c.h      tpflags.h
comparison.h    fileshim.h  py3shims.h

That's from doing a pip install py3c with the virtualenv activated. You can see that this directory is not on the standard compiler search path for header files:

gcc -pthread -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python3.6m/py3c -I/home/jgu/Projects/python-lz4/venv/include -I/usr/include/python3.6m -c lz4/_version.c -o build/temp.linux-x86_64-3.6/lz4/_version.o
lz4/_version.c:38:10: fatal error: py3c/py3c.h: No such file or directory
 #include <py3c/py3c.h>

[encukou]

Do you know how this should be solved?

py3c uses the setuptools' headers argument.

[jonathanunderwood]

Do you know how this should be solved?

I don't. I spent a couple of hours yesterday trying to get to the bottom of it. Here's what I found out:

• The headers argument is inherited directly from distutils, and so setuptools just passes it on.
• Setuptools mangles data_files such that it's not possible to specify a directory for them to be installed in e.g. the files might end up in a wheel, and therefore unavailable to compiler.

The only solution I could see was:

• Don't use setuptools
• Abuse distutils data_files to somehow put the headers in the right place. But that's tricky, when you think that the package might be installed into a virtual env. So, you need to somehow query for the correct path at package install time.

It seems like this use case just isn't accommodated with the tools we have today, alas.

[encukou]

@xoviat, if you have a bit of time, could you sketch how you're using the PyPI package? It seems we're missing something here.

from sysconfig import get_paths
from wheel.paths import get_install_paths

def get_wheel_header_paths():
    return [os.path.dirname(get_install_paths('dummy')['headers'])]

py3c_include_dirs = get_wheel_header_paths() + \
    [get_paths()['include']] + \
    [get_paths()['platinclude']]

[mtelka]

I think the main problem here is that make install (the preferred way how to install py3c by py3c authors) does something different than python setup.py install (this is something that pip install very likely does).

The make install puts py3c.h in the well known include path and all other headers into the py3c/ subdir. Something like this:

$ find /usr/include/py3c* -type f | sort
/usr/include/py3c.h
/usr/include/py3c/capsulethunk.h
/usr/include/py3c/comparison.h
/usr/include/py3c/compat.h
/usr/include/py3c/fileshim.h
/usr/include/py3c/py3shims.h
/usr/include/py3c/tpflags.h
$

It also installs the py3c.pc file which suggests that to find py3c you should use -I/usr/include (which is usual default, so py3c.pc is usually just noop).

OTOH, python setup.py install puts all headers into the py3c/ subdir:

$ find /usr/include/py3c* -type f | sort
/usr/include/py3c/capsulethunk.h
/usr/include/py3c/comparison.h
/usr/include/py3c/compat.h
/usr/include/py3c/fileshim.h
/usr/include/py3c/py3c.h
/usr/include/py3c/py3shims.h
/usr/include/py3c/tpflags.h
$

The /usr/include/py3c/py3c.h file is not in the default include path and since python setup.py install does not produce the pkgconfig file (py3c.pc) there is no way how any external software could find py3c automatically. In this case the py3c.pc should suggest both -I/usr/include and -I/usr/include/py3c.

So I think the easiest solution, at least for now, is to teach python setup.py install how to install the pkgconfig file too. I'm not sure how exactly one could do that, but I know that pycairo provides install_pkgconfig setup.py subcommand which is (IIRC) called automatically by python setup.py install to install the .pc file.

[encukou]

python setup.py install (which should nowadays be python -m pip install .) installs (preferably) into a virtual environment, and that doesn't really have a concept of header file locations or pkgconfig.
make install installs system-wide, assuming a UNIX-style system (probably with some Linux-isms, unintentionally). Python packaging is not really meant for this kind of system integration.

IMO, the best solution is to ignore PyPI and treat py3c as a C library.

[mtelka]

I'm sorry, but pip install does not by default install into a virtual environment:

$ cd /tmp
$ wget https://github.com/encukou/py3c/archive/refs/tags/v1.4.tar.gz
$ tar xf v1.4.tar.gz
$ cd py3c-1.4
$ python -m pip install .
Defaulting to user installation because normal site-packages is not writeable
Processing /tmp/py3c-1.4
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: py3c
  Building wheel for py3c (setup.py) ... done
  Created wheel for py3c: filename=py3c-1.4-py2.py3-none-any.whl size=9399 sha256=86726f711842852d2a92388248550c20f183e0c8066299ec42ec621bad1f9bc1
  Stored in directory: /home/marcel/.cache/pip/wheels/87/88/2a/5ecb2637b4c8361d8d726048b40f1355dccf2690db6c1ac543
Successfully built py3c
Installing collected packages: py3c
Successfully installed py3c-1.4
$ find ~/.local/
/home/marcel/.local/
/home/marcel/.local/include
/home/marcel/.local/include/python3.9
/home/marcel/.local/include/python3.9/py3c
/home/marcel/.local/include/python3.9/py3c/tpflags.h
/home/marcel/.local/include/python3.9/py3c/comparison.h
/home/marcel/.local/include/python3.9/py3c/py3c.h
/home/marcel/.local/include/python3.9/py3c/capsulethunk.h
/home/marcel/.local/include/python3.9/py3c/compat.h
/home/marcel/.local/include/python3.9/py3c/py3shims.h
/home/marcel/.local/include/python3.9/py3c/fileshim.h
/home/marcel/.local/lib
/home/marcel/.local/lib/python3.9
/home/marcel/.local/lib/python3.9/site-packages
/home/marcel/.local/lib/python3.9/site-packages/py3c-1.4.dist-info
/home/marcel/.local/lib/python3.9/site-packages/py3c-1.4.dist-info/top_level.txt
/home/marcel/.local/lib/python3.9/site-packages/py3c-1.4.dist-info/REQUESTED
/home/marcel/.local/lib/python3.9/site-packages/py3c-1.4.dist-info/direct_url.json
/home/marcel/.local/lib/python3.9/site-packages/py3c-1.4.dist-info/RECORD
/home/marcel/.local/lib/python3.9/site-packages/py3c-1.4.dist-info/LICENSE.MIT
/home/marcel/.local/lib/python3.9/site-packages/py3c-1.4.dist-info/WHEEL
/home/marcel/.local/lib/python3.9/site-packages/py3c-1.4.dist-info/INSTALLER
/home/marcel/.local/lib/python3.9/site-packages/py3c-1.4.dist-info/METADATA
$ pip list --user
Package Version
------- -------
py3c    1.4
$ python -V
Python 3.9.16
$ pip --version
pip 23.0 from /usr/lib/python3.9/vendor-packages/pip (python 3.9)
$

[encukou]

pip install does not by default install into a virtual environment:

Right, although some see it as a historical mistake.

I meant that pip is designed to install to virtualenvs (among other things), so it can't assume a header path or pkgconfig location is available. The packaging formats made for pip/PyPI don't handle *nix system integration well.