[Feature Request] Optionally build wheels with bdist_wheel instead of pip

[Time0o]

I'm in the following situation: I'm trying to build a a C++/Python project structured like this:

├── CMakeLists.txt
├── include
├── source
└── python
    ├── setup.py
    └── ...

i.e. Python code and bindings are separated from the main C++ source code.

Now, I'm unable to to build this project with cibuildwheels because when pip wheel ... is executed from the package directory (python in this case), pip copies the contents of that directory to a temporary directory before running setup.py. At this point there is no clean way (it seems) of referencing my project's root directory from setup.py anymore so the toplevel CMakeLists.txt can't be found and the build fails.

I have previously used multibuild which offers the option of building with python setup.py bdist_wheel instead, avoiding this problem. I think it would be sensible of making this possible with cibuildwhells as well by adding a command line option and modifying linux.py etc. Is this pull-request worthy?

EDIT: to clarify, my setup.py invokes CMake to build the C++ code as well as the Python bindings.

[Czaki]

As I understand your cmake call does not depend on the python version? In such a case the cmake call should be done before all build using CIBW_BEFORE_ALL
Usage of {project} is described in test command documentation but apply also to CIBW_BEFORE_ALL

Also, Your project structure does not allow to create, a source bundle because sdist cannot collect C++ source code. The problem with bdist_wheel is that it does not provide build isolation between python versions.

So if your cmake depends on the python version then make may not found that there is a need to recompile part of thew code.

[YannickJadoul]

As I understand your cmake call does not depend on the python version? In such a case the cmake call should be done before all build using CIBW_BEFORE_ALL

That doesn't seem like a great solution, in general.

I'm a bit surprised though that we would have never had this problem before. We have had #294/#295/#319. The original issue, #294, might be interesting to look at, though.

[joerick]

Perhaps this could be solved by disabling pip's build isolation?

CIBW_ENVIRONMENT: "PIP_NO_BUILD_ISOLATION=1"

[Time0o]

@Czaki I hope I understand this correctly, I'm not well versed in Python package distribution:

My project is built either by just using the toplevel CMakeLists.txt, in which case just the C++ code is built or by running setup.py which builds the C++ code + Python bindings. The latter runs CMake as part of the build_ext step (and passes PYTHON_EXECUTABLE=... to CMake in the process so it does depend on the Python version).

I'm currently only interested in binary distributions so I haven't given sdist any thought. I'm not sure I understand the build isolation part, wouldn't every invocation of python setup.py bdist_wheel create a unique (per Python version) wheel?

[Czaki]

My project is built either by just using the toplevel CMakeLists.txt, in which case just the C++ code is built or by running setup.py which builds the C++ code + Python bindings. The latter runs CMake as part of the build_ext step (and passes PYTHON_EXECUTABLE=... to CMake in the process so it does depend on the Python version).

So what in your code guarantee that python dependent parts will be recompiled if build without build isolation? Or you plan to create a separate cibuildwheel call per python version?

[Time0o]

My project is built either by just using the toplevel CMakeLists.txt, in which case just the C++ code is built or by running setup.py which builds the C++ code + Python bindings. The latter runs CMake as part of the build_ext step (and passes PYTHON_EXECUTABLE=... to CMake in the process so it does depend on the Python version).

So what in your code guarantee that python dependent parts will be recompiled if build without build isolation? Or you plan to create a separate cibuildwheel call per python version?

I'm still not sure I understand so I'll explain what I did when I just built wheels on manylinux "manually":

Inside the Docker container, for every Python installation under /opt/python, I ran /opt/python/.../bin/python setup.py bdist_wheel -d wheelhouse inside my python directory. As far as I can tell that built everything from scratch every time (maybe not the most efficient way to do it but my project is not that large) and created separate, per Python version, files under python/build and python/wheelhouse. Isn't that what would happen with cibuildwheel as well?

[Czaki]

And how did you clean the build result between python setup.py bdist_wheel -d wheelhouse calls?

cibuildwheel use build isolation to be sure that some artifacts from the previous build does not pollute the next wheel.

There are two options. Cleaning is built inside your setup script (or you use temporary dir for build in cmake) then

CIBW_ENVIRONMENT: "PIP_NO_BUILD_ISOLATION=1"

should be enough
or you need not add proper clean command inside

CIBW_BEFORE_BUILD: clean_command

[henryiii]

So what you really want is to be able to control the “.” in pip wheel .? Or the running directory of pip (though the . is the better thing to change I think so your output doesn’t get nested).

[Time0o]

I think my main problem is that setuptools isn't really meant to work with files not included in the build directory and its subdirectories. So maybe I have to rethink what I'm doing. Alternatively it's possible to set TMPDIR=. to make pip place all build files in the cwd, similar to running bdist_wheel.

[YannickJadoul]

So what you really want is to be able to control the “.” in pip wheel .? Or the running directory of pip (though the . is the better thing to change I think so your output doesn’t get nested).

This is possible, though:

positional arguments:
  package_dir           Path to the package that you want wheels for. Must be
                        a subdirectory of the working directory. When set, the
                        working directory is still considered the 'project'
                        and is copied into the Docker container on Linux.
                        Default: the working directory.

https://cibuildwheel.readthedocs.io/en/stable/options/#command-line-options

(cfr. #294/#295/#319, mentioned above)

[henryiii]

A few points now that I'm at a keyboard:

• You should never build with python setup.py bdist_wheel. That is considered an implementation detail by the core devs, and may not do the right thing if you do not import setuptools first (pip injects setuptools's modifications directly before invoking bdist_wheel). The pip command should be flexible enough to do any build you need. python -m build should also, I think, but that's still young and we don't have support for it here yet.
  • I don't think this is the real issue, here, though.
• If you run cibuildwheel in the /python directory, cibuildwheel will mount the subfolder only in the docker run, so "PIP_NO_BUILD_ISOLATION=1"will not fix it (though, since you are using artifacts/code from outside the build directory, you will need that setting to use pip from inside the/python` directory)
• The ideal solution I think is to run pip wheel python instead of pip wheel ., but I don't think we have a hook for that. I'd verify it works locally first!
• Pip at least is designed to work with projects that are in a folder, though it's not common - most packages put setup.py at the top level so that pip install git+https://git.repo/some_repo.git works. But you can do pip install git+https://git.repo/some_repo.git#subdirectory=python - I've had to work with a package structured like that before. So I think setuptools might support it (though possibly with PIP_NO_BUILD_ISOLATION=1 / --no-build-isolation passed to pip).

[henryiii]

Ahh, @YannickJadoul points out we do have a hook for that. Yes, that sounds like exactly what you need.

[YannickJadoul]

Ahh, @YannickJadoul points out we do have a hook for that. Yes, that sounds like exactly what you need.

But that fixes the copying into Docker for manylinux, but not the issue with pip wheel only copying the directory with setup.py?

OK, maybe with what you said here:

• Pip at least is designed to work with projects that are in a folder, though it's not common - most packages put setup.py at the top level so that pip install git+https://git.repo/some_repo.git works. But you can do pip install git+https://git.repo/some_repo.git#subdirectory=python - I've had to work with a package structured like that before.

[henryiii]

PIP_NO_BUILD_ISOLATION fixes that issue, I believe.

[henryiii]

Actually, maybe it doesn't: #466 (comment)

Sounds like you still need a step where you collect the outer directories. You have access to everything inside the sdist command so you could collect what you need there. I believe it actually copies the SDist and unpacks it into a new directory. It is designed to catch broken SDists - you are not supposed to distribute without an SDist.

Or, as a last ditch effort, you could use build-backend = "setuptools.build_meta:__legacy__", which does not do the copy.