Experiemental build env backend by pypa/build

[uranusjr]

The current build backend is moved to a subpackage, and the new one can be enabled with --use-feature=build.

[pradyunsg]

Oh fun, I was working on getting #10720 working locally, with the logic moved behind a flag. I guess I should push that so that we can share each other's logic and pick the best parts. :)

[uranusjr]

This one’s partly done as an exercise so I can have a better idea how build’s environment population logic fits into pip. And my conclusion after finishing it is… not very well. It’s doable, but the amount of glue code doesn’t seem worthwhile. I’m currently leaning toward pip should still create the build environment with its own logic (your PR), and run build on top of that instead.

[pradyunsg]

Hmm... I see OverlayBuildEnvironment which is basically mirroring the existing overlay / normal separation.

I do think we can/should get rid of that separation, since it's not particularly meaningful IMO -- and that's something I think we should do as a part of this change.

run build on top of that instead.

I'm not sure that there's not much value in that -- as far as I understand, build is basically environment isolation + a call to pep517 (which will soon become pyproject-hooks). At that point, we might as well call the hooks ourselves instead of doing that via build.

[uranusjr]

In that case it’s probably best if we don’t go through build at all. build’s environment creation is also only a thin layer on virtualenv and venv that pip can easily replicate.

[pradyunsg]

I do think there'd be value in moving to an implementation that aligns our virtual environment creation with build's default w/ venv, and having those two use the exact-same underlying code (i.e. moving that code into build, and using it from there).

[LecrisUT]

Any luck on resurrecting this PR? This one seems like the most self-contained change. There are so many issues that pop up due to the fake-venv, that I hope the default would be moved to the proper venv-ed environment, and only give a flag for the old method when it fails.

[ichard26]

Any luck on resurrecting this PR?

Sadly no. Replacing the build isolation mechanism is a major task with considerable compatibility concerns. That means only a pip maintainer would likely be able to implement the necessary rewrite. None of the active maintainers have this issue on their radar, and even fewer have the necessary experience/expertise to make changes in this area.

For the time being, you should use uv which AFAIK creates a proper virtual environment to isolate the build.