pkgconf can unintentionally break users that install into global python installation

[virtuald]

Now of course we can argue that users shouldn't do that and only use virtualenv, but... well here we are. RobotPy is a pretty small project, but it seems to me that if eventually something like numpy/scipy eventually adopts using pkgconf-pypi then a lot more users will run into this problem.

My goal for pkgconf-pypi is to only find pkgconf packages in the python ecosystem. However, since it provides a pkgconf executable, it's understandable that users would expect it to behave like a system-installed pkgconf. What to do?

The most obvious thing to me would be to provide a pkgconf-pypi executable and a pkgconf executable. The pkgconf executable is just a normal pkgconf executable that doesn't do anything special, and the pkgconf-pypi is our current pkgconf replacement script. Then the special behavior is something you have to explicitly opt into, and just becomes something that has to be configured for python wheel builders.

An alternative solution would be to only enable the special behavior if an environment variable or argument was set/used, but that seems like a Bad Idea.

I know meson can be told to use a different pkgconf executable in a cross.txt file... I'm not sure if there's another way to do it in pyproject.toml so that meson-python/hatch-meson could use that instead. cmake probably can be configured this way too?

[virtuald]

I still like my proposal to have this package provide pkgconf and pkgconf-pypi scripts, where pkgconf is unmodified and pkgconf-pypi provides the special behavior. This would clearly be a breaking change, but... since it hasn't been adopted widely, that seems unfortunate but ok?

If we did this, the documentation would need to be updated to tell users how to opt-in to the special behavior.

meson-python/hatch-meson

I think it would be good for meson-python to have builtin special support for this package. It already writes a 'native' file for meson (https://github.com/mesonbuild/meson-python/blob/cfcfb3554800d42c859d04309da2c1973c9f97ce/mesonpy/__init__.py#L776), so it could do one of the following:

• (my preference) Use importlib.metadata to check if this package is installed, and automatically add pkgconf-pypi to the native file
• Check for pkgconf-pypi in the path and add this to the native file (this seems bad)
• Provide a pyproject.toml setting to opt in, and if set check for pkgconf-pypi and add it to the native file
• ... users might already be able to opt in by using [tool.meson-python.args], but I don't think meson provides a command-line way to specify the pkgconf executable, only a PKG_CONFIG environment variable

scikit-build-core/cmake

... I don't know much about scikit-build but their documentation doesn't discuss pkg-config at all, so I assume that any special support would need to be added by users in their CMakelists.txt.

https://cmake.org/cmake/help/latest/module/FindPkgConfig.html seems to indicate that the pkg-config path can be set via an environment variable, but not otherwise (and also, https://cmake.org/cmake/help/latest/command/cmake_pkg_config.html#command:cmake_pkg_config seems to indicate that maybe they're trying to do their own implementation so that cmake doesn't depend on pkg-config at all?).

other build systems

I can't think of any that are commonly used to build native python wheels.

Downsides

If the special behavior is opt-in, people might not learn about it and use it. Which would be sad, but... meh.

[FFY00]

Now of course we can argue that users shouldn't do that and only use virtualenv, but... well here we are. RobotPy is a pretty small project, but it seems to me that if eventually something like numpy/scipy eventually adopts using pkgconf-pypi then a lot more users will run into this problem.

I still like my proposal to have this package provide pkgconf and pkgconf-pypi scripts, where pkgconf is unmodified and pkgconf-pypi provides the special behavior. This would clearly be a breaking change, but... since it hasn't been adopted widely, that seems unfortunate but ok?

What if we just disable the Python specific functionality when not running in a virtual environment, always yielding to whatever is in PATH? We could then provide a pkgconf-pypi script, sorta like you propose, for users that explicitly want the Python ecosystem enabled one.

I think straight out disabling the functionality will require adding special handling in a lot of places, but if we are running inside a virtual environment, then it should be safe to assume we want the semi-wrap, right?

Admitedly, I am a bit out of touch with the complex real-world scenarios where this is causing trouble, as I haven't run into them personally, so I think I should leave the final decision regarding this to you @virtuald.

[virtuald]

Hm. I like that idea.

The one problem I can think of is that on CI like github actions I don't think it's setting up a virtualenv? So you'd need to do something custom anyways.

[FFY00]

Then we look for the EXTERNALY-MANAGED file — it already solved this CI and Docker problem for us.