Failure to find python Interpreter and Development.Module running under yocto/oe

[sv662]

I am attempting to use scikit-build-core 0.10.7 to build python3-pyzmq bindings at 26.2.0 in a yocto environment.
There was no existing recipe within meta-openembedded yet, so I have attempted to create my own.
It is starting to build, but there appears to be a failure in the execution of cmake where it cannot locate python.

CMake Error at /src/platform/build/tmp/work/cortexa9t2hf-neon-xilinx-linux-gnueabi/python3-pyzmq/26.2.0/recipe-sysroot-native/usr/share/cmake-3.28/Modules/FindPackageHandleStandardArgs.cmake:230 (message): Could NOT find Python (missing: Interpreter Development.Module)

The environment for this recipe appears to be being setup correctly by bitbake. The local recipe native sysroot does have python3 in it and the PYTHON environment variables are being set to point at those locations, so I'm wondering what it is trying to look for that it is not finding. Python is this context is version 3.12.

The PYTHON env variable is set:
export PYTHON="/src/platform/build/tmp/work/cortexa9t2hf-neon-xilinx-linux-gnueabi/python3-pyzmq/26.2.0/recipe-sysroot-native/usr/bin/python3-native/python3"

... and that location does indeed have the python3 executable:
$ ls -l ./work/cortexa9t2hf-neon-xilinx-linux-gnueabi/python3-pyzmq/26.2.0/recipe-sysroot-native/usr/bin/python3-native/python3*
lrwxrwxrwx 1 10 Nov 26 16:24 ./work/cortexa9t2hf-neon-xilinx-linux-gnueabi/python3-pyzmq/26.2.0/recipe-sysroot-native/usr/bin/python3-native/python3 -> python3.12
-rwxr-xr-x 7 14360 Nov 20 15:59 ./work/cortexa9t2hf-neon-xilinx-linux-gnueabi/python3-pyzmq/26.2.0/recipe-sysroot-native/usr/bin/python3-native/python3.12

I'm attaching the run.do_compile and log.do_compile files from the build so you can see the environment and calls into the build for context.

Any assistance you could give would be gratefully received.

Does scikit-build-core fully pass the environment through to cmake or is there some additional setup that needs to be done?
Would the following command line args to python when running the build within pyzmq have any adverse effect on scikit-build-core?
-m build --no-isolation --wheel --outdir

log.do_compile.txt
run.do_compile.txt

[henryiii]

I'll try to look at this soon, probably after Monday.

[LecrisUT]

The thing I'm worried about with openembeded is getting the cross-compilation right and there are no contamination of packages. With FindPython you have the additional complication that you need to make sure the python of the bitbake executor is not used because afaiu you want to pick up the python.h and library of the runtime system.

I tried to look into it, but the hardest part is setting up the initial docker environment with openembeded, and a test environment. Could you help share a dockerfile with a minimal setup?

[sv662]

pyzmq_test.zip

Hi @LecrisUT

I don't know if this would be the sort of thing you were expecting, but I've put together a failing example. Uses a dockerfile and docker compose setup to create a build container based on ubuntu. The docker compose service will run the build.sh script which pulls all the meta layers using repo and kick off a bitbake using a new layer with the recipes I was trying to scikit-build-core and pyzmq. You can always override the command and just go in on a bash prompt and interact with it directly on the terminal after that inital build fails.

Removing the REQUIRED_VERSION line from local.conf would put it back to the version released in scarthgap.

This might not be the normal way to use docker, it's just how I know to set it up to build a project with source on the host machine mounted as a volume.

I think the only environment dependency would be setting env variables DOCKER_UID and DOCKER_GID to match your users id and groups like this:
export DOCKER_GID=$(id -g)
export DOCKER_UID=$(id -u)

To run this do the following:
unzip pyzmq_test.zip
cd pyzmq_test
docker compose build yocto_xlnx
docker compose run --rm yocto_xlnx

If that's not what you were expecting or you have any trouble with it, do let me know.

[coolo]

@sv662 I'm in the same boat as you and what helped kicking this further down the road was this patch:

@@ -0,0 +1,22 @@
+From 42573aad35732223bb1467cbfdd35a5081e80d33 Mon Sep 17 00:00:00 2001
+From: Stephan Kulow <stephan.kulow@siemens.com>
+Date: Thu, 19 Dec 2024 12:56:28 +0100
+Subject: [PATCH] fix: check for PYTHON_INCLUDE_DIR
+
+---
+ src/scikit_build_core/builder/builder.py | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/src/scikit_build_core/builder/builder.py b/src/scikit_build_core/builder/builder.py
+index 437265f..18d55b9 100644
+--- a/src/scikit_build_core/builder/builder.py
++++ b/src/scikit_build_core/builder/builder.py
+@@ -203,7 +203,7 @@ class Builder:
+         python_sabi_library = (
+             get_python_library(self.config.env, abi3=True) if limited_api else None
+         )
+-        python_include_dir = get_python_include_dir()
++        python_include_dir = os.getenv("PYTHON_INCLUDE_DIR") or get_python_include_dir()
+         numpy_include_dir = get_numpy_include_dir()
+ 
+         # Classic Find Python

I hope it helps to find a proper solution. At least I've got myself a pyzmq-26.2 package now :)

[henryiii]

🤖 AI text below 🤖

I dug into this with a faithful cross-compile reproduction (Debian + a real arm-linux-gnueabihf toolchain + multiarch libpython:armhf, using the same _PYTHON_SYSCONFIGDATA_NAME sysconfig redirection that Yocto's python3targetconfig sets up). I was able to reproduce the exact Could NOT find Python (missing: Interpreter Development.Module) error and isolate what's actually causing it.

Short version: the problem is the include-directory hint, not the interpreter.

What I found:

• The host Python_EXECUTABLE hint that scikit-build-core passes is not the culprit. As long as the target include dir is valid and version-consistent, FindPython accepts the host executable fine, in both native and CMAKE_CROSSCOMPILING modes.
• The missing: Interpreter Development.Module message is produced when the include dir hint is unusable — either it points somewhere without a Python.h, or the headers report a different Python version than the interpreter. When Development.Module can't resolve, FindPython drops Interpreter from the results too, which is why both show up as "missing".
• Under Yocto's target sysconfigdata, sysconfig.get_path("include") resolves to the on-device path (/usr/include/python3.12), which isn't staged at build time. scikit-build-core hands that to FindPython, and it has no Python.h.

That's exactly why @coolo's patch works: $PYTHON_INCLUDE_DIR carries the real staged recipe-sysroot/.../python3.12 path. Swapping it in made my repro configure cleanly. So the right fix is along those lines — honoring PYTHON_INCLUDE_DIR/PYTHON_LIBRARY from the environment when set.

Note that cmake.python-hints = false alone will not fix this: it drops all hints (including the include dir), and since this python -m build flow sets no toolchain file / CMAKE_FIND_ROOT_PATH pointing at recipe-sysroot, FindPython then has nothing valid to find.

To make sure I target the actual trigger (a missing/wrong include path vs. a native↔target Python version skew — both give the identical message), could you attach a few things from a failing do_compile?

1. …/build/CMakeFiles/CMakeError.log and the generated CMakeInit.txt (in the temp build dir scikit-build-core creates).
2. In the recipe's do_compile environment (with _PYTHON_SYSCONFIGDATA_NAME set, as in your run script):

   python3 -c "import sysconfig; print(sysconfig.get_path('include')); print(sysconfig.get_config_var('LIBDIR'))"

3. The Python version of python3-native vs. the target python3 (just to rule out a version skew).

Thanks for the detailed report and the repro — that made this much easier to chase down.