Building with cross compilation

[Ryan Taylor (rtayl)]

I'm building with cross-compilation trying to use the CPU backend for a riscv machine but I'm getting some error:

/usr/include/python3.10/pyconfig.h:88:14: fatal error: 'riscv64-linux-gnu/python3.10/pyconfig.h' file not found
88 | # include <riscv64-linux-gnu/python3.10/pyconfig.h>
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[Nhat Nguyen (nhat-nguyen)]

Ryan Taylor (@rtayl) Thanks for reporting! Would you mind providing us with step-by-step instructions on how to reproduce the issue? We have yet tested the project in any cross-compilation scenarios so it would be good to understand what we are missing here.

[Ryan Taylor (rtayl)]

Ryan Taylor (@rtayl) Thanks for reporting! Would you mind providing us with step-by-step instructions on how to reproduce the issue? We have yet tested the project in any cross-compilation scenarios so it would be good to understand what we are missing here.

Nhat Nguyen (@nhat-nguyen) I'm building it like a normal cross-compile. So I add sysroot dir path and toolchain dir path to CMAKE_C_FLAGS and CMAKE_CXX_FLAGS in CMakeCache.txt. Then to build: TRITON_BUILD_WITH_CLANG_LLD=true TRITON_BUILD_WITH_CCACHE=true python3 -m pip install --no-build-isolation -vvv '.[tests]'

I was looking at this: https://stackoverflow.com/questions/33512541/cross-compiling-gdb-for-arm-with-python-failed
and it looks like:

pyconfig.h sets preprocessor defines to tell the python source code about platform-depended things like endianness and data type sizes, so the proper pyconfig.h is definitely needed to correctly compile python source

So I'll try to grab an appropriate pyconfig.h for riscv, not sure where to get that, I'll look around, I would think this would be included in any native riscv build.

If anyone here has any other suggestions or input that would be appreciated.

[Ryan Taylor (rtayl)]

So update, I was able to get a riscv pyconfig.h and put it in the appropriate spot to get past the error above. However, I now have an error about being unable to build wheel?

`error: subprocess-exited-with-error

× Building wheel for triton (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
full command: /usr/bin/python3 /home/me/.local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_wheel /tmp/tmpfwcc8bld
cwd: /home/me/triton_shared/triton/python
Building wheel for triton (pyproject.toml) ... error
ERROR: Failed building wheel for triton
Failed to build triton
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (triton)
Exception information:
Traceback (most recent call last):
File "/home/me/.local/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 179, in exc_logging_wrapper
status = run_func(*args)
File "/home/me/.local/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 67, in wrapper
return func(self, options, args)
File "/home/me/.local/lib/python3.10/site-packages/pip/_internal/commands/install.py", line 432, in run
raise InstallationError(
pip._internal.exceptions.InstallationError: ERROR: Failed to build installable wheels for some pyproject.toml based projects (triton)
Remote version of pip: 24.1.2
Local version of pip: 24.1.2
Was pip installed by pip? True
Removed build tracker: '/tmp/pip-build-tracker-7merz9ux'
`
When I try to reproduce the issue using the command line:

me@me-desk:~/triton_shared/triton/python$ /usr/bin/python3 /home/rtaylor/.local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_wheel /tmp/tmpfwcc8bld Traceback (most recent call last): File "/home/me/.local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module> main() File "/home/me/.local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 331, in main hook_input = read_json(pjoin(control_dir, 'input.json')) File "/home/me/.local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 36, in read_json with open(path, encoding='utf-8') as f: FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpfwcc8bld/input.json' me@me-desk:~/triton_shared/triton/python$ pip install wheel Defaulting to user installation because normal site-packages is not writeable Requirement already satisfied: wheel in /usr/lib/python3/dist-packages (0.37.1) me@me-desk:~/triton_shared/triton/python$
EDIT: It seems like there are a lot of native paths in the build and it would require significant work to get this to cross compile.

[Nhat Nguyen (nhat-nguyen)]

Thanks for providing the updates. I have not had time to try the repro steps. But it just occurred to me that since we're entirely dependent on the triton project, isn't this more of a question of whether triton supports cross-compilation? If so, I think you would be able to get more support over there: https://github.com/triton-lang/triton

[Ryan Taylor (rtayl)]

Nhat Nguyen (@nhat-nguyen) Yes, I suppose this is true, however, I doubt they will care as they don't openly support CPU.