OSX Arm64 cross-compilation of pytorch extensions fails on conda-forge

[stefdoerr]

Solution to issue cannot be found in the documentation.

• I checked the documentation.

Issue

See related issue here:
pytorch/pytorch#146702

And failing PRs:
conda-forge/pytorch_scatter-feedstock#67
conda-forge/torchmd-net-feedstock#46

The culprit seems to be patch 0016 https://github.com/conda-forge/pytorch-cpu-feedstock/blob/main/recipe/patches/0016-point-include-paths-to-PREFIX-include.patch

2.5.1-cpu_mkl_py310hb8c326f_101    # this build worked in November 2024
2.5.1-cpu_mkl_py310_h3d4b842_111   # this one not anymore

Installed packages

See failing logs here:

https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=1169317&view=results

Environment info

See failing logs here:

https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=1169317&view=results

[h-vetinari]

Thanks for opening this issue, and sorry for the breakage. I'm copying a comment you made on the pytorch_scatter PR, because it's relevant

It's this third path getting set by pytorch cpp_extension which is messing things up. Although I cannot figure out how it can change between the first line or if something is modifying it later on. To me it looks as if the same path is added twice.

The last path comes from this feedstock

pytorch-cpu-feedstock/recipe/patches/0016-point-include-paths-to-PREFIX-include.patch

Lines 24 to 33 in be20390

	paths = [
	lib_include,
	# Remove this once torch/torch.h is officially no longer supported for C++ extensions.
	os.path.join(lib_include, 'torch', 'csrc', 'api', 'include'),
	+ # add site-packages/torch/include again (`lib_include` may have been pointing to
	+ # $PREFIX/include), as some torch-internal headers are still in this directory
	+ os.path.join(_TORCH_PATH, 'include'),
	]
	if cuda and IS_HIP_EXTENSION:
	paths.append(os.path.join(lib_include, 'THH'))

I had added this in e0e34e3, because it was necessary to make the torch.compile tests find the right headers in $PREFIX/include.

I needed to leave in the local (SP_DIR-driven) path in there as well, because

However, what didn't work was moving the headers to %LIBRARY_INC%. That's because the torch headers actually have a nested structure:

Lib/site-packages/torch/include/torch/csrc/api/include/torch/all.h
                        ^^^^^^^                ^^^^^^^
                     entry point 1          entry point 2

And just moving torch/csrc/api/include/torch/all.h to %LIBRARY_INC% will not resolve #include <torch/all.h>. Pytorch generally adds both paths to the includes, and so does the test infrastructure.

I don't understand how this works out on unix TBH (where we're moving the headers to $PREFIX/include, but not handling torch/csrc/api/include explicitly AFAICT). Maybe it somehow gets persisted in INTERFACE_INCLUDE_DIRECTORIES?

Anyway, I think I'll try patching this, unless someone has better ideas?

It's worth nothing that upstream pytorch - i.e. before the patch we apply here - does the moral equivalent of

    paths = [
        os.path.join(_TORCH_PATH, 'include'),
        os.path.join(_TORCH_PATH, 'include', 'torch', 'csrc', 'api', 'include'),
    ]

So whatever happened in the pytorch of the build environment would also look up the headers in the site-packages of build-prefix (which is symlinked to $BUILD_PREFIX/index). Now we end up mixing the two different paths to the environments, where in fact we only wanted to fix the $SP_DIR vs. $PREFIX/include situation within the same environment.

I think the simplest fix for now is to restrict that patch to windows (where we don't have symlinks, which make this work transparently on unix). There's another alternative fix I have in mind if we want to keep the patch universal, but AFAICT right now, that's not really necessary anyway.

[h-vetinari]

This should be fixed by #346, builds will be online in about 6h.

[stefdoerr]

Thank you sooo so much! 💟 cross-compilation working again now