best practice for including compiler version in abi for dependencies (pybind11/nanobind)

[minrk]

Your question:

The problem

For packages built with pybind11 and nanobind, the major version of the C++ compiler is part of the ABI. That means any package built with nanobind that links to another package via its nanobind API has a runtime requirement that both were built with the same major version of the compiler. This is not currently handled by the pybind11-abi metapackage or other compiler run_exports.

I don't think it comes up very often, but it does in the fenics stack. For fenics, I've added a fenics-basix-nanobind-abi package with a custom version that includes the nanobind and cxx_compiler versions. In the past, I had the same that bundled the pybind11-abi version and compiler version. I've only made one of these because everything in fenics depends on basix, but technically every package build with nanobind (that might be linked against in host) should have one. But this is a general problem, if not a common one. pybind11 has a pybind11-abi package, and nanobind will soon as well. But this is only half a solution, because it doesn't take the build-time compiler version into account, which is also part of the ABI in packages built with either pybind11 or nanobind.

My fenics-basix-nanobind-abi approach also doesn't solve the problem as well as I'd like because downstream packages build just fine since the compiler version is not actually restricted in the build env, but they won't import. that means e.g. cross-compiled outputs pass all CI when they are completely broken, when really the build should have been prevented in the first place due to the compiler version conflict.

Also, I've only dealt with this on linux/mac and don't actually know what needs to be restricted on Windows.

Question

I've struggled to come up with a general solution, but it seems like having nanobind or pybind11 in host dependencies should:

1. restrict the C++ compiler version at build time when in host requirements, and
2. restrict install-time packages to those built with the same C++ compiler

One hard part is that pybind11/nanobind packages are host dependencies, so can't easily restrict the cxx compiler version in the build environment. Is there a package in the host env that could be used as a proxy? Or should we be adding a metapackage to the build env to keep things in sync?

Does anyone have a recommendation/experience for a best practice for ensuring ABI compatibility that is sensitive to the compiler version? Is defining one of these abi metapackage outputs for every nanobind-built package really the way to go?

Possible solution

My only idea so far for a general solution is to update the pybind11/nanobind-abi packages (or separately add nanobind-cxx-abi) to generate a version for each compiler version with run_constrains on the compiler version and nanobind and strong run_exports on itself.

so e.g. nanobind-cxx-abi 0.13.12 gxx_* would have:

where:

• 0 is the versioning scheme (increment when we change how it works)
• 13 is the nanobind internals version
• 12 is cxx_compiler_version
• gxx is cxx_compiler

and would look something like:

run_exports:
  strong:
    - {{ pin_subpackage('nanobind-cxx-abi', max_pin='x.x.x.x') }} {{ cxx_compiler }}_*
run_constrained:
  # constrain compiler version (only works if it's in the build env)
  - {{ cxx_compiler }}_impl_{{ target_platform }} {{ cxx_compiler_version }}.*
  # constrain nanobind abi (host env)
  - nanobind-abi {{ nanobind_internals_version }}
  # another compiler constraint that would work on the host env? libstdcxx_devel?

and then packages make sure this is the same version in both the build and host requirements. But as convoluted as that is, I'm not even sure it would work.

[traversaro]

xref related issue on pybind11 side:

• pybind11 2.12.0 on Windows creates extensions that are not ABI compatible with extensions compiled with pybind11 2.11.0 and/or with a different version of MSVC pybind11-feedstock#95
• Does pybind11-abi shield from using different versions of GCC pybind11-feedstock#77

[minrk]

It looks like most of the discussion is around "try to get these packages to not include the compiler version in the abi" rather than accepting the current behavior and expressing the dependencies that they have.

[traversaro]

It looks like most of the discussion is around "try to get these packages to not include the compiler version in the abi" rather than accepting the current behavior and expressing the dependencies that they have.

Indeed, I cross-linked exactly to make sure that people looking to that discussions could find also this related discussion.

[carterbox]

pybind11 and nanobind are both header-only packages, right? As header-only packages, "host" doesn't really have any significance for them since the same artifacts are used regardless of the host/target platform.

They could be used as build dependencies instead of host dependencies so that they can directly interact with the compiler packages. If the solution requires moving these packages to the build section, we could add a linting rule / migration to the channel to have new builds move these packages there.

[beckermr]

The issue here is pass items at runtime between two different libraries compiled with these packages. The layout of objects in memory can differ and cause ABI issues.

[leofang]

cc @rwgk for vis

[rwgk]

Cross-referencing a relevant recent pybind11 PR:

pybind/pybind11#5439 (which in turns references pybind/pybind11#4953, pybind/pybind11#5375, pybind/pybind11#5296).

I'm not sure how exactly those relate to the discussion here, but I believe the latest pybind11 code on master encodes general C++ ABI compatibility as good as one can make it (caveats are pretty esoteric; see discussions under 4953).

The current pybind11 release (2.13.6) does NOT include 5439, 4953, 5375. I.e. if anyone here sees problems or has ideas for further improvements, please let me know.

[traversaro]

@rwgk Thanks a lot for all the work on pybind/pybind11#5439 ! From what understand that solves most of the issues with pybind11 used in the conda-forge context. To actually use it in conda-forge however, we would most probably need to wait for pybind11 2.14 release, there is already a plan/idea/estimate/roadmap for when that could be released?

[rwgk]

is already a plan/idea/estimate/roadmap for when that could be released?

Short answer: unfortunately not.

I just realized we're in some back and forth with #5450 (merged but unreleased) and #5486 (basically replacing 5450).

If it wasn't for that, I think we could release soon.

[traversaro]

Thanks for the reply and the work @rwgk ! It seems that pybind/pybind11#5486 was merged, so if there is anything else in which external contributors can help in getting a pybind11 release out (that will fix conda-forge/pybind11-feedstock#77, conda-forge/pybind11-feedstock#95, conda-forge/pybind11-feedstock#96 and unblock the numpy 2 migration for some pybind11-using feedstocks), feel free to ask.

[rwgk]

This small PR is meant to be a preparation for the next pybind11 release:

• feat: make PYBIND11_INTERNALS_VERSION 6 the default on all platforms. pybind/pybind11#5512

See the rationale in the PR description. Please let me know your thoughts.

[traversaro]

This small PR is meant to be a preparation for the next pybind11 release:

• feat: make PYBIND11_INTERNALS_VERSION 6 the default on all platforms. pybind/pybind11#5512

See the rationale in the PR description. Please let me know your thoughts.

The change would be great from the conda-forge side, as we could go back to have a single version of pybind11-abi package for each pybind11 version. Thanks a lot for working on this!