Skip to content

CMake: Install toml11 and nlohmann_json for inclusion in downstream code #1757

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 10 commits into
base: dev
Choose a base branch
from

Conversation

franzpoeschel
Copy link
Contributor

@franzpoeschel franzpoeschel commented May 15, 2025

This is an attempt at finding a solution for this issue ComputationalRadiationPhysics/picongpu#5355

Problem: If a downstream code wants to use nlohmann_json / toml11 as well, then it must use the same version used also in openPMD, otherwise it might happen that the same symbol defined by toml11/nlohmann_json might exist in different implementations (different versions) in libopenPMD.so and the compiled binary of the downstream. The linker might then jump between implementations, causing undefined behavior.

Attempted solution in this PR:

  1. By default, do not use internally shipped dependencies, but look for an installation of toml11/nlohmann_json in the system. (not yet implemented)
  2. If using an internally shipped dependency, then install it along with openPMD such that these versions become available in the software environment.
  3. Downstream codes can then use the same logic to agree on common versions of toml11/nlohmann_json.

Alternatives:

  • Maybe include toml11 / nlohmann_json into their own namespaces in openPMD instead?

TODO:

  • Look for nlohmann_json/toml11 in the system software environment by default
  • Implement this logic also for workflows using openPMD_toml11_src / openPMD_json_src
  • Add a flag to opt-out from this. (Yes, opt-out, this can only be a viable solution for the described issue if we can assume that an openPMD installation will share its versions of toml11 / nlohmann_json by default)
  • Open PRs at toml11/nlohmann_json for binary symbol versioning with inline namespace

@franzpoeschel franzpoeschel added bug discussion dependencies Pull requests that update a dependency file labels May 15, 2025
@franzpoeschel franzpoeschel added this to the 0.16.2 milestone May 15, 2025
@franzpoeschel franzpoeschel force-pushed the cmake_install_internal_dependencies branch from 0a0ef54 to c3d0e0d Compare May 15, 2025 14:16
@psychocoderHPC
Copy link
Member

@franzpoeschel Please take care that dependency fetching from the web is on some systems not possible due to restrictions that login nodes or compute nodes are not allowed to download things from the web.
@chillenzer has experience with that because he solved the issue already in https://github.com/alpaka-group/mallocMC

@franzpoeschel
Copy link
Contributor Author

@franzpoeschel Please take care that dependency fetching from the web is on some systems not possible due to restrictions that login nodes or compute nodes are not allowed to download things from the web. @chillenzer has experience with that because he solved the issue already in https://github.com/alpaka-group/mallocMC

Dependency fetching is already part of our build system, this PR does not change that, but rather adds an install step. Offline builds are fully supported (and we use them for distribution via package managers), see CMake variables:

openPMD_USE_INTERNAL_TOML11
openPMD_USE_INTERNAL_JSON
openPMD_toml11_src
openPMD_json_src

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug dependencies Pull requests that update a dependency file discussion
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants