Skip to content

Conversation

franzpoeschel
Copy link
Contributor

@franzpoeschel franzpoeschel commented May 15, 2025

This is an attempt at finding a solution for this issue ComputationalRadiationPhysics/picongpu#5355

Problem: If a downstream code wants to use nlohmann_json / toml11 as well, then it must use the same version used also in openPMD, otherwise it might happen that the same symbol defined by toml11/nlohmann_json might exist in different implementations (different versions) in libopenPMD.so and the compiled binary of the downstream. The linker might then jump between implementations, causing undefined behavior.

Attempted solution in this PR:

  1. By default, do not use internally shipped dependencies, but look for an installation of toml11/nlohmann_json in the system. (not yet implemented)
  2. If using an internally shipped dependency, then install it along with openPMD such that these versions become available in the software environment.
  3. Downstream codes can then use the same logic to agree on common versions of toml11/nlohmann_json.

Alternatives:

  • Maybe include toml11 / nlohmann_json into their own namespaces in openPMD instead?

TODO:

  • Look for nlohmann_json/toml11 in the system software environment by default
  • Implement this logic also for workflows using openPMD_toml11_src / openPMD_json_src
  • Add a flag to opt-out from this. (Yes, opt-out, this can only be a viable solution for the described issue if we can assume that an openPMD installation will share its versions of toml11 / nlohmann_json by default)
  • Open PRs at toml11/nlohmann_json for binary symbol versioning with inline namespace

@franzpoeschel franzpoeschel added bug discussion dependencies Pull requests that update a dependency file labels May 15, 2025
@franzpoeschel franzpoeschel added this to the 0.16.2 milestone May 15, 2025
@franzpoeschel franzpoeschel force-pushed the cmake_install_internal_dependencies branch from 0a0ef54 to c3d0e0d Compare May 15, 2025 14:16
@psychocoderHPC
Copy link
Member

@franzpoeschel Please take care that dependency fetching from the web is on some systems not possible due to restrictions that login nodes or compute nodes are not allowed to download things from the web.
@chillenzer has experience with that because he solved the issue already in https://github.com/alpaka-group/mallocMC

@franzpoeschel
Copy link
Contributor Author

@franzpoeschel Please take care that dependency fetching from the web is on some systems not possible due to restrictions that login nodes or compute nodes are not allowed to download things from the web. @chillenzer has experience with that because he solved the issue already in https://github.com/alpaka-group/mallocMC

Dependency fetching is already part of our build system, this PR does not change that, but rather adds an install step. Offline builds are fully supported (and we use them for distribution via package managers), see CMake variables:

openPMD_USE_INTERNAL_TOML11
openPMD_USE_INTERNAL_JSON
openPMD_toml11_src
openPMD_json_src

Comment on lines +1080 to +1086
install(CODE "
execute_process(
COMMAND ${CMAKE_COMMAND} \
--build \"${CMAKE_BINARY_DIR}\" \
--target fetchednlohmann_json-install
)
")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So far, I have intentionally turned the install of sub-projects off. Instead of this code, we could set JSON_Install ON:

https://github.com/nlohmann/json/blob/v3.12.0/CMakeLists.txt#L48

Comment on lines +1089 to +1093
install(CODE "
execute_process(
COMMAND ${CMAKE_COMMAND} \
--build \"${CMAKE_BINARY_DIR}\" \
--target fetchedtoml11-install
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TOML11_INSTALL is the equivalent option there:
https://github.com/ToruNiina/toml11/blob/v4.4.0/CMakeLists.txt#L27-L28

Comment on lines +1080 to +1086
install(CODE "
execute_process(
COMMAND ${CMAKE_COMMAND} \
--build \"${CMAKE_BINARY_DIR}\" \
--target fetchednlohmann_json-install
)
")

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Different question: Why would you hack together a manual installation here? Isn't it conceptually closer to what you want to express to link to json/toml11 publicly?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See Axel's comment above; this did not work because I was not aware about CMake options in nlohmann-json/toml11 which we set to suppress installations.
I'm only leaving this PR open rn because we might in future add nlohmann-json/toml11 functionality to our public API, in which case we would need to install these libraries somehow along with the main library. There is at the moment no concrete plan to work on this further.

@franzpoeschel
Copy link
Contributor Author

Closing this for now, might reuse the code later.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug dependencies Pull requests that update a dependency file discussion

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants