[python-package] Vendor `libgomp.so` in Linux wheels for manylinux compliance?

[rdbisme]

Description

Hello. LightGBM depends on libgomp, but it doesn't embed it. Which means it's not really manylinux compliant considering that most docker images don't ship libgomp by default.

I understand that embedding OpenMP might be problematic because other softwares might depend on it and bundle it bringing duplication.

Maybe a dedicated wheel could be crafted to ship various libgomp such that other python projects might depend on it to avoid duplication but to still be fully manylinux compliant?

If this is to allow linking with optimized OpenMP libraries for HPC, I'd argue that in that case the wheel should be recompiled anyway from source to enable specific local optimizations anyway (maybe using something like spack).

Reproducible example

auditwheel -v show lightgbm-4.6.0-py3-none-manylinux_2_28_x86_64.whl
lightgbm-4.6.0-py3-none-manylinux_2_28_x86_64.whl is consistent with
the following platform tag: "linux_x86_64".

The wheel references external versioned symbols in these
system-provided shared libraries: libdl.so.2 with versions
{'GLIBC_2.2.5'}, libgomp.so.1 with versions {'OMP_1.0', 'GOMP_4.5',
'GOMP_4.0', 'GOMP_1.0'}, libgcc_s.so.1 with versions {'GCC_3.0'},
libpthread.so.0 with versions {'GLIBC_2.2.5'}, libm.so.6 with versions
{'GLIBC_2.2.5', 'GLIBC_2.27'}, libstdc++.so.6 with versions
{'GLIBCXX_3.4.11', 'GLIBCXX_3.4.5', 'CXXABI_1.3.9', 'GLIBCXX_3.4.14',
'GLIBCXX_3.4.9', 'GLIBCXX_3.4.18', 'CXXABI_1.3.5', 'GLIBCXX_3.4.22',
'GLIBCXX_3.4.15', 'CXXABI_1.3.7', 'CXXABI_1.3.8', 'CXXABI_1.3',
'CXXABI_1.3.3', 'GLIBCXX_3.4.19', 'GLIBCXX_3.4.21', 'GLIBCXX_3.4.20',
'GLIBCXX_3.4'}, libc.so.6 with versions {'GLIBC_2.14', 'GLIBC_2.3',
'GLIBC_2.38', 'GLIBC_2.32', 'GLIBC_2.17', 'GLIBC_2.2.5', 'GLIBC_2.4',
'GLIBC_ABI_DT_RELR', 'GLIBC_2.6', 'GLIBC_2.3.4', 'GLIBC_2.34'}

This constrains the platform tag to "manylinux_2_38_x86_64". In order
to achieve a more compatible tag, you would need to recompile a new
wheel from source on a system with earlier versions of these
libraries, such as a recent manylinux image.

The following external shared libraries are required by the wheel:
{
    "libgomp.so.1": "/lib/libgomp.so.1.0.0"
}

In order to achieve the tag platform tag 'manylinux_2_41_x86_64' the
following shared library dependencies will need to be eliminated:

libgomp.so.1

In order to achieve the tag platform tag 'manylinux_2_40_x86_64' the
following shared library dependencies will need to be eliminated:

libgomp.so.1

In order to achieve the tag platform tag 'manylinux_2_39_x86_64' the
following shared library dependencies will need to be eliminated:

libgomp.so.1

Environment info

Command(s) you used to install LightGBM

Additional Comments

[jameslamb]

Thanks for using LightGBM.

It is a very intentional choice to not vendor libgomp.so* in lightgbm Linux wheels, to avoid conflicts with other libraries at runtime.

See:

• Distributing LightGBM with OpenMP #3041
• [python-package] SegFault on MacOS when pytorch is installed #6595 (comment)
• Segment fault when used with FAISS on arm pytorch/pytorch#149201 (comment)

Many other projects do vendor it though, maybe it's something we should consider on Linux (I am very very skeptical of doing it on macOS). For example, some discussion around XGBoost vendoring it:

• Vendor libgomp in the manylinux Python wheel dmlc/xgboost#6461
• Hot fix for libgomp vendoring dmlc/xgboost#6482
• [CI] Use manylinux2010_x86_64 container to vendor libgomp dmlc/xgboost#6485

Maybe a dedicated wheel could be crafted to ship various libgomp such that other python projects might depend on it to avoid duplication but to still be fully manylinux compliant?

Yes, definitely a possible solution. BUT... not something we'd do in the LightGBM project. If that's done, it should be something that other OpenMP-using projects all agree on. Without that coordination, we'd still have duplication problems.

There is significant prior discussion on this:

• here: [python-package] SegFault on MacOS when pytorch is installed #6595 (comment)
• and outside of LightGBM: https://pypackaging-native.github.io/key-issues/native-dependencies/blas_openmp/

[jameslamb]

@rdbisme is there some specific issue that you encountered with lightgbm Linux wheels?

We'd consider changing the way the OpenMP dependency is managed here, but not just "because the manylinux standards say you should". As I said the current behavior is an intentional choice based on quite a lot of research and experience from user reports, not an oversight or accidental omission.

[rdbisme]

Thanks for your feedback @jameslamb.

The problem we had with our users is that they submitted some code which uses lightgbm, which once containerized, didn't work because the docker image was missing libgomp. The general docker runtime assumes to deal with manylinux compliant wheels, so it doesn't install any additional runtime dependencies.

My personal view on the matter is that manylinux wheels should work on a minimal python container, without implicit dependencies. And should be optimized for "batteries-included" instead of HPC brute performance.

For all the advanced uses (e.g. someone might want to use another OpenMP provider or implementation like MKL or custom tuned OpenMP) that should be covered with a full rebuild of the wheel which can then be customized to the local HPC infrastructure and not solved by the manylinux wheel. Potentially using tools like spackpm.

On the weird interactions on macOS or Windows, I'm not expert enough. I guess Windows doesn't ship openMP by default either? Does macOS?

For the duplication, I think having a Python package holding the .so and version it should allow multiple different packages to coordinate on it leveraging standard Python dependency resolution? I might try as a weekend project to set that wheel up... Where should it be discussed tho?

[jameslamb]

customized to the local HPC infrastructure

should be optimized for "batteries-included" instead of HPC brute performanc

The choice to not vendor libgomp.so in lightgbm is not at all about optimizing for different hardware. It's about portability and compatibility with other libraries.

I guess Windows doesn't ship openMP by default either? Does macOS?

Let's keep this issue focused on LightGBM's Linux wheels. Any examples that I shared which mention other operating systems are only there to show some ways that "multiple OpenMP libraries loaded into the same process" can cause issues.

I think having a Python package holding the .so and version it should allow multiple different packages to coordinate on it leveraging standard Python dependency resolution?

Yes, I already addressed this above and linked to previous discussions about it.

Where should it be discussed tho?

A new issue at https://github.com/pypackaging-native/pypackaging-native/issues or a post in the Packaging topic at https://discuss.python.org/ would probably be good places.

Please do read the things I linked and a few more links below and do some other research before posting there. A discussion that just says "what if we do this?" without addressing the known complexities is unlikely to generate much interest.

• @thomasjpfan 's comments here: [python-package] SegFault on MacOS when pytorch is installed #6595 (comment)
• https://pypackaging-native.github.io/key-issues/native-dependencies/blas_openmp/
• https://discuss.python.org/t/pep-725-specifying-external-dependencies-in-pyproject-toml-round-2/103890
• https://discuss.python.org/t/conflicting-binary-extensions-in-different-packages/25332/5
• https://discuss.python.org/t/pep-817-wheel-variants-beyond-platform-tags/105860

If you do start a discussion / implementation somewhere, please link it here.

I'm not trying to discourage you from doing this... a better solution for OpenMP would be very welcome for a lot of scientific Python libraries! I just want to be sure you understand that the question you're asking isn't a new one, and there's much prior work and discussion any proposed implementation will have to consider.

manylinux wheels should work on a minimal python container, without implicit dependencies

I do understand why it's annoying to have to apt-get install libomp-dev or similar for lightgbm. I'm open to vendoring libgomp.so in LightGBM's manylinux wheels!

I've moved this to feature request and added it to our feature request tracked (#2302)

Here's what I'd want convincing answers to before we did that:

• what are the licensing implications?
  • lightgbm is an MIT project, what would have to change if we started redistributing a binary copy of libgomp.so?
• how much would it affect binary size?
  • how much would libgomp.so add to the compressed size of lightgbm wheels?
  • if it's more than, say, 50 MB that'd make me more hesitant to adopt this, as PyPI by default limits projects to 100MiB per file and we want to stay as far below that as possible
• how would it change the risk of conflicts with other libraries?
  • maybe name-mangling with auditwheel repair could avoid some of the types of conflicts mentioned in [python-package] SegFault on MacOS when pytorch is installed #6595 but I'm not sure
  • how much more likely would adding yet another libgomp.so copy make problems like this on aarch64?

    • LightGBM/docs/FAQ.rst

      Line 209 in 4040edf

      17. Loading LightGBM fails like: ``cannot allocate memory in static TLS block``

    • [ci] [python-package] temporarily stop testing against scikit-learn nightlies, load lib_lightgbm earlier #6654 (comment)
    • (though maybe will not be as important if we upgrade to GLIBC 2.28 for aarch64, as mentioned in [ci] aarch64 python-package job failing: "cannot allocate memory in static TLS block" #6509 (comment), which I'm planning in WIP: [ci] move CI container images to GHCR, workflows to this repo #7109 anyway)