segmentation fault using new pulp version (+rust) with latest highspy in Ubuntu

[pchtsp]

While migrating the core of PuLP to Rust, I got an interesting error that I'm now sharing.

In windows all tests pass and everything looks reasonable. However, when I try to run the same code on Linux, I get a segmentation fault only when I try to use HiGHS as a solver through the python API (highspy). So PuLP in Rust + Linux + HiGHS python API = segmentation fault. It fails at the first test, any test.

Some expensive LLM agent finally managed to "solve it" in pulp, or more accurately find a workaround.It's possible to look for the esoteric fix here: https://github.com/coin-or/pulp/blob/rust_core/pulp/__init__.py. I have no idea if the fix makes sense. Do any of you?

[odow]

While migrating the core of PuLP to Rust

Complete aside: cool!

[pchtsp]

this was the conclusion the LLM agent reached and can be read in the link I shared:

# Pre-load highspy's C extension with RTLD_GLOBAL before any other imports.
# highspy._core uses C++ exception handling (__cxa_throw etc.) that must be globally
# visible when pulp._rustcore (a Rust/pyo3 extension) is later loaded into the same
# process. Without RTLD_GLOBAL, the C++ runtime state gets split across multiple
# symbol scopes, causing a segfault when highspy.Highs.run() is called.
# We use find_spec("highspy") (the package) not find_spec("highspy._core") (the
# submodule) to avoid triggering highspy/__init__.py which would import _core with
# RTLD_LOCAL before our explicit RTLD_GLOBAL call can take effect.
try:
    import ctypes as _ctypes
    import glob as _glob
    import importlib.util as _ilu

    _pkg_spec = _ilu.find_spec("highspy")
    if _pkg_spec is not None and _pkg_spec.submodule_search_locations:
        for _loc in _pkg_spec.submodule_search_locations:
            for _so in _glob.glob(_loc + "/_core*.so"):
                _ctypes.CDLL(_so, _ctypes.RTLD_GLOBAL)
                break
    del _ctypes, _glob, _ilu, _pkg_spec
except Exception:
    pass

[jajhall]

Does this mean anything to you, @galabovaa ?

[mathgeekcoder]

This is interesting! @pchtsp: I think you're right to call it a workaround instead of a fix (AFAIK it's usually better to use LOCAL for isolation, as GLOBAL can cause other issues down the line).

pybind11 should capture and wrap any C++ exceptions and so highspy._core library shouldn't expose any (in theory). We probably should build with visibility=hidden to reduce potential clash of exported symbols. Though this won't help backwards compatibility.

I'm guessing it's only an issue calling lp.solverModel.run()? i.e., can it build the model etc. in highspy? I'm trying to identify if it might have anything to do with threading/GIL.