Error using HiGHS solver in Python-MIP

[jerak]

Description
Hello, first of all I am not sure if this is an issue related to Highspy, so please let me know if that is indeed not the case.

I am trying to use the HiGHS-solver as an alternative to the CBC-solver that I previously used. I am using the Python-MIP wrapper (see: Python-MIP.

I am using a Windows machine, using Python 3.9.

To achieve this, I have done the following:

• Installed the latest Python-MIP version on master, i.e., the version that contains this fix: #377
• Installed highspy 1.9.0
• Installed cffi 1.17.0

AttributeError: function/symbol 'Highs_destroy' not found in library 
'~\venv\lib\site-packages\highspy\_core.cp39-win_amd64.pyd': error 0x7f

Operating System Information

• Operating System, version: Windows 11
• Python version: 3.9
• Python-MIP version (we recommend you to test with the latest version): latest master-version

Question
Did anyone using the HiGHS-solver already encounter this error before? Does anyone have suggestions on the steps to take?

Update
I also tried using different a different version of highspy: 1.7.1.dev1, but no luck.

[galabovaa]

@jerak, Thank you for getting in touch. Python-MIP is a separate package not supported by us and in the past months we have made some major updates. My initial guess would be that the Python-MIP highs wrapper may need to be modified to incorporate our latest changes.

Could you please try highspy 1.7.2 and let us know if that gives you an error?

[jerak]

Hi @galabovaa, thank you for your quick reply.

I installed highspy 1.7.2, but I unfortunately get the same error

[galabovaa]

OK, thank you for checking! I think for this one it is worth opening an issue in the Python-MIP repo, if you haven't already.

[jajhall]

I hope you get some response from Python-mip, but bear in mind what's said here

[rschwarz]

(Note: Since we're also affected by this, I'm using this issue to take notes from debugging and sharing potential solutions.)

I understand that you can not support python-mip, but I feel like this might be an issue related to the C API and how the DLL for HiGHS is built and/or distributed with highspy.

The function Highs_destroy is still part of the C API, and should be used as a destructor to properly free the memory allocated by Highs_create, right?

As such, it's used by other packages as well, including HiGHS.jl.

The way python-mip dynamically links to the HiGHS library might be to blame, but I assumed that using the library that comes with highspy should be safe.

On my linux notebook, with highspy==1.9.0, the functions are available:

env/lib/python3.12/site-packages/highspy$ objdump -T _core.cpython-312-x86_64-linux-gnu.so | grep "Highs_destroy"
0000000000171170 g    DF .text	000000000000003b  Base        Highs_destroy
env/lib/python3.12/site-packages/highspy$ objdump -T _core.cpython-312-x86_64-linux-gnu.so | grep "Highs_create"
0000000000171140 g    DF .text	000000000000002f  Base        Highs_create

It seems as if the same can not be said about the Windows version of highspy, although it's difficult for me to verify.

If I try a version of objdump from mingw, I get no match:

~/Downloads/highspy-1.10.0-cp313-cp313-win_amd64/highspy$ x86_64-w64-mingw32-objdump -t _core.cp313-win_amd64.pyd | grep Highs_destroy
~/Downloads/highspy-1.10.0-cp313-cp313-win_amd64/highspy$ x86_64-w64-mingw32-objdump -t _core.cp313-win_amd64.pyd | grep Highs_create

Since HiGHS.jl is working on Windows, it means that the issue above might be resolved by using the highs.dll as provided by HiGHS_jll.

This library does show the relevant functions:

~/Downloads/HiGHS.v1.10.0.x86_64-w64-mingw32-cxx11/bin$ x86_64-w64-mingw32-objdump -t libhighs.dll | grep Highs_create
[1052](sec  1)(fl 0x00)(ty   20)(scl   2) (nx 0) 0x000000000000d950 Highs_create
~/Downloads/HiGHS.v1.10.0.x86_64-w64-mingw32-cxx11/bin$ x86_64-w64-mingw32-objdump -t libhighs.dll | grep Highs_destroy
[1053](sec  1)(fl 0x00)(ty   20)(scl   2) (nx 0) 0x000000000000d990 Highs_destroy

Thus, there is hope that the libhighs.dll will also work for python-mip on Windows (to be tested later...)

[jajhall]

@galabovaa What does this mean to you?

[galabovaa]

I think this may be related to how the dll inside the python wheel is built in contrast to how the JuMP dll is built.

What I find odd is the C API part, because highspy uses pybind11 to link with C++ and I did not think it should use the C API and particularly highs_destroy. Julia on the other hand links with the C API and would find highs_destroy.

I see that 'Highs_destroy' is called from python-mip/highs.py.

I guess if people try to link with the C Api form highspy they may run into errors. Looking at https://github.com/coin-or/python-mip/blob/master/mip/highs.py, it seems that this is what python-mip are trying to do.

@mathgeekcoder do you fell this is a sensible way of using highspy? Do you think it is a functionality we should aim to support?

Looking at python-mip now, I guess that either the wrapper should be rewritten and depend on highspy properly on the python side, or we should add something to highspy which would allow users to call the C API in the attempted way.

To be honest, thinking back, I am not sure why I included the C API in the first place. I guess they always go together, so I didn't think much at the time.

[rschwarz]

I admit that using the shared library that is shipped with highspy is abuse on our side.
So, to be clear, I'm not insisting on a fix on any kind and am sorry for any confusion caused.

I found this to be the easiest way to get HiGHS in the hands of Python users, and it works well on Linux and OS X, but not on Windows. Maybe there are good technical reasons for this discrepancy, and then we will find a different workaround.

PS: The fact that python-mip uses the C API directly, and not highspy, is by design. As a consequence it might be best to rely on a "proper" shared library for HiGHS there.

[galabovaa]

Yes I think relying on a "proper" dll in that case may be best.

When I was working on the highspy build I had many problems on Windows with linking libraries properly. One reason we decided to use scikit-build-core for that part.

[rschwarz]

OK, good to know, then I will try to steer (our) python-mip users to different installation steps. Thanks for the explanation.

UPDATE: It looks like the https://github.com/metab0t/highsbox project was created just for that purpose. I'll see if it will work as a drop-in replacement.

[mathgeekcoder]

I'm late to the conversation, but I'll give my opinion too. Totally agree with your thoughts so far!

If Python-MIP exploited the highspy bindings instead of C bindings, then using the pyd should be fine (and might clean up the implementation too). Otherwise, not sure if we can trust it.

I'm also not entirely sure about the "standard" libhighs.dll build since it's C++. I've not used "cffi", so maybe you'd explicitly want the C bindings to be certain. That said, the highsbox project might be a great option for you and your users.