Unable to run PyMultiNest due to ValueError

[ameliahenkel]

Hello. I am trying to run a script (hotokezaka_bns_inference_test.py) on my Mac and am currently unable to do so; I've been stuck on this error for about a week, and am wondering if you can provide any insight:

Traceback (most recent call last):
File "/Users/meem/Desktop/kn_pe_em/gemma-dev-main/hotokezaka_bns_inference_test.py", line 1, in
import bilby
File "/Users/meem/opt/anaconda3/envs/gemma/lib/python3.12/site-packages/bilby/init.py", line 21, in
from . import core, gw, hyper
File "/Users/meem/opt/anaconda3/envs/gemma/lib/python3.12/site-packages/bilby/core/init.py", line 1, in
from . import grid, likelihood, prior, result, sampler, series, utils, fisher
File "/Users/meem/opt/anaconda3/envs/gemma/lib/python3.12/site-packages/bilby/core/grid.py", line 6, in
from .prior import Prior, PriorDict
File "/Users/meem/opt/anaconda3/envs/gemma/lib/python3.12/site-packages/bilby/core/prior/init.py", line 1, in
from .analytical import *
File "/Users/meem/opt/anaconda3/envs/gemma/lib/python3.12/site-packages/bilby/core/prior/analytical.py", line 2, in
from scipy.special import (
File "/Users/meem/opt/anaconda3/envs/gemma/lib/python3.12/site-packages/scipy/special/init.py", line 826, in
from . import _basic
File "/Users/meem/opt/anaconda3/envs/gemma/lib/python3.12/site-packages/scipy/special/_basic.py", line 22, in
from ._multiufuncs import (assoc_legendre_p_all,
File "/Users/meem/opt/anaconda3/envs/gemma/lib/python3.12/site-packages/scipy/special/_multiufuncs.py", line 142, in
sph_legendre_p = MultiUFunc(
^^^^^^^^^^^
File "/Users/meem/opt/anaconda3/envs/gemma/lib/python3.12/site-packages/scipy/special/_multiufuncs.py", line 41, in init
raise ValueError("All ufuncs must have type numpy.ufunc."
ValueError: All ufuncs must have type numpy.ufunc. Received (<ufunc 'sph_legendre_p'>, <ufunc 'sph_legendre_p'>, <ufunc 'sph_legendre_p'>)

If it's relevant, I was also receiving this message previously, but now I've 'graduated' to only receiving the above error:
Traceback (most recent call last):
File "/Users/meem/Desktop/kn_pe_em/gemma-dev-main/Amelia.py", line 109, in
result = bilby.run_sampler(
^^^^^^^^^^^^^^^^^^
File "/Users/meem/opt/anaconda3/envs/gemma/lib/python3.12/site-packages/bilby/core/sampler/init.py", line 310, in run_sampler
result = sampler.run_sampler()
^^^^^^^^^^^^^^^^^^^^^
File "/Users/meem/opt/anaconda3/envs/gemma/lib/python3.12/site-packages/bilby/core/sampler/base_sampler.py", line 97, in wrapped
output = method(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/meem/opt/anaconda3/envs/gemma/lib/python3.12/site-packages/bilby/core/sampler/pymultinest.py", line 144, in run_sampler
out = pymultinest.solve(
^^^^^^^^^^^^^^^^^^
File "/Users/meem/opt/anaconda3/envs/gemma/lib/python3.12/site-packages/pymultinest/solve.py", line 71, in solve
run(**kwargs)
File "/Users/meem/opt/anaconda3/envs/gemma/lib/python3.12/site-packages/pymultinest/run.py", line 285, in run
lib.run(*args_converted)
^^^^^^^
File "/Users/meem/opt/anaconda3/envs/gemma/lib/python3.12/ctypes/init.py", line 392, in getattr
func = self.getitem(name)
^^^^^^^^^^^^^^^^^^^^^^
File "/Users/meem/opt/anaconda3/envs/gemma/lib/python3.12/ctypes/init.py", line 397, in getitem
func = self._FuncPtr((name_or_ordinal, self))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: dlsym(RTLD_DEFAULT, run): symbol not found

I've never opened an issue like this before, so I'm not sure what additional information is helpful to provide --- just let me know. From my debugging, I've gathered that this is likely either an issue with conflicting versions of numpy and scipy on my computer, or something to do with not having a fortran compiler?

Confusingly, PyMultiNest is working elsewhere on my computer, in another version of a repo I have (entitled GEMMA).

[JohannesBuchner]

Try creating a fresh conda environment, and install pymultinest, then test it with one of the demo scripts.

[ameliahenkel]

Hello! Thank you for your comment; I tried that and have been unable to complete the (Py)MultiNest installation on my computer, due to an error when I run cmake .. in MultiNest/build (am following the installation instructions here:

(base) meem@U-C02G53NGQ05N build % cmake ..
CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):
  Compatibility with CMake < 3.10 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value.  Or, use the <min>...<max> syntax
  to tell CMake that the project requires at least <min> but has been updated
  to work with policies introduced by <max> or earlier.


-- The Fortran compiler identification is GNU 14.2.0
-- The C compiler identification is AppleClang 17.0.0.17000013
-- The CXX compiler identification is AppleClang 17.0.0.17000013
-- Checking whether Fortran compiler has -isysroot
-- Checking whether Fortran compiler has -isysroot - yes
-- Checking whether Fortran compiler supports OSX deployment target flag
-- Checking whether Fortran compiler supports OSX deployment target flag - yes
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Check for working Fortran compiler: /opt/homebrew/bin/gfortran - skipped
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Error at src/CMakeLists.txt:1 (cmake_minimum_required):
  Compatibility with CMake < 3.5 has been removed from CMake.

  Update the VERSION argument <min> value.  Or, use the <min>...<max> syntax
  to tell CMake that the project requires at least <min> but has been updated
  to work with policies introduced by <max> or earlier.

  Or, add -DCMAKE_POLICY_VERSION_MINIMUM=3.5 to try configuring anyway.


-- Configuring incomplete, errors occurred!

I'm wondering if you can provide any advice or have seen this before, please?

If it is helpful, this is what is inside CMakeLists.txt:

cmake_minimum_required(VERSION 3.5)
project(multinest Fortran C CXX)

# By default CMake will install MultiNest to /usr/local on Linux/Mac
# unless you override the install location manually. You can do this
# either by either running:
#  cmake -DCMAKE_INSTALL_PREFIX=/path/to/dir ..
# or by uncommenting and modifying this variable:
#SET(CMAKE_INSTALL_PREFIX /usr/local/multinest)

#=============================================================================
# All done editing
SET(MultiNest_VERSION "3.10")

SET(EXECUTABLE_OUTPUT_PATH ${CMAKE_SOURCE_DIR}/bin)
SET(LIBRARY_OUTPUT_PATH ${CMAKE_SOURCE_DIR}/lib)

include_directories(${CMAKE_SOURCE_DIR}/include)

add_subdirectory(src)

[JohannesBuchner]

I guess it wants you to change cmake_minimum_required(VERSION 3.5) to cmake_minimum_required(VERSION 3.5...3.10)?

[ameliahenkel]

Now I receive the following error message:

(base) meem@U-C02G53NGQ05N build % cmake ..
-- Detected gfortran, adding -ffree-line-length-none compiler flag.
-- Detected gfortran >= 10, adding -std=legacy compiler flag.
-- Detected gfortran-10+, adding -w -fallow-argument-mismatch compiler flag.
-- Could NOT find MPI_C (missing: MPI_C_WORKS) 
-- Could NOT find MPI_CXX (missing: MPI_CXX_WORKS) 
-- Could NOT find MPI_Fortran (missing: MPI_Fortran_WORKS) 
-- Could NOT find MPI (missing: MPI_C_FOUND MPI_CXX_FOUND MPI_Fortran_FOUND) 
-- MPI not found, only non-MPI MultiNest libraries will be built.
CMake Error at src/example_eggbox_C/CMakeLists.txt:7 (FIND_LIBRARY):
  Could not find m using the following names:


-- Configuring incomplete, errors occurred!

[ameliahenkel]

Hello,

As an update, I was able to complete installing MultiNest and PyMultiNest according to the installation instructions by manually adding the suggested fix to every CMakeLists.txt file in MultiNest (including all those in the directories in /src/).

However, when I try to run PyMultiNest on my computer, I receive the following error message. I'm curious if you can advise? This script used to run without errors in the past.

(base) meem@U-C02G53NGQ05N lc_sampling % python bilby_m1_m2_r.py                                  
/Users/meem/opt/anaconda3/lib/python3.8/site-packages/astropy/units/quantity.py:673: RuntimeWarning: overflow encountered in exp
  result = super().__array_ufunc__(function, method, *arrays, **kwargs)
11:46 bilby INFO    : Running for label 'INkd_Sdk_14_14_10p0_May_07_25NewM1Prior', output will be saved to 'bilby-pymultinest-m1m2r'
problem: dlopen(/Users/meem/opt/anaconda3/lib/python3.8/site-packages/MultiNest/lib/libmultinest.dylib, 0x0006): tried: '/Users/meem/opt/anaconda3/lib/python3.8/site-packages/MultiNest/lib/libmultinest.dylib' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')), '/libmultinest.dylib' (no such file), '/Users/meem/opt/anaconda3/lib/python3.8/site-packages/MultiNest/lib/libmultinest.dylib' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/meem/opt/anaconda3/lib/python3.8/site-packages/MultiNest/lib/libmultinest.dylib' (no such file), '/Users/meem/opt/anaconda3/lib/python3.8/site-packages/MultiNest/lib/libmultinest.dylib' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')), '/Users/meem/opt/anaconda3/lib/python3.8/site-packages/MultiNest/lib/libmultinest.3.10.dylib' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')), '/libmultinest.3.10.dylib' (no such file), '/Users/meem/opt/anaconda3/lib/python3.8/site-packages/MultiNest/lib/libmultinest.3.10.dylib' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/meem/opt/anaconda3/lib/python3.8/site-packages/MultiNest/lib/libmultinest.3.10.dylib' (no such file), '/Users/meem/opt/anaconda3/lib/python3.8/site-packages/MultiNest/lib/libmultinest.3.10.dylib' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64'))
Traceback (most recent call last):
  File "/Users/meem/opt/anaconda3/lib/python3.8/site-packages/pymultinest/run.py", line 16, in _load_library
    return cdll.LoadLibrary(libname)
  File "/Users/meem/opt/anaconda3/lib/python3.8/ctypes/__init__.py", line 459, in LoadLibrary
    return self._dlltype(name)
  File "/Users/meem/opt/anaconda3/lib/python3.8/ctypes/__init__.py", line 381, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen(/Users/meem/opt/anaconda3/lib/python3.8/site-packages/MultiNest/lib/libmultinest.dylib, 0x0006): tried: '/Users/meem/opt/anaconda3/lib/python3.8/site-packages/MultiNest/lib/libmultinest.dylib' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')), '/libmultinest.dylib' (no such file), '/Users/meem/opt/anaconda3/lib/python3.8/site-packages/MultiNest/lib/libmultinest.dylib' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/meem/opt/anaconda3/lib/python3.8/site-packages/MultiNest/lib/libmultinest.dylib' (no such file), '/Users/meem/opt/anaconda3/lib/python3.8/site-packages/MultiNest/lib/libmultinest.dylib' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')), '/Users/meem/opt/anaconda3/lib/python3.8/site-packages/MultiNest/lib/libmultinest.3.10.dylib' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')), '/libmultinest.3.10.dylib' (no such file), '/Users/meem/opt/anaconda3/lib/python3.8/site-packages/MultiNest/lib/libmultinest.3.10.dylib' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/meem/opt/anaconda3/lib/python3.8/site-packages/MultiNest/lib/libmultinest.3.10.dylib' (no such file), '/Users/meem/opt/anaconda3/lib/python3.8/site-packages/MultiNest/lib/libmultinest.3.10.dylib' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/meem/opt/anaconda3/lib/python3.8/site-packages/bilby/core/sampler/base_sampler.py", line 319, in _verify_external_sampler
    __import__(external_sampler_name)
  File "/Users/meem/opt/anaconda3/lib/python3.8/site-packages/pymultinest/__init__.py", line 19, in <module>
    from .run import run
  File "/Users/meem/opt/anaconda3/lib/python3.8/site-packages/pymultinest/run.py", line 52, in <module>
    lib = _load_library('libmultinest')
  File "/Users/meem/opt/anaconda3/lib/python3.8/site-packages/pymultinest/run.py", line 50, in _load_library
    sys.exit(1)
SystemExit: 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "bilby_m1_m2_r.py", line 197, in <module>
    result = bilby.run_sampler(
  File "/Users/meem/opt/anaconda3/lib/python3.8/site-packages/bilby/core/sampler/__init__.py", line 190, in run_sampler
    sampler = sampler_class(
  File "/Users/meem/opt/anaconda3/lib/python3.8/site-packages/bilby/core/sampler/pymultinest.py", line 79, in __init__
    super(Pymultinest, self).__init__(
  File "/Users/meem/opt/anaconda3/lib/python3.8/site-packages/bilby/core/sampler/base_sampler.py", line 904, in __init__
    super(_TemporaryFileSamplerMixin, self).__init__(**kwargs)
  File "/Users/meem/opt/anaconda3/lib/python3.8/site-packages/bilby/core/sampler/base_sampler.py", line 238, in __init__
    self._verify_external_sampler()
  File "/Users/meem/opt/anaconda3/lib/python3.8/site-packages/bilby/core/sampler/base_sampler.py", line 321, in _verify_external_sampler
    raise SamplerNotInstalledError(
bilby.core.sampler.base_sampler.SamplerNotInstalledError: Sampler pymultinest is not installed on this system

[JohannesBuchner]

Looks like the compiler used a different (python) environment than when you are running it?

[sahiljhawar]

@ameliahenkel Can you try using these commands,

If you are on MacOS make sure to have gfortran and gcc and export their paths

FC=$(which gfortran-14)
CC=$(which gcc)
CXX=$(which g++)

you may have to change 14 to your gfortran version or maybe remove it completely based on your system.

And if you are on Linux directly do that following

git clone https://github.com/JohannesBuchner/MultiNest && cd MultiNest/build && rm -rf * && cmake -DCMAKE_POLICY_VERSION_MINIMUM=3.5 .. && make && cd ../..

LD_LIBRARY_PATH=$YOURDIR/MultiNest/lib:$LD_LIBRARY_PATH
DYLD_LIBRARY_PATH=$YOURDIR/MultiNest/lib:$DYLD_LIBRARY_PATH

And finally
pip install pymultinest

[sarahshi]

On Mac Silicon, it appears that pip install pymultinest did not properly install the package. Using conda install -c conda-forge pymultinest solved all my issues. I put together this installation guide, in case it is helpful for anyone else.

Instructions for DFENS Installation on Mac Silicon, with backends homebrew, conda, pip
brew install cmake
brew install gcc
brew install open-mpi

conda env create -f xyz.yml # contains pip install mpi4py and other standard packages
cd Documents/GitHub # This is locally where I wanted MultiNest to reside
git clone https://github.com/JohannesBuchner/MultiNest
cd MultiNest/build
cmake ..cmake -DCMAKE_POLICY_VERSION_MINIMUM=3.5 .. # New step required given
deprecation, pure cmake does not work
cp -v ~/Documents/GitHub/MultiNest/lib/lib* ~/anaconda3/lib/ # New step required given
deprecation
conda activate xyz
conda install -c conda-forge pymultinest # THIS IS IMPORTANT: IT MUST BE CONDA-FORGE
INSTALLED, NOT PIP
python pymultinest_demo.py # Download this
https://github.com/JohannesBuchner/PyMultiNest/blob/master/pymultinest_demo.py and check
that it runs properly
Finally, add this to your .bashrc profile on Mac.
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/Users/yourname/GitHub/MultiNest/lib
export DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH:/Users/yourname/GitHub/MultiNest/lib