scipy does not works in mwpython on macos

[balbasty]

Scipy imports fail with mwpython.

This has already been reported to mathworks with no response

The reason why mwpython is needed at all seems to be graphics related

Here's the error I get

Traceback (most recent call last):
  File "/Users/balbasty/Dropbox/Workspace/code/fil/spm2mne/Matlab2MNE_check.py", line 41, in <module>
    raw = spm_2_mne_raw(mD)
  File "/Users/balbasty/Dropbox/Workspace/code/fil/spm2mne/spm_mne_converter.py", line 38, in spm_2_mne_raw
    info = mne.create_info(ch_names=channel_names, sfreq=sampling_freq, ch_types=channel_types)
  File "/Users/balbasty/miniforge3/envs/spm/lib/python3.10/site-packages/lazy_loader/__init__.py", line 82, in __getattr__
    submod = importlib.import_module(submod_path)
  File "/Users/balbasty/miniforge3/envs/spm/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/balbasty/miniforge3/envs/spm/lib/python3.10/site-packages/mne/_fiff/meas_info.py", line 21, in <module>
    from ..utils import (
  File "/Users/balbasty/miniforge3/envs/spm/lib/python3.10/site-packages/lazy_loader/__init__.py", line 82, in __getattr__
    submod = importlib.import_module(submod_path)
  File "/Users/balbasty/miniforge3/envs/spm/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/Users/balbasty/miniforge3/envs/spm/lib/python3.10/site-packages/mne/utils/numerics.py", line 20, in <module>
    from scipy import sparse
  File "/Users/balbasty/miniforge3/envs/spm/lib/python3.10/site-packages/scipy/__init__.py", line 134, in __getattr__
    return _importlib.import_module(f'scipy.{name}')
  File "/Users/balbasty/miniforge3/envs/spm/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/Users/balbasty/miniforge3/envs/spm/lib/python3.10/site-packages/scipy/sparse/__init__.py", line 315, in <module>
    from . import csgraph
  File "/Users/balbasty/miniforge3/envs/spm/lib/python3.10/site-packages/scipy/sparse/csgraph/__init__.py", line 187, in <module>
    from ._laplacian import laplacian
  File "/Users/balbasty/miniforge3/envs/spm/lib/python3.10/site-packages/scipy/sparse/csgraph/_laplacian.py", line 7, in <module>
    from scipy.sparse.linalg import LinearOperator
  File "/Users/balbasty/miniforge3/envs/spm/lib/python3.10/site-packages/scipy/sparse/linalg/__init__.py", line 131, in <module>
    from ._isolve import *
  File "/Users/balbasty/miniforge3/envs/spm/lib/python3.10/site-packages/scipy/sparse/linalg/_isolve/__init__.py", line 4, in <module>
    from .iterative import *
  File "/Users/balbasty/miniforge3/envs/spm/lib/python3.10/site-packages/scipy/sparse/linalg/_isolve/iterative.py", line 5, in <module>
    from scipy.linalg import get_lapack_funcs
  File "/Users/balbasty/miniforge3/envs/spm/lib/python3.10/site-packages/scipy/linalg/__init__.py", line 203, in <module>
    from ._misc import *
  File "/Users/balbasty/miniforge3/envs/spm/lib/python3.10/site-packages/scipy/linalg/_misc.py", line 3, in <module>
    from .blas import get_blas_funcs
  File "/Users/balbasty/miniforge3/envs/spm/lib/python3.10/site-packages/scipy/linalg/blas.py", line 213, in <module>
    from scipy.linalg import _fblas
ImportError: dlopen(/Users/balbasty/miniforge3/envs/spm/lib/python3.10/site-packages/scipy/linalg/_fblas.cpython-310-darwin.so, 0x0002): Symbol not found: _caxpy$NEWLAPACK
  Referenced from: <54500246-AE5B-3FDD-A054-9F941C9CE5D8> /Users/balbasty/miniforge3/envs/spm/lib/python3.10/site-packages/scipy/linalg/_fblas.cpython-310-darwin.so
  Expected in:     <81FFBDC1-F06D-3E68-825F-32A9B9F898B0> /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate

I suppose some library path/flag is not set properly by mwpython. I don't have any idea how to fix this, but we should find a workaround. Otherwise, spm-python is essentially unsuable with the rest of the neuroimaging ecosystem (which is highly scipy-dependent).

[balbasty]

A slightly more reproducible example:

import ctypes
ctypes.cdll.LoadLibrary("/Users/balbasty/miniforge3/envs/spm/lib/python3.10/site-packages/scipy/linalg/_fblas.cpython-310-darwin.so")

works in ipython but not in mwpython -m IPython

This is all run from inside a conda env with:

python                    3.10.16         h870587a_1_cpython    conda-forge

[balbasty]

Some more clues: it seems that conda does not interract well with DYLD_LIBRARY_PATH being set.

See: conda/conda#1169

But mwpython does set its own DYLD_LIBRARY_PATH.
If I edit the mwpython script so that it sets DYLD_FALLBACK_LIBRARY_PATH instead, I can import scipy.sparse again, but not matlab (it thinks it is not running through mwpython anymore).

[balbasty]

OK got it. The paths to the matlab libs must be set through DYLD_LIBRARY_PATH but the paths to the python libs must be set through DYLD_FALLBACK_LIBRARY_PATH.

The following patch to mwpython does the trick. Should we ship it with spm-python?

--- /Applications/MATLAB_R2024b.app/bin/mwpython	2025-02-27 14:42:41
+++ scripts/mwpython	2025-02-27 14:43:10
@@ -224,14 +224,17 @@
 MW_PYTHON_LIBRARY_PATH=.:${MCRROOT}/runtime/${arch}: ;
 MW_PYTHON_LIBRARY_PATH=${MW_PYTHON_LIBRARY_PATH}:${MCRROOT}/bin/${arch} ;
 MW_PYTHON_LIBRARY_PATH=${MW_PYTHON_LIBRARY_PATH}:${MCRROOT}/sys/os/${arch};
-if [ -n $python_libdir ];
-then
-  MW_PYTHON_LIBRARY_PATH=${MW_PYTHON_LIBRARY_PATH}:${python_libdir}
-fi
 DYLD_LIBRARY_PATH=${MW_PYTHON_LIBRARY_PATH}:${DYLD_LIBRARY_PATH}:
 export DYLD_LIBRARY_PATH;
 [ $VERBOSE -eq 1 ] && echo DYLD_LIBRARY_PATH is ${DYLD_LIBRARY_PATH};
 
+if [ -n $python_libdir ];
+then
+  DYLD_FALLBACK_LIBRARY_PATH=${python_libdir}:/usr/local/lib:/usr/lib
+  export DYLD_FALLBACK_LIBRARY_PATH;
+  [ $VERBOSE -eq 1 ] && echo DYLD_FALLBACK_LIBRARY_PATH is ${DYLD_FALLBACK_LIBRARY_PATH};
+fi
+
 if [ $USING_VIRTUAL_ENV -eq 1 ]
 then
   site_packages_path="${VIRTUAL_ENV}/lib/python${python_version}/site-packages"

[balbasty]

For completeness, I've implemented my own version of mwpython in python (yes this is circular) here. I only use DYLD_FALLBACK_LIBRARY_PATH and it works wirth most packages that used to fail (scipy.sparse, pytest).

The one remining package that fails is matplotlib, which is pretty annoying. I've tried a bunch of different plotting backends and all crash, although not necessarily in the same way.