test_ndarray fails in anaconda

[jjhidalgo]

I am getting some errors when running tests in linux with anaconda.

I solved the lto-wrapper error by updating the conda base environment. However, I get the following error

./test_ndarray
/home/user/miniconda3/lib/python3.7/site-packages/numpy/init.py:140: UserWarning: mkl-service package failed to import, therefore Intel(R) MKL initialization ensuring its correct out-of-the box operation under condition when Gnu OpenMP had already been loaded by Python process is not assured. Please install mkl-service package, see http://github.com/IntelPython/mkl-service
from . import _distributor_init
Initialisation of forpy failed!!! Can not test. Errorcode = 2

mkl-service is installed and importing numpy in a script works.

in a not updated conda environment with a version of numpy that does not use mkl-service, I need to use -fno-lto to solve the lto-wrapper problem. However I still get the error:

./test_ndarray
Initialisation of forpy failed!!! Can not test. Errorcode = 2

None of the previous tests fails in both cases.

[ylikx]

I have not encountered this problem yet, thanks for posting the issue. It's strange that it fails in the second case, probably an import error? I assume that you did conda activate?

Maybe for debugging purposes you can try to replace the call err_clear statements with call err_print statements in the forpy-numpy-initialisation code (in forpy_initialize_numpy).

Does ignoring Python warnings help? with export PYTHONWARNINGS=ignore in the shell.

Which OS and anaconda version are you using? Maybe I can reproduce the error somehow.

[jjhidalgo]

I am using conda 4.8.3 on CentOS-7. In conda I have the pygimli environment activated. Numpy version is 1.18.5. Ignoring Python warnings did not help.

I am pasting the full error message after replacing call err_clear with call err_print:

./test_ndarray
Traceback (most recent call last):
File "/home/user/miniconda3/envs/pg/lib/python3.7/site-packages/numpy/core/init.py", line 24, in
from . import multiarray
File "/home/user/miniconda3/envs/pg/lib/python3.7/site-packages/numpy/core/multiarray.py", line 14, in
from . import overrides
File "/home/user/miniconda3/envs/pg/lib/python3.7/site-packages/numpy/core/overrides.py", line 7, in
from numpy.core._multiarray_umath import (
ImportError: /home/user/miniconda3/envs/pg/lib/python3.7/site-packages/numpy/core/_multiarray_umath.cpython-37m-x86_64-linux-gnu.so: undefined symbol: PyExc_RecursionError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/user/miniconda3/envs/pg/lib/python3.7/site-packages/numpy/init.py", line 142, in
from . import core
File "/home/user/miniconda3/envs/pg/lib/python3.7/site-packages/numpy/core/init.py", line 50, in
raise ImportError(msg)
ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

• The Python version is: Python3.7 from "/home/user/miniconda3/envs/pg/bin/python3"
• The NumPy version is: "1.18.5"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: /home/user/miniconda3/envs/pg/lib/python3.7/site-packages/numpy/core/_multiarray_umath.cpython-37m-x86_64-linux-gnu.so: undefined symbol: PyExc_RecursionError

[ylikx]

Could be a problem with the anaconda install. It looks like libpythonX.Y is not loaded (see [1]).
Which is strange since it should be in the output of python3-config --ldflags
I could not reproduce the issue yet, because I tried to update an older conda in a VM to the newest version and it broke the install (as in [2], I can't even import numpy in a script). I'll probably try a clean install in the VM.
Also in numpy/numpy#15523 they get a message similar to your initial error message due to a broken package.

[1] https://stackoverflow.com/questions/49784583/numpy-import-fails-on-multiarray-extension-library-when-called-from-embedded-pyt
[2] conda/conda#9957

[jjhidalgo]

This is the output of python3-config --ldflags:
-L/home/user/miniconda3/envs/pg/lib/python3.7/config-3.7m-x86_64-linux-gnu -L/home/user/miniconda3/envs/pg/lib -lpython3.7m -lcrypt -lpthread -ldl -lutil -lrt -lm

You are right libpythonX.Y is not loaded. I added the path to libpython3.so to LDFLAGS in Makefile and all tests passed. I seems it is an anaconda problem.

Thanks for your help.

[ylikx]

Glad that it worked. Which link command did you use in the end?
I could reproduce the error with a fresh install of miniconda (conda 4.8.3). With the original linking command libpython is statically linked, but the conda numpy expects that the dynamic library is loaded.

[jjhidalgo]

I only changed FC and LDFLAGS in Makefile to

FC = x86_64-conda_cos6-linux-gnu-gfortran
LDFLAGS= -fno-lto `$(PYTHON_CONFIG) --ldflags` /home/user/miniconda3/envs/pg/lib/libpython3.so

Linking libpython3.7m.so also works.

[ylikx]

Thanks for sharing.

I compared the output of python3-config --ldflags of an older conda install (4.5.4.) with the most recent one (4.8.3.). The older one has the additional flags -Xlinker -export-dynamic. With these flags it also works with a statically linked libpython.

[jjhidalgo]

Adding the additional flags also works for me.