Segmentation fault

[Silflame]

I encountered a similar Segmentation fault error (#65 (comment)) when running calc.compute() using the latest Docker image (built from the provided Dockerfile).
Reproduction details:

• pyspi version: (latest from GitHub)
• Environment: Docker (built with Dockerfile from the repo)
• Input shape: 250 variables × 230 time points input_data.csv
• Command:

     from scipy import io as sio
     from oct2py import octave
     from pyspi.calculator import Calculator
     import pandas as pd
     import numpy as np
     ts = pd.read_csv('input_data.csv', header=None)
     ts = np.array(ts)
     calc = Calculator(dataset=ts, normalise=True, detrend=True)
     calc.compute()
  

When I reduce the variable count to 80 (input shape: 80 × 230), the code runs without error. This strongly suggests the issue is data dimension dependent. Would you be able to check again with a input matrix of size 250 × 230 and see if the crash occurs on your end? If not, could you recommend debugging steps?

Thanks!

[benfulcher]

Thanks for the clear report. I'll see if @joshuabmoore can have a quick check and get back to you.

[joshuabmoore]

Hi @Silflame,

I did a quick test using input_data.csv and it looks like the segmentation fault you are encountering is specific to the GraphicalLasso and GraphicalLassoCV estimators, which affects these Covariance and Precision SPIs:

• cov_GraphicalLassoCV,
• cov_GraphicalLassoCV
• cov-sq_GraphicalLasso
• cov-sq_GraphicalLassoCV
• prec_GraphicalLasso
• prec_GraphicalLassoCV
• prec-sq_GraphicalLasso
• prec-sq_GraphicalLassoCV

This issue seems to be dataset dependent. If your covariance matrix is ill-conditioned, this can cause numerical instabilities for the GraphicalLasso algorithm. Running GraphicalLasso directly on the scaled data confirms this:

FloatingPointError: Non SPD result: the system is too ill-conditioned for this solver. The system is too ill-conditioned for this solver

While an ill-conditioned problem shouldn't cause a segmentation fault on its own, the crash appears to be triggered by multi-threaded BLAS running out of per-thread memory with the full 250 x 230 dataset. For the smaller dataset (e.g., 80 x 230), the per-thread requirement simply isn't large enough to trigger the crash. One workaround to prevent running into memory issues would be to force BLAS to a single thread by setting your environment variables before importing numpy:

import os
os.environ["OMP_NUM_THREADS"] = "1"
os.environ["MKL_NUM_THREADS"] = "1"
os.environ["OPENBLAS_NUM_THREADS"] = "1"
os.environ["VECLIB_MAXIMUM_THREADS"] = "1"

import numpy as np
# rest of your imports 

and then computing your SPIs as usual. However, note that this will increase the computation time of all SPIs which currently rely on BLAS/LAPACK.

Since these SPIs are unlikely to return meaningful results for your data, I would recommend simply creating a new configuration YAML file with the above estimators removed:

calc = Calculator(dataset=ts, normalise=True, detrend=True, configfile="config_no_GLASSO.yaml")

I hope this helps. Let me know if you encounter any further issues!