installing on AWS linux (2 or 2023) - numpy/numba compatibility and use of deprecated imp library

[dllahr]

Hello

Apologies in advance, this may be a cellxgene issue and not a cellxgene-gateway issue, but it appeared for me installing cellxgene-gateway.

Here's how I installed cellxgene-gateway:

1. create AWS EC2 instance using Amazon Linux 2023
   • using security group to open ports 22, 80, 5005
2. updates: dnf update, dnf upgrade report nothing to do
3. install miniconda
4. conda create -n cellxgene python
5. conda activate cellxgene
6. sudo dnf install gcc gcc-c++
7. nohup pip install cellxgene-gateway &> install.log
8. download test data: wget https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/example-dataset/pbmc3k.h5ad -O cellxgene_data/pbmc3k.h5ad

Then had to fiddle to get it working:
test it out just run cellxgene by itself - error message:

Traceback (most recent call last):
  File "/home/ec2-user/miniconda3/envs/cellxgene/bin/cellxgene", line 5, in <module>
    from server.cli.cli import cli
  File "/home/ec2-user/miniconda3/envs/cellxgene/lib/python3.12/site-packages/server/cli/cli.py", line 3, in <module>
    from .annotate import annotate
  File "/home/ec2-user/miniconda3/envs/cellxgene/lib/python3.12/site-packages/server/cli/annotate.py", line 12, in <module>
    import pandas as pd
  File "/home/ec2-user/miniconda3/envs/cellxgene/lib/python3.12/site-packages/pandas/__init__.py", line 22, in <module>
    from pandas.compat import is_numpy_dev as _is_numpy_dev  # pyright: ignore # noqa:F401
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ec2-user/miniconda3/envs/cellxgene/lib/python3.12/site-packages/pandas/compat/__init__.py", line 18, in <module>
    from pandas.compat.numpy import (
  File "/home/ec2-user/miniconda3/envs/cellxgene/lib/python3.12/site-packages/pandas/compat/numpy/__init__.py", line 4, in <module>
    from pandas.util.version import Version
  File "/home/ec2-user/miniconda3/envs/cellxgene/lib/python3.12/site-packages/pandas/util/__init__.py", line 2, in <module>
    from pandas.util._decorators import (  # noqa:F401
  File "/home/ec2-user/miniconda3/envs/cellxgene/lib/python3.12/site-packages/pandas/util/_decorators.py", line 14, in <module>
    from pandas._libs.properties import cache_readonly
  File "/home/ec2-user/miniconda3/envs/cellxgene/lib/python3.12/site-packages/pandas/_libs/__init__.py", line 13, in <module>
    from pandas._libs.interval import Interval
  File "pandas/_libs/interval.pyx", line 1, in init pandas._libs.interval
ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

fixed this by doing: pip install --upgrade numba==0.59.1 - this replaces:

• numpy 2.0.1 with 1.26.4
• llvmlite-0.43.0 with llvmlite-0.42.0
• numba-0.60.0 with numba-0.59.1
• (from Latest numba incompatible with numpy 1.22 ContinuumIO/anaconda-issues#12944)

Error message then becomes:

Traceback (most recent call last):
  File "/home/ec2-user/miniconda3/envs/cellxgene/bin/cellxgene", line 5, in <module>
    from server.cli.cli import cli
  File "/home/ec2-user/miniconda3/envs/cellxgene/lib/python3.12/site-packages/server/cli/cli.py", line 4, in <module>
    from .launch import launch
  File "/home/ec2-user/miniconda3/envs/cellxgene/lib/python3.12/site-packages/server/cli/launch.py", line 12, in <module>
    from server.app.app import Server
  File "/home/ec2-user/miniconda3/envs/cellxgene/lib/python3.12/site-packages/server/app/app.py", line 16, in <module>
    import server.common.rest as common_rest
  File "/home/ec2-user/miniconda3/envs/cellxgene/lib/python3.12/site-packages/server/common/rest.py", line 26, in <module>
    from server.common.fbs.matrix import decode_matrix_fbs
  File "/home/ec2-user/miniconda3/envs/cellxgene/lib/python3.12/site-packages/server/common/fbs/matrix.py", line 5, in <module>
    from flatbuffers import Builder
  File "/home/ec2-user/miniconda3/envs/cellxgene/lib/python3.12/site-packages/flatbuffers/__init__.py", line 15, in <module>
    from .builder import Builder
  File "/home/ec2-user/miniconda3/envs/cellxgene/lib/python3.12/site-packages/flatbuffers/builder.py", line 15, in <module>
    from . import number_types as N
  File "/home/ec2-user/miniconda3/envs/cellxgene/lib/python3.12/site-packages/flatbuffers/number_types.py", line 18, in <module>
    from . import packer
  File "/home/ec2-user/miniconda3/envs/cellxgene/lib/python3.12/site-packages/flatbuffers/packer.py", line 22, in <module>
    from . import compat
  File "/home/ec2-user/miniconda3/envs/cellxgene/lib/python3.12/site-packages/flatbuffers/compat.py", line 19, in <module>
    import imp
ModuleNotFoundError: No module named 'imp'

Edited the file miniconda3/envs/cellxgene/lib/python3.12/site-packages/flatbuffers/compat.py - based on docs for python imp.find_module indicates it is deprecated and should be replaced with importlib.util.find_spec

• line 19: import imp --> import importlib
• line 56: imp.find_module('numpy') --> importlib.util.find_spec('numpy')

cellxgene and cellxgene-gateway now both work.

I'm guessing there is a much better / cleaner way to get this working! Not sure if this is something that needs to be fixed or I did something terribly wrong.

linux version: 6.1.102-111.182.amzn2023.x86_64
conda version: 24.5.0
python version: 3.12.4