🚸 User journey in updating to new LaminR version

[falexwolf]

I'm just documenting this here so that we can learn from it!

Ok, first I try to connect to an instance. Because I have the latest version of laminr I thought I should be good regarding reticulate. But that was wrong, I got an error:

> laminr::lamin_connect("laminlabs/lamindata") 
Error: .onLoad failed in loadNamespace() for 'laminr', details:
  call: NULL
  error: 'py_require' is not an exported object from 'namespace:reticulate'
> packageVersion("reticulate")
[1] ‘1.40.0’
> packageVersion("laminr")
[1] ‘1.1.1’

I manually update reticulate

> install.packages("reticulate")
Error in install.packages : Updating loaded packages

Restarting R session...

> install.packages("reticulate")
trying URL 'https://cran.rstudio.com/bin/macosx/big-sur-arm64/contrib/4.4/reticulate_1.42.0.tgz'
Content type 'application/x-gzip' length 2850215 bytes (2.7 MB)
==================================================
downloaded 2.7 MB


The downloaded binary packages are in
	/var/folders/b2/nq98s8w96zq2mdcj6rdplvqr0000gn/T//RtmpFB6EWF/downloaded_packages
> 
> packageVersion("reticulate")
[1] ‘1.42.0’
> packageVersion("laminr")
[1] ‘1.1.1’

Now I get a new error with a loooong traceback

Details

> laminr::lamin_connect("laminlabs/lamindata") 
Downloading cpython-3.11.13-macos-aarch64-none (download) (17.2MiB)
 Downloading cpython-3.11.13-macos-aarch64-none (download)
Downloading botocore (13.0MiB)
Downloading numpy (5.1MiB)
Downloading pandas (10.3MiB)
Downloading pyarrow (29.4MiB)
Downloading pydantic-core (1.8MiB)
Downloading pygments (1.2MiB)
Downloading django (7.9MiB)
Downloading h5py (2.7MiB)
Downloading psycopg2-binary (3.1MiB)
 Downloading pygments
 Downloading pydantic-core
 Downloading h5py
 Downloading psycopg2-binary
 Downloading numpy
 Downloading django
 Downloading pandas
 Downloading botocore
   Building scipy==1.6.1
 Downloading pyarrow
  × Failed to build `scipy==1.6.1`
  ├─▶ The build backend returned an error
  ╰─▶ Call to `setuptools.build_meta:__legacy__.build_wheel` failed (exit
      status: 1)

      [stdout]
      INFO: lapack_opt_info:
      INFO: lapack_armpl_info:
      INFO: customize UnixCCompiler
      INFO: C compiler: cc -DNDEBUG -g -fwrapv -O3 -Wall -arch
      arm64 -mmacosx-version-min=11.0 -Wno-nullability-completeness
      -Wno-expansion-to-defined -Wno-undef-prefix -fPIC

      creating /var/folders/b2/nq98s8w96zq2mdcj6rdplvqr0000gn/T/tmp647divw9/var
      creating
      /var/folders/b2/nq98s8w96zq2mdcj6rdplvqr0000gn/T/tmp647divw9/var/folders
      creating
      /var/folders/b2/nq98s8w96zq2mdcj6rdplvqr0000gn/T/tmp647divw9/var/folders/b2
      creating
      /var/folders/b2/nq98s8w96zq2mdcj6rdplvqr0000gn/T/tmp647divw9/var/folders/b2/nq98s8w96zq2mdcj6rdplvqr0000gn
      creating
      /var/folders/b2/nq98s8w96zq2mdcj6rdplvqr0000gn/T/tmp647divw9/var/folders/b2/nq98s8w96zq2mdcj6rdplvqr0000gn/T
      creating
      /var/folders/b2/nq98s8w96zq2mdcj6rdplvqr0000gn/T/tmp647divw9/var/folders/b2/nq98s8w96zq2mdcj6rdplvqr0000gn/T/tmp647divw9
      INFO: compile options: '-MMD -MF
      /var/folders/b2/nq98s8w96zq2mdcj6rdplvqr0000gn/T/tmp647divw9/file.c.d
      -c'
      INFO: cc:
      /var/folders/b2/nq98s8w96zq2mdcj6rdplvqr0000gn/T/tmp647divw9/file.c
      INFO:   libraries armpl_lp64_mp not found in
      ['/Users/falexwolf/Library/Caches/org.R-project.R/R/reticulate/uv/cache/builds-v0/.tmpeYzIOV/lib',
      '/usr/local/lib', '/usr/lib']
      INFO:   NOT AVAILABLE
      INFO:
      INFO: lapack_mkl_info:
      INFO:   libraries mkl_rt not found in
      ['/Users/falexwolf/Library/Caches/org.R-project.R/R/reticulate/uv/cache/builds-v0/.tmpeYzIOV/lib',
      '/usr/local/lib', '/usr/lib']
      INFO:   NOT AVAILABLE
      INFO:
      INFO: lapack_ssl2_info:
      INFO:   libraries fjlapackexsve not found in
      ['/Users/falexwolf/Library/Caches/org.R-project.R/R/reticulate/uv/cache/builds-v0/.tmpeYzIOV/lib',
      '/usr/local/lib', '/usr/lib']
      INFO:   NOT AVAILABLE
      INFO:
      INFO: openblas_lapack_info:
      INFO:   libraries openblas not found in
      ['/Users/falexwolf/Library/Caches/org.R-project.R/R/reticulate/uv/cache/builds-v0/.tmpeYzIOV/lib',
      '/usr/local/lib', '/usr/lib']
      INFO:   NOT AVAILABLE
      INFO:
      INFO: openblas_clapack_info:
      INFO:   libraries openblas,lapack not found in
      ['/Users/falexwolf/Library/Caches/org.R-project.R/R/reticulate/uv/cache/builds-v0/.tmpeYzIOV/lib',
      '/usr/local/lib', '/usr/lib']
      INFO:   NOT AVAILABLE
      INFO:
      INFO: flame_info:
      INFO:   libraries flame not found in
      ['/Users/falexwolf/Library/Caches/org.R-project.R/R/reticulate/uv/cache/builds-v0/.tmpeYzIOV/lib',
      '/usr/local/lib', '/usr/lib']
      INFO:   NOT AVAILABLE
      INFO:
      INFO: accelerate_info:
      INFO:   NOT AVAILABLE
      INFO:
      INFO: atlas_3_10_threads_info:
      INFO: Setting PTATLAS=ATLAS
      INFO:   libraries tatlas,tatlas not found in
      /Users/falexwolf/Library/Caches/org.R-project.R/R/reticulate/uv/cache/builds-v0/.tmpeYzIOV/lib
      INFO:   libraries tatlas,tatlas not found in /usr/local/lib
      INFO:   libraries tatlas,tatlas not found in /usr/lib
      INFO: <class 'numpy.distutils.system_info.atlas_3_10_threads_info'>
      INFO:   NOT AVAILABLE
      INFO:
      INFO: atlas_3_10_info:
      INFO:   libraries satlas,satlas not found in
      /Users/falexwolf/Library/Caches/org.R-project.R/R/reticulate/uv/cache/builds-v0/.tmpeYzIOV/lib
      INFO:   libraries satlas,satlas not found in /usr/local/lib
      INFO:   libraries satlas,satlas not found in /usr/lib
      INFO: <class 'numpy.distutils.system_info.atlas_3_10_info'>
      INFO:   NOT AVAILABLE
      INFO:
      INFO: atlas_threads_info:
      INFO: Setting PTATLAS=ATLAS
      INFO:   libraries ptf77blas,ptcblas,atlas not found in
      /Users/falexwolf/Library/Caches/org.R-project.R/R/reticulate/uv/cache/builds-v0/.tmpeYzIOV/lib
      INFO:   libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
      INFO:   libraries ptf77blas,ptcblas,atlas not found in /usr/lib
      INFO: <class 'numpy.distutils.system_info.atlas_threads_info'>
      INFO:   NOT AVAILABLE
      INFO:
      INFO: atlas_info:
      INFO:   libraries f77blas,cblas,atlas not found in
      /Users/falexwolf/Library/Caches/org.R-project.R/R/reticulate/uv/cache/builds-v0/.tmpeYzIOV/lib
      INFO:   libraries f77blas,cblas,atlas not found in /usr/local/lib
      INFO:   libraries f77blas,cblas,atlas not found in /usr/lib
      INFO: <class 'numpy.distutils.system_info.atlas_info'>
      INFO:   NOT AVAILABLE
      INFO:
      INFO: lapack_info:
      INFO:   libraries lapack not found in
      ['/Users/falexwolf/Library/Caches/org.R-project.R/R/reticulate/uv/cache/builds-v0/.tmpeYzIOV/lib',
      '/usr/local/lib', '/usr/lib']
      INFO:   NOT AVAILABLE
      INFO:
      INFO: lapack_src_info:
      INFO:   NOT AVAILABLE
      INFO:
      INFO:   NOT AVAILABLE
      INFO:

      [stderr]
      setup.py:563: DeprecationWarning:

        `numpy.distutils` is deprecated since NumPy 1.23.0, as a result
        of the deprecation of `distutils` itself. It will be removed for
        Python >= 3.12. For older Python versions it will remain present.
        It is recommended to use `setuptools < 60.0` for those Python
      versions.
        For more details, see:
          https://numpy.org/devdocs/reference/distutils_status_migration.html


        from numpy.distutils.core import setup
      Running from SciPy source directory.
      /Users/falexwolf/Library/Caches/org.R-project.R/R/reticulate/uv/cache/builds-v0/.tmpeYzIOV/lib/python3.11/site-packages/numpy/distutils/system_info.py:1973:
      UserWarning:
          Lapack (http://www.netlib.org/lapack/) libraries not found.
          Directories to search for the libraries can be specified in the
          numpy/distutils/site.cfg file (section [lapack]) or by setting
          the LAPACK environment variable.
        return getattr(self, '_calc_info_{}'.format(name))()
      /Users/falexwolf/Library/Caches/org.R-project.R/R/reticulate/uv/cache/builds-v0/.tmpeYzIOV/lib/python3.11/site-packages/numpy/distutils/system_info.py:1973:
      UserWarning:
          Lapack (http://www.netlib.org/lapack/) sources not found.
          Directories to search for the sources can be specified in the
          numpy/distutils/site.cfg file (section [lapack_src]) or by setting
          the LAPACK_SRC environment variable.
        return getattr(self, '_calc_info_{}'.format(name))()
      Traceback (most recent call last):
        File "<string>", line 11, in <module>
        File
      "/Users/falexwolf/Library/Caches/org.R-project.R/R/reticulate/uv/cache/builds-v0/.tmpeYzIOV/lib/python3.11/site-packages/setuptools/build_meta.py",
      line 216, in build_wheel
          return self._build_with_temp_dir(['bdist_wheel'], '.whl',
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File
      "/Users/falexwolf/Library/Caches/org.R-project.R/R/reticulate/uv/cache/builds-v0/.tmpeYzIOV/lib/python3.11/site-packages/setuptools/build_meta.py",
      line 202, in _build_with_temp_dir
          self.run_setup()
        File
      "/Users/falexwolf/Library/Caches/org.R-project.R/R/reticulate/uv/cache/builds-v0/.tmpeYzIOV/lib/python3.11/site-packages/setuptools/build_meta.py",
      line 254, in run_setup
          self).run_setup(setup_script=setup_script)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File
      "/Users/falexwolf/Library/Caches/org.R-project.R/R/reticulate/uv/cache/builds-v0/.tmpeYzIOV/lib/python3.11/site-packages/setuptools/build_meta.py",
      line 145, in run_setup
          exec(compile(code, __file__, 'exec'), locals())
        File "setup.py", line 588, in <module>
          setup_package()
        File "setup.py", line 584, in setup_package
          setup(**metadata)
        File
      "/Users/falexwolf/Library/Caches/org.R-project.R/R/reticulate/uv/cache/builds-v0/.tmpeYzIOV/lib/python3.11/site-packages/numpy/distutils/core.py",
      line 135, in setup
          config = configuration()
                   ^^^^^^^^^^^^^^^
        File "setup.py", line 499, in configuration
          raise NotFoundError(msg)
      numpy.distutils.system_info.NotFoundError: No BLAS/LAPACK libraries
      found. Note: Accelerate is no longer supported.
      To build Scipy from sources, BLAS & LAPACK libraries need to be
      installed.
      See site.cfg.example in the Scipy source directory and
      https://docs.scipy.org/doc/scipy/reference/building/index.html for
      details.

      hint: This usually indicates a problem with the package or the build
      environment.
  help: `scipy` (v1.6.1) was included because `lamindb` (v1.6.2) depends on
        `scipy`
uv error code: 1
-- Current requirements -------------------------------------------------
 Python:   >=3.10, <3.14
 Packages: numpy, lamindb>=1.2
-------------------------------------------------------------------------
Error in uv_get_or_create_env() : 
  Call `py_require()` to remove or replace conflicting requirements.
Error: Installation of Python not found, Python bindings not loaded.
See the Python "Order of Discovery" here: https://rstudio.github.io/reticulate/articles/versions.html#order-of-discovery.

I can only resolve this by choosing my python env manually:

> reticulate::use_condaenv("py312")
> 
> laminr::lamin_connect("laminlabs/lamindata") 
→ the database (1.7a1) is ahead of your installed lamindb package (1.6.0)
→ consider updating lamindb: pip install lamindb>=1.7
→ connected lamindb: laminlabs/lamindata

[lazappi]

> laminr::lamin_connect("laminlabs/lamindata") 
Error: .onLoad failed in loadNamespace() for 'laminr', details:
  call: NULL
  error: 'py_require' is not an exported object from 'namespace:reticulate'
> packageVersion("reticulate")
[1] ‘1.40.0’
> packageVersion("laminr")
[1] ‘1.1.1’

Yes, the py_require() function only exists from {reticulate} v1.41.0. I should bump the dependency.

> laminr::lamin_connect("laminlabs/lamindata") 
Downloading cpython-3.11.13-macos-aarch64-none (download) (17.2MiB)
 Downloading cpython-3.11.13-macos-aarch64-none (download)
Downloading botocore (13.0MiB)
Downloading numpy (5.1MiB)
Downloading pandas (10.3MiB)
Downloading pyarrow (29.4MiB)
Downloading pydantic-core (1.8MiB)
Downloading pygments (1.2MiB)
Downloading django (7.9MiB)
Downloading h5py (2.7MiB)
Downloading psycopg2-binary (3.1MiB)
 Downloading pygments
 Downloading pydantic-core
 Downloading h5py
 Downloading psycopg2-binary
 Downloading numpy
 Downloading django
 Downloading pandas
 Downloading botocore
   Building scipy==1.6.1
 Downloading pyarrow
  × Failed to build `scipy==1.6.1`

I ran into this last week as well, I was hoping it was just my system but I guess not. For some reason uv has started resolving to a really old version of scipy that doesn't compile. I haven't checked but I'm guessing it only happens through {reticulate} or you would have noticed earlier. We might need to add a specific scipy requirement somewhere, that should hopefully resolve things.

[falexwolf]

Yes, the py_require() function only exists from {reticulate} v1.41.0. I should bump the dependency.

Yes, please!

I ran into this last week as well, I was hoping it was just my system but I guess not. For some reason uv has started resolving to a really old version of scipy that doesn't compile. I haven't checked but I'm guessing it only happens through {reticulate} or you would have noticed earlier. We might need to add a specific scipy requirement somewhere, that should hopefully resolve things.

I'd say we wait until this is resolved within uv.

[falexwolf]

Yes, the py_require() function only exists from {reticulate} v1.41.0. I should bump the dependency.

Fixes like this should also get released within a matter of days, not within a matter of weeks or months. WDYT? Can you walk me through the release process on CRAN and then I could help with making sure fixes get released?

[Zethson]

uv has tons of tests and I would be surprised if it's a uv resolver problem (the uv resolver certainly still has issues). Such issues can occur when:

1. You're using a Python version which isn't supported by the more recent scipy releases. Then it finds one where Python wasn't pinned. This might be the case because it only checks for greater than 3.7: https://github.com/scipy/scipy/blob/5ab7426247900db9de856e790b8bea1bd71aec49/setup.py#L30
2. Some incompatibility between other packages which can then lead to [1] just above.

Considering that we have to make a release for the other issue, I don't see a reason not to pin it. I had wanted to pin minimum versions of dependencies for a while as we have a nice script for that at scverse but I haven't gotten to it yet.

[falexwolf]

I agree that lower bounding 3.7 is a frequent solution to uv resolver issues. Good idea!

In fact we should lower bound to 3.10 because this is our minimum.

[lazappi]

Fixes like this should also get released within a matter of days, not within a matter of weeks or months. WDYT? Can you walk me through the release process on CRAN and then I could help with making sure fixes get released?

Let's talk about this separately. I already wrote some notes about the release process but I could update them. CRAN has some limitations on how often you can make releases though. I think we are due for another one once we sort out the current issues.