use prefix-hosted channels

[lucascolley]

This allows devs to take advantage of sharded repodata by default if they have set disable-sharded = false under [repodata-config] in their pixi config (and hopefully without them having to do anything if the default changes in the future).

Currently, this step is unneeded if one also sets "https://conda.anaconda.org/conda-forge" = ["https://prefix.dev/conda-forge"] under [mirrors] in their config, but I guess this is a step in a good direction!

[lucascolley]

looks like we're at least partially blocked on prefix-dev/prefix-dev#21

[rgommers]

Don't we already use sharded repodata? pixi is way faster than conda/mamba especially on low-bandwidth connections, and I thought it was because of sharded repodata.

[lucascolley]

I don't think so. @baszalmstra we have to use the "https://prefix.dev/conda-forge" channel to get sharded repodata, right?

[baszalmstra]

right?

Yes, but you also need to configure pixi to use it currently. See: https://pixi.sh/latest/reference/pixi_configuration/#repodata-config

This will be the default in a future pixi version.

[lucascolley]

adding the following locally is sufficient to get sharded repodata working without this PR:

[mirrors]
"https://conda.anaconda.org" = ["https://prefix.dev/"]
# prefix.dev doesn't mirror conda-forge's label channels
# pixi uses the longest matching prefix for the mirror
"https://conda.anaconda.org/conda-forge/label" = [
  "https://conda.anaconda.org/conda-forge/label",
]

[repodata-config."https://prefix.dev"]
disable-sharded = false

After this PR, one would no longer need the [mirrors] section.

[lucascolley]

I think this is ready for merge, I don't know why NumPy is failing in CI

[rgommers]

I'm also getting weird errors locally, hard to tell why.

[lucascolley]

I'm also getting weird errors locally, hard to tell why.

I don't know what the root cause was, but pixi upgrade has bumped us from Python 3.12 to Python 3.13 which seems to have fixed things. The errors were saying No site-packages found in build-install for Python 3.12, perhaps spin is making some unwarranted assumptions about the build environment.

Anyway, this is ready for merge.

[rgommers]

Mostly looks good, but now it's test-jax that isn't working in either the jax or the array-api envs. Gives:

E   ValueError: '--array-api-backend' must be in ['numpy', 'array_api_strict', 'torch', 'dask.array']; got ['jax.numpy']

which is caused by an import error:

File ~/code/pixi-dev-scipystack/scipy/.pixi/envs/jax/lib/python3.12/site-packages/jaxlib/xla_client.py:33
     30 import ml_dtypes
     31 import numpy as np
---> 33 from . import xla_extension as _xla
     35 # Note this module does *not* depend on any Python protocol buffers. The XLA
     36 # Python bindings are currently packaged both as part of jaxlib and as part
     37 # of TensorFlow. If we use protocol buffers here, then importing both jaxlib
   (...)     44 # Pylint has false positives for type annotations.
     45 # pylint: disable=invalid-sequence-index
     47 ifrt_programs = _xla.ifrt_programs

ImportError: /home/rgommers/code/pixi-dev-scipystack/scipy/.pixi/envs/jax/lib/python3.12/site-packages/jaxlib/xla_extension.so: cannot enable executable stack as shared object requires: Invalid argument

That's on Linux. Does it work for you?

[lucascolley]

test-jax works for me in the jax env on macOS

[lucascolley]

jax-ml/jax#26781

Looks like we are waiting for conda-forge/jaxlib-feedstock#308, but either way not relevant to the changes in this PR.

We can pin jaxlib to <0.5.1 if you want

[rgommers]

Looks like we're all good now, thanks Lucas!