Pixi as an alternative to conda-lock

[scottyhq]

Is your feature request related to a problem? Please describe.
conda-lock has worked really well for this project! Looking forward it might be worth also supporting or switching to pixi for environment management https://pixi.sh

Describe the solution you'd like
Try using pixi instead of conda-lock and supporting pixi.lock as another file that can be installed

pangeo-docker-images/base-image/Dockerfile

Line 75 in 77b90e0

&& mamba install conda-lock -y \

pangeo-docker-images/base-image/Dockerfile

Lines 163 to 168 in 77b90e0

	; if test -f "conda-lock.yml" ; then echo "Using conda-lock.yml" & \
	conda-lock install --name ${CONDA_ENV} \
	; elif test -f "environment.yml" ; then echo "Using environment.yml" & \
	mamba env create --name ${CONDA_ENV} -f environment.yml \
	; else echo "No conda-lock.yml or environment.yml! *creating default env*" ; \
	mamba create --name ${CONDA_ENV} pangeo-notebook \

Describe alternatives you've considered
Stick with what's been working fine

Additional context
Discussion of pixi on conda-lock repo conda/conda-lock#615

Also https://github.com/prefix-dev/pixi-docker

[weiji14]

Been thinking about this too. I'd seriously consider making the jump once repo2docker supports pixi lock files (jupyterhub/repo2docker#1339), though we can test things out using the Dockerfile path too. We'll need to document this change though as some people might be depending on the conda-lock files and changing to pixi.lock will break their workflows.

[weiji14]

Was wondering how fast pixi would generate a lock file, so I tried out the pre-v3 release of conda-lock that allows generating a pixi.toml file from a conda environment.yml file (added in conda/conda-lock#664). Following the draft docs at conda/conda-lock#708, install pixi and conda-lock from main branch and then run:

cd pangeo-notebook/
conda-lock render-lock-spec --kind=pixi.toml  --stdout --file environment.yml --file ../base-notebook/environment.yml --pixi-project-name=pangeo-notebook --platform linux-64 --platform linux-aarch64 --platform osx-64 --platform osx-arm64 > pixi.toml

outputs the following pixi.toml file (this is from the branch at #592):

# This file was generated by conda-lock for the pixi environment manager.
# For more information, see <https://github.com/conda/conda-lock> and <https://pixi.sh>.
# Source files:
# - environment.yml
# - ../base-notebook/environment.yml

[project]
name = "pangeo-notebook"
platforms = ["linux-64", "linux-aarch64", "osx-64", "osx-arm64"]
channels = ["conda-forge"]

[dependencies]
adlfs = "*"
argopy = "<0.1.16"
awscli = "*"
black = "*"
boto3 = "*"
bottleneck = "*"
cartopy = "*"
cdsapi = "*"
cf_xarray = "*"
cfgrib = "*"
ciso = "*"
cmocean = "*"
contextily = "*"
dask-geopandas = "*"
dask-ml = "*"
datashader = "*"
descartes = "*"
duckdb-cli = "*"
earthaccess = "*"
eofs = "*"
erddapy = "*"
esmpy = "*"
fastjmd95 = "*"
flox = "*"
fsspec = "*"
gcm_filters = "*"
gcsfs = "*"
gdal = "*"
geocube = "*"
geopandas = "*"
geopy = "*"
geoviews-core = "*"
gh = "*"
gh-scoped-creds = "*"
git-lfs = "*"
gsw = "*"
h5netcdf = "*"
h5py = "*"
holoviews = "*"
hvplot = "*"
intake = "*"
intake-esm = "*"
intake-geopandas = "*"
intake-stac = "*"
intake-xarray = "*"
ipdb = "*"
ipykernel = "*"
ipyleaflet = "*"
ipympl = "*"
ipytree = "*"
ipywidgets = "*"
jupyter-panel-proxy = "*"
jupyter-resource-usage = "*"
jupyter-sshd-proxy = "*"
jupyterlab-git = "*"
jupyterlab-lsp = "*"
jupyterlab-myst = "*"
jupyterlab_code_formatter = "*"
kerchunk = "*"
line_profiler = "*"
lonboard = "*"
lxml = "*"
lz4 = "*"
matplotlib-base = "*"
memory_profiler = "*"
metpy = "*"
nb_conda_kernels = "*"
nbstripout = "*"
nc-time-axis = "*"
netcdf4 = "*"
numbagg = "*"
numcodecs = "*"
numpy = "*"
numpy_groupies = "*"
odc-stac = "*"
openssh = "*"
pandas = "*"
panel = "*"
pangeo-notebook = "2024.11.15.*"
param = "*"
parcels = "*"
pip = "*"
planetary-computer = "*"
pop-tools = "*"
pyarrow = "*"
pycamhd = "*"
pydap = "*"
pystac = "*"
pystac-client = "*"
python = "3.12.*"
python-blosc = "*"
python-duckdb = "*"
python-gist = "*"
python-graphviz = "*"
python-lsp-ruff = "*"
python-xxhash = "*"
rasterio = "*"
rechunker = "*"
rio-cogeo = "*"
rioxarray = "*"
ruff = "*"
s3fs = "*"
satpy = "*"
scikit-image = "*"
scikit-learn = "*"
scipy = "*"
seaborn = "*"
snakeviz = "*"
sparse = "*"
stackstac = "*"
tiledb-py = "*"
timezonefinder = "*"
watermark = "*"
xarray = "*"
xarray-spatial = "*"
xarray_leaflet = "*"
xarrayutils = "*"
xbatcher = "*"
xclim = "*"
xesmf = "*"
xgboost = "*"
xgcm = "*"
xhistogram = "*"
xmip = "*"
xmitgcm = "*"
xpublish = "*"
xrft = "*"
xskillscore = "*"
xvec = "*"
xxhash = "*"
zarr = "*"

Then, I tried to generate a pixi.lock file from the pixi.toml file. This was ran using pixi=0.38.0:

pixi update

and... it took 42 seconds 🤯, compared to 23m using conda-lock + mamba=1.5.9.

That said, pixi environments are typically project-centric (one environment per project), rather than a global-centric (one environment, many projects) - https://pixi.sh/latest/switching_from/conda/#environment-vs-project. There is a way to create 'global' detached-environments, but it doesn't look recommended.

We'll need to think through this a bit for pangeo-docker-images, ideally we could get the benefits of the fast resolvo solver used by pixi, but still have dependency specification files compatible with tools from the rest of the conda ecosystem.

[weiji14]

Oo, it does look like pixi allows exporting conda explicit specification files at least, according to https://pixi.sh/latest/reference/cli/#project-export-conda-explicit-spec:

pixi project export conda-explicit-spec .

produces several default_{platform}_conda_spec.txt files. Sample from linux-64:

# Generated by `pixi project export`
# platform: linux-64
@EXPLICIT
https://conda.anaconda.org/conda-forge/linux-64/_libgcc_mutex-0.1-conda_forge.tar.bz2#d7c89558ba9fa0495403155b64376d81
https://conda.anaconda.org/conda-forge/linux-64/libgomp-14.2.0-h77fa898_1.conda#cc3573974587f12dda90d96e3e55a702
https://conda.anaconda.org/conda-forge/linux-64/_openmp_mutex-4.5-2_gnu.tar.bz2#73aaf86a425cc6e73fcf236a5a46396d
https://conda.anaconda.org/conda-forge/linux-64/libgcc-14.2.0-h77fa898_1.conda#3cb76c3f10d3bc7f1105b2fc9db984df
https://conda.anaconda.org/conda-forge/linux-64/libgcc-ng-14.2.0-h69a702a_1.conda#e39480b9ca41323497b05492a63bc35b
...

Now if we can find a way for it to export to a unified conda-lock.yml file, that would be sweet.

[weiji14]

Now if we can find a way for it to export to a unified conda-lock.yml file, that would be sweet.

Found https://github.com/basnijholt/pixi-to-conda-lock through conda/conda-lock#780. I tested it out with conda-lock=3.0.0 and pixi-to-conda-lock=0.3.0 like so:

cd pangeo-notebook/
conda-lock render-lock-spec --kind=pixi.toml  --stdout --file environment.yml --file ../base-notebook/environment.yml --pixi-project-name=pangeo-notebook --platform linux-64 --platform linux-aarch64 --platform osx-64 --platform osx-arm64 > pixi.toml
pixi update  # generate pixi.lock file
pixi-to-conda-lock pixi.lock  # generate conda-lock.yml file

and... it does give a conda-lock.yml file, but there were differences between the pixi-to-conda-lock tool and conda-lock (content_hash is missing, and packages weren't sorted in the same order based on platform). Still, something to keep an eye on.