Use pixi for CI instead of conda

[purepani]

Currently, CI uses conda to test. It seems to be a pretty significant portion of the CI time, most of which seems to be on conda being slow(?). Additionally, there are currently 2 places that dependencies have to be maintained, with the conda build requiring a pip install within a build script. I think both of these can be improved by using pixi. It is built off of a much faster resolver(particularly using the uv resolver), and it has support for readying out of pyproject.toml. I am fairly certain that using this would obviate the need for everything in the recipe/ and build/ directories, with that info being able to be put into pyproject.toml and deduplicated.

I'm unsure what the general migration plan is(i.e. whether the plan is to completely move to a pyproject.toml do the standard "upload to conda-forge" workflow), but I think the speed improvements in CI might be able to justify it in the short term, depending on the current migration plan timeline.
@casperdcl

[casperdcl]

I'd prefer GHA cached art[ie]facts (e.g. dask/distributed#6855) rather than committing lockfiles.

[purepani]

I am personally for committing lock files in everything, but that doesn't matter here: you don't need to commit lock files for pixi. It will just create the lock file on the fly.
Caching is just one(but definitely a major) component of the CI speed here.

[purepani]

Generating the lock file on the fly is also sort of cursed to me, but on the other hand, it's not really any different than doing it with conda without a lock file so 🤷.
The caching is sort of orthogonal to the proposal though. You can cache with conda, and you can choose not to cache with pixi. The main gain of using pixi is the speed of resolution and environment creation, and also the consolidation of dependencies in one spot.

[JeppeKlitgaard]

I would be interested in contributing towards increased used of pixi within CIL. Haven't set up a few CIL environments over the last couple of weeks, I think there may be other significant benefits outside just speeding up the CI.

That is, if there is appetite for such a change. I understand using the conda CLI has served the project well over the years. Would also be happy to discuss this at the upcoming user meet