Feature/pixi

[caitlinadams]

This adds support for using the pixi package manager, including:

• Updates to the pyproject.toml file to leverage pixi
• Pixi tasks to run tests and export environment
• Using Pixi in the GitHub CI action

[caitlinadams]

@abradley60 -- can you try running some of your AWS code with the pixi environment? All tests are passing using pixi, but I'm aware we don't have tests for everything right now. I've tried the download-etad function, and that's still working, but haven't tried anything else.

[abradley60]

Is this meant to be changed from sar-pipeline?

[caitlinadams]

No, I must have done that by mistake when trying to work out the pixi export. Will revert

[abradley60]

@abradley60 -- can you try running some of your AWS code with the pixi environment? All tests are passing using pixi, but I'm aware we don't have tests for everything right now. I've tried the download-etad function, and that's still working, but haven't tried anything else.

Yep will do

[abradley60]

General Question @caitlinadams does the pixi build rely solely on the dependancies in the pyproject.toml?

edit - the project.toml references the appropriate conda env file?

[caitlinadams]

General Question @caitlinadams does the pixi build rely solely on the dependancies in the pyproject.toml?

That's my understanding. Pixi refer to the pixi releated code in the pyproject.toml as the manifest. It's the bit that you interact with -- so when you run pixi add <package> pixi will automatically update the manifest with the package you added. There's then also the lock file, which we shouldn't interact with -- this is the explicit dependency list, generated when we install a package. It's valuable to explicitly list installed packages, their dependencies, and their versions.

Whenever you run a command with pixi, it will double check that your environment is up-to-date with the manifest (pyproject.toml), update the lock file if the manifest has changed, and then build the environment (see pixi docs.

edit - the project.toml references the appropriate conda env file?

The pyproject.toml file doesn't actually interact with the conda env file at all. The way I built our pixi environment was:

1. Identify all packages that we import in the code
2. Set up a pixi workspace
3. manually run pixi add <package> for every package we directly import
4. [Optionally] run the pixi run export_conda > environment.pixi.yml to manually create a version of the pixi environment as a conda style env file

For all future development, the way we'd use pixi is:

1. Add a new package with pixi add <package> -- this automatically updates the pyproject.toml and the lock file
2. run the pixi run export_conda > environment.pixi.yml to manually update the conda style env file based on the pixi env

Does that help clarify?

[abradley60]

General Question @caitlinadams does the pixi build rely solely on the dependancies in the pyproject.toml?

That's my understanding. Pixi refer to the pixi releated code in the pyproject.toml as the manifest. It's the bit that you interact with -- so when you run pixi add <package> pixi will automatically update the manifest with the package you added. There's then also the lock file, which we shouldn't interact with -- this is the explicit dependency list, generated when we install a package. It's valuable to explicitly list installed packages, their dependencies, and their versions.

Whenever you run a command with pixi, it will double check that your environment is up-to-date with the manifest (pyproject.toml), update the lock file if the manifest has changed, and then build the environment (see pixi docs.

edit - the project.toml references the appropriate conda env file?

The pyproject.toml file doesn't actually interact with the conda env file at all. The way I built our pixi environment was:

1. Identify all packages that we import in the code
2. Set up a pixi workspace
3. manually run pixi add <package> for every package we directly import
4. [Optionally] run the pixi run export_conda > environment.pixi.yml to manually create a version of the pixi environment as a conda style env file

For all future development, the way we'd use pixi is:

1. Add a new package with pixi add <package> -- this automatically updates the pyproject.toml and the lock file
2. run the pixi run export_conda > environment.pixi.yml to manually update the conda style env file based on the pixi env

Does that help clarify?

Yep that makes sense thanks for the explanation. Given the exported environment.pixi.yml file doesn't contain versions for pip packages, I imagine there is a chance these could eventually deviate from our code. I guess the solution is to always build from the pixi file?

[caitlinadams]

Yep that makes sense thanks for the explanation. Given the exported environment.pixi.yml file doesn't contain versions for pip packages, I imagine there is a chance these could eventually deviate from our code. I guess the solution is to always build from the pixi file?

I've just tested this -- if we explicitly list the versions in the pyproject.toml file, they will come through into the environment.pixi.yml file on export.

As such, there is an extra step of manually reviewing the pyproject.toml file after running pixi add <package> or pixi add --pypi <package>:

1. Run pixi list -x to check what version got installed
2. Find the package in the pyproject.toml file and add the version number if it's missing (which will be the case if from pypi)
3. Do any formatting to tidy up the pyproject.toml (pip packages don't naturally format as one package per line in pyproject.toml, but I think this format is much more readable)

[caitlinadams]

Looks great, thanks for making those additions. I've put a few suggestions -- but mostly to remove upper bounds I forgot about!

[abradley60]

@caitlinadams any idea on the test failure?

[caitlinadams]

Looking at the error and the GitHub action definition, I wonder whether it's getting a mismatch between the lockfile and the pyroject.toml. The action seems to want to run pixi install --locked. This will abort if the lockfile isn't up-to-date with the pyproject.toml (see: https://pixi.sh/dev/reference/cli/pixi/install/#arg---locked)

So, it should be a matter of pulling the change to the manifest, running pixi install --all and then pushing the changes to the lockfile back up.