Globally shared features

[NichtJens]

Problem description

Use case:

We provide a jupyterhub installation to our users in a HPC environment. They are allowed to create their own data-analysis envs to be used within jupyter.

In order to use some resources, the users need to match versions of a few packages to the versions of our installation. For instance:

1. Plotting: matplotlib, ipympl, ...
2. GPU nodes: cuda, cupy, ...
3. etc.

We would like to simplify this by providing globally shared features that can be used by every user in their respective toml files preferably without copy and paste.

---

Could pixi support such a use case?

Solution ideas:

Sadly, TOML does not support linking other toml files into the current file.

The obvious idea would be to allow an entry in ~/.pixi/config.toml that can point to one/several (?) files that contain feature definitions. The user needs to configure this once and then it works for all their envs.

If feature names appear in the global definition as well as a local pixi.toml, the local version wins allowing users to "overwrite" a global feature if they need to.

Similarly, if a list of several global feature files is allowed, the earlier/later (?) entry wins allowing a user to chain, e.g., ours and their global features into a merged list.

[Hofer-Julian]

Thanks for opening this issue @NichtJens.

That is indeed an interesting use case. What you could do already is to create "feature" packages which depend on the packages you want to include.

Each package could be defined by a rattler-build recipe and then you depend on it in your own pixi.toml like this:

[workspace]
channels = [
  "https://prefix.dev/conda-forge",
]
platforms = ["osx-arm64", "osx-64", "linux-64", "win-64"]
preview = ["pixi-build"]

[dependencies]
plotting_packages = { path = "/path/to/plotting_packages" }

[NichtJens]

Thanks @Hofer-Julian for the idea. That would probably work as well ... This might actually be preferable to my suggestion since the pixi.toml files stay self-contained, i.e., there's no magic global information that can be referenced in the pixi.toml without "including" it explicitly. What you provide IS the explicit include statement.

If you don't mind, could you give an example of what /path/to/plotting_packages would look like?
Would we need to build a package or is the recipe enough?
(Sorry, if the answers are obvious ... I admit I haven't had a chance to look into rattler :))

[Hofer-Julian]

Something like this should work:

# recipe.yaml
package:
  name: plotting_packages
  version: 0.1.0

build:
  number: 0

requirements:
  run:
    - matplotlib
    - numpy
    - jupyterlab

Just put the recipe.yaml in some folder and reference it in your pixi.toml