Clarification on multi-environment disk usage

[sabarish-vm]

Hi,

I am confused about how Pixi manages storage isolation across multi-environments. Consider the following pixi.toml file.

When the environments base and api are created, is it somehow possible to avoid creating duplicate copies of the packages in these environments? My impression was that Pixi handles deduplication automatically via the global cache.

However, when I check the disk sizes of envs/base and envs/api, they both appear to contain full duplicates of overlapping packages. This makes me wonder: can't Pixi resolve the dependency versions once globally, and then expose only the specific features/packages requested per environment? Is this not possible, or am I missing a configuration step?

[workspace]
channels = ["conda-forge"]
name = "envs"
platforms = ["osx-arm64"]

[dependencies]
python = "3.12.*"

[feature.core.dependencies]
ipython = ">=9.14.1,<10"
numpy = ">=2.4.6,<3"
scipy = ">=1.17.1,<2"
euporie = ">=2.10.4,<3"
matplotlib = ">=3.10.9,<4"
seaborn = ">=0.13.2,<0.14"
pandas = ">=3.0.3,<4"
jupyter = ">=1.1.1,<2"
notebook = ">=7.5.7,<8"
astropy = ">=7.2.0,<8"

[feature.aider.dependencies]
python = "3.12.*"

[feature.aider.pypi-dependencies]
aider-chat = "*"

[feature.api-test.dependencies]
fastapi = "*"
uvicorn = "*"

[environments]
base = {features = ["core"],solve-group = "group1"}
aider = {features = ["aider"], no-default-feature = true}
api = {features = ["api-test"],solve-group = "group1"}
combined = {features = ["core", "api-test"],solve-group = "group1"}

[delcenjo]

What du reports here is misleading — the packages are shared on disk, not really duplicated.

Pixi (via rattler) keeps one global package cache and, when it creates an environment, it doesn't copy files into .pixi/envs/<name>/ — it links them from that cache:

• On macOS / APFS (your osx-arm64 platform) it uses reflinks (copy-on-write clones via clonefile). The cloned files share their underlying blocks until modified, but du reports each clone at its full apparent size. So summing du envs/base + du envs/api looks like full duplication even though the real allocated space is paid roughly once.
• On filesystems without CoW it uses hardlinks instead (same inode in each env), which du also over-counts per path.

So numpy appearing "fully" in both base and api is expected: same blocks, two paths.

True duplication only happens when the cache and the envs are on different filesystems (a clone/hardlink can't cross them) or the FS supports neither, in which case pixi falls back to a real copy.

On the second question: no — each environment is a full, self-contained prefix. Pixi deduplicates at the storage layer (shared blocks/inodes), not by exposing a subset of one shared resolved environment per feature.

To sanity-check link sharing: stat -f %l <file> on macOS (stat -c %h <file> on Linux) shows the hardlink count; for APFS clones, compare the apparent size with actual disk pressure rather than trusting du.

[sabarish-vm]

Hi @delcenjo

Thank you.

I do see your point. So in this case I would then expect that upon running stat -f %l <file> I should get a number greater than 1 ?
When I run it on any/all of the files inside the .pixi folder, it always shows 1 for all the files in the env directories. Doesn't that mean that there is no link-sharing ?

[delcenjo]

Hi @sabarish-vm,

Short answer: a link count of 1 is exactly what you should expect on APFS, and it does not mean sharing is broken. stat -f %l is the right test for hardlinks, but it's the wrong tool for what pixi actually does on your filesystem.

Why the count stays 1

On APFS, pixi/rattler doesn't hardlink from the cache — it clones via the clonefile() syscall (reflinks / copy-on-write). The linking strategy is, in order of preference:

1. reflink / clone (CoW) — used on APFS, Btrfs, XFS-with-reflink, etc.
2. hardlink — fallback on filesystems without CoW
3. copy — last resort

A clone is not a hardlink:

• A hardlink is a second directory entry pointing to the same inode, so st_nlink goes up (2, 3, …). That's what stat -f %l counts.
• A clone (clonefile) gets its own inode with its own link count of 1, but its data extents are shared with the source until one side is modified, at which point only the changed blocks are copied.

So clonefile does not increment the hardlink count. Seeing 1 everywhere is the signature of clones, not the absence of sharing. (If pixi were hardlinking, that's when you'd see counts > 1 — but on APFS it doesn't, because clones are strictly better: they give you the same disk savings and keep environments isolated when files are edited.)

How to actually verify the sharing

stat (link count) and du (apparent per-file blocks) both mislead you here — du sums each clone's full apparent size and can't see that the extents are shared, so du envs/base + du envs/api will look like full duplication even when it isn't. The ground truth is real volume usage via df.

Quick proof that clonefile works on your volume:

df -h .

# clone (-c uses clonefile)
cp -c some/big/file ./clone.test
df -h .        # free space barely moves  -> blocks shared

# real copy for comparison
cp some/big/file ./copy.test
df -h .        # free space drops by the file size -> real duplication

rm clone.test copy.test

For the environments themselves, don't trust du -sh envs/*. Instead watch what's actually reclaimed:

df -h .
# delete one env (or the cache) and re-measure
rm -rf .pixi/envs/api
df -h .        # this delta is the *real* unique cost of that env

One thing worth checking

clonefile() can't cross volumes. If your project's .pixi/ and the rattler package cache live on different filesystems/disks, pixi can't clone or hardlink across them and falls back to a real copy — genuine duplication. Confirm they're on the same APFS volume:

pixi info   # shows the cache directory

If the cache (typically ~/Library/Caches/rattler/…) and your project are both on your main internal APFS disk, you're getting clones and paying for the shared blocks roughly once — regardless of what du or stat -f %l suggest.

(And re: your earlier question — yes, each environment is still a full, self-contained prefix; pixi deduplicates at the storage layer, not by exposing a subset of one shared resolved environment.)