Add a "conceptual" layer archiving mode

[ncoghlan]

The initial layer archiving mode is a concrete "deployment" archiving model: each layer definition in the stack specification corresponds to a concrete deployment artifact which needs to be deployed on the target system. If one package in a layer needs to be updated, the entire layer needs to be republished.

The advantage of this model is that the deployment artifacts are self-contained: if you have the required artifacts, you can create the environment on a target system without any further need for internet access. It also entirely avoids the need to have Python package installation tools installed in the deployed environments.

While this mechanism works, it comes with assorted practical complications around managing the versioning and potential parallel installation of the framework layers, as well as ensuring all the required framework layers for a given application layer are installed prior to attempt to use it. There's also runtime complexity around ensuring all the deployed environments are accessible for Python imports and for shared object loading.

However, there's a potential way to decouple the actual deployment artifacts from the stack definitions used to determine the lock files for each layer. Specifically:

• runtime layers change to ship with uv as the only additional package installed in the runtime environment. The other runtime dependency declarations are still used to constrain the locking of any upper layers, but they're no longer shipped in the runtime layer itself.
• a uv.toml configuration file definition is also included in the runtime archive, defining how to retrieve individual packages based on the lock time config. This config file will also point uv's caching definition to a defined location (defaulting to var/cache/uv inside the runtime layer archive, as that should consistently be on the same file system as the installed application layer environments, which is needed for uv's package caching mechanism to be fully effective)
• framework layers no longer produce deployment artifacts at all. They're solely used to constrain the other layers that depend on them to a common set of package versions at lock time.
• the full requirements lock file is added to the built application layer archives
• application layers change to deploy conventional isolated virtual environments with no sys.path and dynamic library loading path customisations. Their post-installation scripts still adjust the virtual environment for its deployment location, and then do something new: they use the base environment's uv installation (and its uv config file) to install the locked dependencies for that application environment, relying on uv's caching mechanism to ensure only one copy of each required package is downloaded and installed.

uv can be used as the installer in this approach, as these are regular isolated virtual environments, so they won't be affected by astral-sh/uv#2500

The environments deployed this way will also be complete (with their RECORD files and executable wrappers fully populated), rather than having certain files that potentially encode absolute paths (or their hashes) deleted.

Deployments still need to ensure the right runtime is installed for each application layer, but there's no need to identify and install the individual framework layers. Framework versioning also gets a lot easier, since the published application layer archives don't have a runtime dependency on the framework layers in this archiving model - the framework layers are just expressing a set of dependencies that are intentionally being kept to the same versions across different application environments.

This approach also intrinsically allows partial dependency sharing even as layers are updated.

Compared to the benefits, the arguable list of downsides is relatively short:

• needs a Python package installer at installation time (true, but those are rarely far away given ensurepip in the standard library, so having one preinstalled in the base runtime environments shouldn't be that significant of a concern)
• needs access to an index server at installation time (true, but it would be possible to make that a controlled server, rather than installing directly from public index servers)
• needs to perform additional work at installation time (arguable, as it doesn't take many saved package downloads from the improved sharing across layer versions to compensate for a lot of local package installations from the shared package cache)
• needs application environments to be installed on the same filesystem as their base runtime environments (true, but these are already required to be installed in the same target folder, so this shouldn't be that significant of a new concern)
• managing the package cache so it doesn't grow excessively large over time may be a hassle (true, but this problem also exists when it comes to managing previously installed framework layers that no application layers are currently referencing)
• ??? anything else I've missed?

[neilmehta24]

Overall, I think that building the framework on-the-fly is a good idea. I've been wondering why venvstacks wasn't something like "ship just a lock file and download requirements from that lock file", but there are definitely advantages to the current layering system that we should make sure we keep. Here are some worries that I have, that relate to moving away from discrete pre-determined layers:

runtime layers change to ship with uv as the only additional package installed in the runtime environment

uv, as I understand it, is a whole ecosystem, including a python-build-standalone downloader. We would need to configure uv to be in an offline-only mode, or at least a mode where it's offline by default. We wouldn't want a network request made by uv that the dev/user didn't expect. There might be some flexibility here on online usage, but at the very least we would need to understand and scope the uv behavior

application layers change to deploy conventional isolated virtual environments with no sys.path and dynamic library loading path customisations

This may dramatically increase the rate of disk usage by LM Studio. Right now, one of our largest dependencies for MLX, torch, doesn't replicate on disk whenever we release an update for the app layer. Looking at the version numbers, we've been updating the MLX app layer about once per week. The MLX app layer only takes up 18% of the disk space of the deployment

needs access to an index server at installation time

We would need to ensure that we won't run into signing/distribution issues. For the LM Studio venvstacks apps, this would require that we host our own package mirror, so that downtime/yanks on PyPI don't affect the usability of LM Studio. We would also need to refactor the download tracker in LM Studio; is there a way to track the download progress here? Also, we sign the libraries/executables in the venvstacks build; this refactor means we may need to figure out the signing piece again

this problem also exists when it comes to managing previously installed framework layers that no application layers are currently referencing

Managing the removal of framework layers is at least possible with the current implementation, we can inspect the sitecustomize of each app layer we're using. Sure there may be some zombie framework layers, but with a uv cache we would have to solve for every environment we want to keep, right?

anything else I've missed?

How do we manage custom wheels whose SHA has changed, but version has not? We don't want to break older versions of our MLX engine if we rebuild and upload a custom wheel. If the version string of the wheel always needs to change, the mechanics of creating a new lock file would be more tedious.

[ncoghlan]

we sign the libraries/executables in the venvstacks build; this refactor means we may need to figure out the signing piece again

Ouch, that is a piece of the problem that definitely didn't occur to me when writing this up (and I don't have a good answer for it).

How do we manage custom wheels whose SHA has changed, but version has not? We don't want to break older versions of our MLX engine if we rebuild and upload a custom wheel. If the version string of the wheel always needs to change, the mechanics of creating a new lock file would be more tedious.

That's also an excellent point. Even with a custom index server, you're still only allowed one wheel per platform tag per release. That means you would have to declare exact string matches against full local version strings, instead of just capturing the wheel contents in the published layer archives.

[ncoghlan]

Regarding disk usage, the key idea behind the conceptual layers is the way uv works: it implicitly installs into shared cache directories, and then hard links or copy-on-write clones those installed archives into the target environment (search for the --link-mode option in https://docs.astral.sh/uv/reference/cli/#uv-pip-install)

[ncoghlan]

For the time being, I've added a design note for the current layer archive deployment model in #147.

This conceptual layering idea still seems like a good way to go in the future, but there's no rush on enabling it, so we can take the time to consider whether the trade-offs mentioned above are genuine trade-offs, or if we can tweak the conceptual layering design concept to avoid losing those desirable properties of the current layer archiving model.

[ncoghlan]

Some notes on uv.lock options that we'd likely want to set if/when pursuing this idea:

• https://docs.astral.sh/uv/reference/settings/#cache-dir
• https://docs.astral.sh/uv/reference/settings/#exclude-newer
• https://docs.astral.sh/uv/reference/settings/#index
• https://docs.astral.sh/uv/reference/settings/#no-build
• https://docs.astral.sh/uv/reference/settings/#python-downloads (setting to never disables Python runtime downloads)
• https://docs.astral.sh/uv/reference/settings/#python-preference (additional mechanism to ensure uv doesn't try to use its own Python installs)
• https://docs.astral.sh/uv/reference/settings/#upgrade (disallow package upgrades beyond the pinned dependency set)
• https://docs.astral.sh/uv/reference/settings/#pip_only-binary
• https://docs.astral.sh/uv/reference/settings/#pip_require-hashes
• https://docs.astral.sh/uv/reference/settings/#pip_torch-backend

(Interesting, I hadn't noticed the dedicated PyTorch support before! This opens up some very interesting possibilities around the new standardised pylock.toml format and the fact that it records full artifact download URLs, not just package names, versions, and hashes)