Created virtual env does not honour sys.base_prefix correctly

[pelson]

Issue

When creating a virtual environment, the base prefix of the virtual environment should be the same as the base prefix of the Python that is creating the virtual environment - there should be no additional resolve steps with regards to symlinks.

This behaviour is seen in venv, and was implemented as expected until #2686, where symlinks were resolved.

Why is this important?

In big (scientific) institutions, it is common to have a network mounted filesystem which can be access from all managed machines. This could be to mount a homespace, or to mount some data etc. (I've seen both). To scale this up, it is necessary to have multiple filesystem servers all using the same underlying storage. Machines are then clustered to point to different servers, but the user doesn't know which server a machine is talking to. When combining this with something such as autofs, you find that the server being contacted is in the path (e.g. /nfs/some-machine/), and to smoothen this out accross machines, the managed machines get a canonical symlink (e.g. /project/{my-project} which symlinks to the specific machine mountpoint). Essentially:

machine1:

ls -ltr /project/my-project -> /nfs/nfs-server1

machine2:

ls -ltr /project/my-project -> /nfs/nfs-server2

With this setup, you can create a virtual environment in /project/my-project which works on both machines IFF the symlink is not resolved. This is the behaviour of CPython, and also the behaviour of venv and virtualenv<20.25.1.

Reproducer

Written in the form of a pytest (w validation against venv also):

import os
import pathlib
import sys
import subprocess
import typing

import pytest


def read_venv_config(venv_prefix: pathlib.Path) -> typing.Dict[str, str]:
    venv_cfg_path = venv_prefix / 'pyvenv.cfg'

    cfg = {}
    with venv_cfg_path.open('rt') as cfg_fh:
        for line in cfg_fh:
            name, _, value = line.partition('=')
            cfg[name.strip()] = value.strip()
    return cfg


@pytest.mark.parametrize("venv_impl", ["venv", "virtualenv"])
def test_symlink_python(tmp_path: pathlib.Path, venv_impl: str) -> None:
    py_link = tmp_path / "some-other-prefix"
    pathlib.Path(py_link).symlink_to(sys.base_prefix)

    dest_venv = tmp_path / "some-venv"

    py_bin = py_link / 'bin' / 'python'
    subprocess.run([py_bin, '-m', venv_impl, str(dest_venv)], check=True)

    cfg = read_venv_config(dest_venv)

    py_base_prefix = subprocess.check_output([py_bin, '-c', 'import sys; print(sys.base_prefix)'], text=True).strip()

    # Get the link of the py bin. Don't recursively resolve this (like pathlib.resolve would do)
    py_bin_link = os.readlink(dest_venv / 'bin' / 'python')
    assert py_bin_link == str(py_link / 'bin' / 'python')

    assert str(py_link) == py_base_prefix
    assert py_base_prefix + '/bin' == cfg['home']

The result is a pass in 20.25.0 and a fail since:

>       assert py_base_prefix + '/bin' == cfg['home']
E       AssertionError: assert '/tmp/pytest-of-pelson/pytest-401/test_symlink_python_virtualenv0/some-other-prefix/bin' == '/path/to/my/environment/bin'
E         - /path/to/my/environment/bin
E         + /tmp/pytest-of-pelson/pytest-401/test_symlink_python_virtualenv0/some-other-prefix/bin

It is worth noting that the implementation of #2686 has a bug in the fact that the symlink at dest_venv / 'bin' / 'python' points to sys.base_prefix / 'bin' / 'python', and not the resolved symlink location of sys.base_prefix. Therefore the home value is inconsistent with the symlink that is created by virtualenv. The test validates this.

Implications

There are two issues which #2686 closed:

To be honest, I'm not sure what #2682 is asking for. Perhaps is is a request for behaviour that is different for venv and also the std library (wrt. sys.base_prefix) @mayeut.

For #2684, I also don't fully understand the reason for this being a virtualenv issue (this is my problem, not a problem with the issue itself) - and somebody who knows what the correct behaviour should be would need to chime in (perhaps @ofek?).

[pfmoore]

It seems to me there are two separate use cases related to symlinks (as well as #2684, which is related to canonicalisation of filenames, rather than symlinks).

In #2682, the use case is running /foo/bar/bin/python virtualenv.py env_name, where /foo/bar/bin/python is a symlink to /base_python/bin/python (in particular, note that there's no Python environment in /foo/bar, there's just the symlink in bin). In that case, you have to resolve the symlink to find the actual base environment, where the lib directory and other parts of the environment are situated.

In the case here, though, you are running /foo/bar/bin/python virtualenv.py env_name, where /foo/bar is itself a symlink to a full Python environment. In that case, the environment can be referenced either by its base name, or via the symlink /foo/bar/{bin,lib,...}. For your use case, you want the symlink to be retained, to allow portability when the symlink is constant, but the linked-to environment changes.

To be honest, I don't think there's a clear "one size fits all" solution here. We could say that if the python executable itself is a symlink, then resolve it, but if a component is a symlink, then don't. But that seems fairly hacky, and prone to errors if there are situations we haven't considered.

My personal experience with symlinks is on Windows, where if an exe loads a DLL saved alongside it, a symlink won't run as the OS doesn't follow the symlink to find the DLL. So by analogy with that, I'd have been inclined to say that #2682 wasn't a bug, but was simply a consequence of how symlinks work when trying to find dependent files relative to the executable. But I'm clearly in a minority here, as #2682 was accepted and fixed.

I don't have a good answer here, except to say that we should probably follow the behaviour of the core venv module, simply because in the absence of an obviously correct rule, consistency is key. Then, if anyone wants a change to the rules, they should start with venv rather than here.

[ofek]

I don't have a good answer here, except to say that we should probably follow the behaviour of the core venv module, simply because in the absence of an obviously correct rule, consistency is key. Then, if anyone wants a change to the rules, they should start with venv rather than here.

This is interesting, I thought it was other way around and we do novel stuff here which venv may or may not choose to incorporate. One such example is virtualenv being way faster.

[pfmoore]

I'd say that behaviour should be consistent, but quality of life details (like performance, additional shell integrations) are what distinguishes virtualenv. So venv defines the base mechanism, we make it more user friendly.

[mayeut]

I thought it was other way around and we do novel stuff here which venv may or may not choose to incorporate

I thought the same.

I don't have a good answer here, except to say that we should probably follow the behaviour of the core venv module, simply because in the absence of an obviously correct rule, consistency is key. Then, if anyone wants a change to the rules, they should start with venv rather than here.

Issues were filed in both cpython & virtualenv (with a link to the cpython issue). While the cpython issue is still opened, the virtualenv one was "fixed" introducing a regression in other use-cases.

If the aim of virtualenv is to be bug to bug compatible with venv (i.e. consistent behavior), which might differ depending on the python minor version or the python implementation, then so be it. In that case, the bug report template in virtualenv should probably be updated to reflect just that.

To be honest, I don't think there's a clear "one size fits all" solution here. We could say that if the python executable itself is a symlink, then resolve it, but if a component is a symlink, then don't. But that seems fairly hacky, and prone to errors if there are situations we haven't considered.

As of now, there are 3 identified use-cases related to symlinks that might be addressed by the hack mentioned (I don't know enough about the Windows one to be sure nothing else would be required).
IMHO, living with the status-quo of one or the other being broken does not seem beneficial to the ecosystem as a whole. At least those 3 are identified and can be unit-tested (but, per consistent behavior with venv, this might have to wait on the cpython issue being addressed first).

[pelson]

It seems to me there are two separate use cases related to symlinks (as well as #2684, which is related to canonicalisation of filenames, rather than symlinks).

Agreed. I think the terms "prefix symlink" and "executable symlink" can be helpful for clarity of discussion. There is a third type of use case employed by the manylinux container images, that of "executable wrappers", which essentially are scripts with "/path/to/a/real/bin/python" "$@" (I think this may also cover the concept for Windows discussed in astral-sh/uv#8433 (comment)). I mention this for completeness - I don't have a proposal on what the behaviour should be at this point.

In #2682, the use case is running /foo/bar/bin/python virtualenv.py env_name, where /foo/bar/bin/python is a symlink to /base_python/bin/python (in particular, note that there's no Python environment in /foo/bar, there's just the symlink in bin). In that case, you have to resolve the symlink to find the actual base environment, where the lib directory and other parts of the environment are situated.

I've spent a lot of time to understand the existing CPython approach when there is no virtual environment, as it seems to work well with the cases I have tested. It also resolves the symlink to determine the prefix (not via the equivalent of realpath). I therefore think your suggestion is a good one.

I had previously written a lot of detail regarding the potential improvements to this, but in practice I think the bug that I'm reporting boils down to calling os.readlink recursively, instead of calling os.path.realpath until we arrive at a non-symlink (in https://github.com/pypa/virtualenv/pull/2686/files).

In terms of next steps, there are now three options:

1. Do nothing until decided in CPython (in the meantime virtualenv is different to venv and uv venv)
2. Revert the original change (in the meantime the original issue in venv base-executable path does not resolve symlinks using realpath() #2682 is re-opened)
3. Move to recursive realpath in virtualenv (in the meantime the behaviour is more aligned to venv and uv venv, but ultimately achieves it in a slightly different way - unknown bugs could crop up)

Personally, I'm motivated to see this fixed properly everywhere, but also consider the new behaviour of virtualenv to be wrong (until venv standardises it) - clearly, IMO the existing implementation in virtualenv shouldn't stay as it is in the long-term. I therefore would advocate for 2, and in parallel would work on providing as much information to 1 as is helpful to move it forward.

[gaborbernat]

3. Move to recursive realpath in virtualenv (in the meantime the behaviour is more aligned to venv and uv venv, but ultimately achieves it in a slightly different way - unknown bugs could crop up

PR with this would be accepted and released. @pelson please submit oe if possible.

[mayeut]

There is a third type of use case employed by the manylinux container images, that of "executable wrappers"

These only exist because of the different/inconsistent/buggy behaviors when using "executable symlinks". If symlinking the python executable to an arbitrary location (which could also include another python install prefix) was working, then those wrappers would be replaced by symlinks. In the meantime, manylinux assumes that symlinking the python executable outside its own prefix bin directory is not allowed (except in the specific case of virtual environments).