Editable installs are broken after `__editable__` and `__path_hook__` changes

[abhinavsingh]

Description

I am starting to observe broken editable installs since past week.

Specifically, editable install now creates a __editable__.<pkg>.pth and __editable___<pkg>_finder.py files. These paths are returned as part of namespace.__path__, which has broken a lot of tooling.

1. pdoc3 went broke. Skipping __editable__ package seems to fix (hack) it.
   • See Skip __editable__ paths during iter_modules pdoc3/pdoc#408
2. We also noticed, namespace packages that installs a CLI are now also broken, because while the binary is under .venv/bin, it is unable to import the modules since it was installed using -e.
   • Modifying PYTHONPATH is the only option.
3. We also noticed, due to these changes even VSCode/Pylance is now broken, as it is unable to resolve paths to editable installed packages.
   • Adding "python.analysis.extraPaths" to .vscode/settings.json is the way out

I was unable to find any search result on Google for __editable__ and __path_hook__. So, I am unsure what I am up against here.

Expected behavior

1. Installed CLI from namespace package must work fine without modification of PYTHONPATH
2. namespace.__path__ must return actual path instead of __editable__.* which seems to break a lot of existing tooling

pip version

22.2.2

Python version

3.10.5

OS

MacOS 12.5

How to Reproduce

1. Create a namespace package that installs CLI (entry point)
2. Install the package with -e
3. Installed CLI is broken due to missing PYTHONPATH

Our projects are using pyproject.toml and setup.cfg.

Output

`ImportError: cannot import name 'entry_point' from 'namespace.cli' (unknown location)`

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pfmoore]

This is likely because setuptools released a new mechanism for implementing editable installs, as part of their PEP600 support. The pdoc3 project is probably not yet aware of the new mechanism, and will need updating.

There isn’t really anything for pip to do here, so I’m closing this issue. If it turns out that there is a pip problem here, feel free to reopen it.

[abhinavsingh]

@pfmoore This is likely going to create a lot of havoc. Editable install is how people operate in Python community. And if all of a sudden, entire tooling around editable install is going to break, then likely we must be planning better around it.

My 2 cents. May be, instead of enforcing the new mechanism as default, this could have been an opt in to start with. Instead of fighting this out, we decided to simply pin setuptools "setuptools <= 62.6.0". While this has put us out of misery, we are now stuck in the past. FWIW, new mechanism has decided to return a Path which returns in __path_hook__ as part of namespace. While that file actually doesn't exist on the disk.

[pradyunsg]

Please reach out to the setuptools maintainers. While various Python packaging tools do interoperate, the behaviour changes you’re seeing come from setuptools.

The maintainers of pip don’t control / maintain setuptools, which is where I recommend you reach out.

[pradyunsg]

I can actually transfer the issue, so… here you go. :)

[potiuk]

It looks like we started to have the same problem in Airflow.

• pip install -e . after checking out basically any version (including main) and running any Airlfow cli command results in

Traceback (most recent call last):
  File "/home/jarek/.pyenv/versions/airflow-test-cli3/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/home/jarek/code/airflow/airflow/__main__.py", line 38, in main
    args.func(args)
  File "/home/jarek/code/airflow/airflow/cli/cli_parser.py", line 50, in command
    func = import_string(import_path)
  File "/home/jarek/code/airflow/airflow/utils/module_loading.py", line 32, in import_string
    module = import_module(module_path)
  File "/home/jarek/.pyenv/versions/3.9.13/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/jarek/code/airflow/airflow/cli/commands/standalone_command.py", line 35, in <module>
    from airflow.www.app import cached_app
  File "/home/jarek/code/airflow/airflow/www/app.py", line 43, in <module>
    from airflow.www.extensions.init_views import (
  File "/home/jarek/code/airflow/airflow/www/extensions/init_views.py", line 28, in <module>
    from airflow.www.views import lazy_add_provider_discovered_options_to_connection_form
  File "/home/jarek/code/airflow/airflow/www/views.py", line 685, in <module>
    class AirflowBaseView(BaseView):
  File "/home/jarek/code/airflow/airflow/www/views.py", line 699, in AirflowBaseView
    extra_args['sqlite_warning'] = settings.engine.dialect.name == 'sqlite'
AttributeError: module 'airflow.settings' has no attribute 'engine'

This started to happen few days ago and adding Airlfow's source to PYTHONPATH solves the problem.

I did some bisecting and pin-pointed it to version 64.0.0 of setuptools. Version 63.4.3 worked perfectly fine, 64.0.0 breaks it. The reason seems to be as described in the issue. I've added those lines to our pyproject.toml and they solved the problem:

[build-system]
requires = ['setuptools==63.4.3']
build-backend = "setuptools.build_meta"

See: apache/airflow#25848

It is super-easy to reproduce now:

1. checkout main from https://github.com/apache/airflow
2. create and activate a new virtualenv
3. run pip install -e .
4. run airflow standalone

This works perfectly well (because of the 63.4.3 limit)

5. change the line in pyproject.toml to:

requires = ['setuptools==64.0.0']

6. create and activate a new virtualenv
7. run pip install -e .
8. run airflow standalone

You see the stack-trace above

[potiuk]

(The error is also present in all other 64.* and 65* versions BTW).

[abravalheri]

Hi @abhinavsingh,

Could you provide a minimal reproducer explaining more in details the issues you are having? (Maybe it is worthy to split multiple issues in multiple reproducers).

I tried to investigate what you are describing with the following reproducer, but I was not able to find the problematic behaviour you are referring to... (Maybe I understood something wrong? Or maybe the reproducer is missing something? Or -- best case scenario -- the problem might have already gotten solved in the latest release?).

# docker run --rm -it python:3.10.6 bash

cd /tmp && rm -rf /tmp/proj_root
mkdir -p /tmp/proj_root
mkdir -p /tmp/proj_root/namespace/cli
echo "def run(): print('hello world')" > /tmp/proj_root/namespace/cli/__init__.py
cat <<EOF > /tmp/proj_root/pyproject.toml
[build-system]
requires = ["setuptools>=65.1.0"]
build-backend = "setuptools.build_meta"

[project]
name = "namespace.cli"
version = "42"

[project.scripts]
namespace-cli = "namespace.cli:run"
EOF
cd /tmp/proj_root
python3.10 -m venv .venv
.venv/bin/python -m pip install -U pip
.venv/bin/python -m pip install -e .
cd /var
/tmp/proj_root/.venv/bin/python -c 'import namespace; print(f"{namespace.__path__=}")'
# ==> namespace.__path__=_NamespacePath(['/tmp/proj_root/namespace'])
/tmp/proj_root/.venv/bin/namespace-cli
# ==> hello world

As you can see above, the reproducer show that whenever possible setuptools will add the corresponding path entry to __path__.
Cases when this does not occur are likely associated with configurations by which users explicitly opt out from certain packages or there is simply no directory in the file system for the package to be associated with...

---

There are a few other remarks I would like to make:

• In Python, path entries do not need to be existing locations in the file system, this is mentioned in Python docs:

  Path entries need not be limited to file system locations. They can refer to URLs, database queries, or any other location that can be specified as a string.
  ...
  Most path entries name locations in the file system, but they need not be limited to this.
  ...
  Entries in sys.path can name directories on the file system, zip files, and potentially other “locations” (see the site module) that should be searched for modules, such as URLs, or database queries.

  We can also demonstrate this with the following snippet:

  # docker run --rm -it python:3.10.6 bash
  python3.10 -c 'import sys; print(sys.path)'
  # ==> ['', '/usr/local/lib/python310.zip', '/usr/local/lib/python3.10', '/usr/local/lib/python3.10/lib-dynload', '/usr/local/lib/python3.10/site-packages']
  ls -la /usr/local/lib/python310.zip
  # ==> ls: cannot access '/usr/local/lib/python310.zip': No such file or directory

• Most static analysis tools (including the ones used in code editors) have limitations in terms of aspects of the Python language they cannot handle. Import hooks (which have been part of the Python language specification for years, and are listed in PEP 660 as a perfectly viable implementation for editable installs), are notably one of these limitations. This is reported in Due to limitation, static analysis don't support import hook used in editable install v64+ #3518.
  Setuptools understand this limitation, and I am personally happy to contribute to an effort of improving this. However we do need someone to champion a design/architecture/standard that is well received in both packaging and static analysis community. I recommend anyone interested in pushing this forward to engage in the mailing list discussion, or on the thread on discourse.

• I would also recommend setuptools' docs on editable installs. There might be some useful information there.

[abravalheri]

@potiuk I suspect airflow's use case is different.

By having a look on the setup.py, it seems that airflow implements a custom develop command.

Setuptools's implementation of PEP 660 does not use the develop command¹. Instead, setuptools now rely in a complete different command. Please have a look on the docs for a few possible options on how to customise the editable installation.

---

By running:

# docker run --rm -it python:3.10.6 bash

apt update && apt install -y git build-essential
cd /tmp
git clone https://github.com/apache/airflow
cd /tmp/airflow
sed -i 's/==63.4.3/>=65.1.0/g' pyproject.toml
python -m venv .venv
.venv/bin/python -m pip install -U pip
.venv/bin/python -m pip install -e .
cd /var
/tmp/airflow/.venv/bin/python -c 'import airflow.settings; print(dir(airflow.settings))'
# ==> ['AIRFLOW_HOME', 'AIRFLOW_MOVED_TABLE_PREFIX', 'ALLOW_FUTURE_EXEC_DATES', 'CAN_FORK', 'CHECK_SLAS', 'COMPRESS_SERIALIZED_DAGS', 'Callable', 'DAEMON_UMASK', 'DAGS_FOLDER', 'DASHBOARD_UIALERTS', 'DEFAULT_ENGINE_ARGS', 'DONOT_MODIFY_HANDLERS', 'EXECUTE_TASKS_NEW_PYTHON_INTERPRETER', 'Engine', 'GUNICORN_WORKER_READY_PREFIX', 'HEADER', 'HIDE_SENSITIVE_VAR_CONN_FIELDS', 'IS_K8S_OR_K8SCELERY_EXECUTOR', 'KILOBYTE', 'LAZY_LOAD_PLUGINS', 'LAZY_LOAD_PROVIDERS', 'LOGGING_CLASS_PATH', 'LOGGING_LEVEL', 'LOG_FORMAT', 'List', 'MASK_SECRETS_IN_LOGS', 'MEGABYTE', 'MIN_SERIALIZED_DAG_FETCH_INTERVAL', 'MIN_SERIALIZED_DAG_UPDATE_INTERVAL', 'NullPool', 'Optional', 'PLUGINS_FOLDER', 'SASession', 'SIMPLE_LOG_FORMAT', 'SQL_ALCHEMY_CONN', 'STATE_COLORS', 'TIMEZONE', 'TYPE_CHECKING', 'USE_JOB_SCHEDULE', 'Union', 'WEBSERVER_CONFIG', 'WEB_COLORS', '__annotations__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '_get_rich_console', 'atexit', 'conf', 'configure_action_logging', 'configure_adapters', 'configure_logging', 'configure_orm', 'configure_vars', 'create_engine', 'custom_show_warning', 'dag_policy', 'dispose_orm', 'exc', 'executor_constants', 'functools', 'get_airflow_context_vars', 'get_dagbag_import_timeout', 'get_session_lifetime_config', 'import_local_settings', 'initialize', 'json', 'log', 'logging', 'original_show_warning', 'os', 'pendulum', 'pod_mutation_hook', 'prepare_engine_args', 'prepare_syspath', 'reconfigure_orm', 'replace_showwarning', 'scoped_session', 'sessionmaker', 'setup_event_handlers', 'sqlalchemy', 'sys', 'task_instance_mutation_hook', 'task_policy', 'tz', 'validate_session', 'warnings']

I can see that the package is indeed installed in the editable mode, however the engine attribute does not seem to be initialised. Maybe this is related to the customisations you guys are doing in the develop command, but that are no longer invoked when you run pip install -e .?

---

Footnotes

1. The PEP 660 standard is incompatible with the develop command. In the later, setuptools writes files directly to the installation directories, but the PEP 660 standard requires setuptools to handle a .whl file to pip, so pip can take care of creating files. ↩